rss.xml

<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0"><channel><title>Store Halfword Byte-Reverse Indexed</title><link>https://sthbrx.github.io/</link><description>A Power Technical Blog</description><lastBuildDate>Mon, 07 Aug 2023 12:00:00 +1000</lastBuildDate><item><title>Going out on a Limb: Efficient Elliptic Curve Arithmetic in OpenSSL</title><link>https://sthbrx.github.io/blog/2023/08/07/going-out-on-a-limb-efficient-elliptic-curve-arithmetic-in-openssl/</link><description>&lt;p&gt;So I've just managed to upstream some changes to OpenSSL for a &lt;a href="https://github.com/openssl/openssl/blob/master/crypto/ec/ecp_nistp384.c"&gt;new strategy&lt;/a&gt; I've developed for efficient arithmetic used in secp384r1, a curve prescribed by NIST for digital signatures and key exchange. In spite of its prevalence, its implementation in OpenSSL has remained somewhat unoptimised, even as less frequently used curves (P224, P256, P521) each have their own optimisations.&lt;/p&gt;
&lt;p&gt;The strategy I have used could be called a 56-bit redundant limb implementation with &lt;em&gt;Solinas reduction&lt;/em&gt;. Without too much micro-optimisation, we get ~5.5x speedup over the default (Montgomery Multiplication) implementation for creation of digital signatures.&lt;/p&gt;
&lt;p&gt;How is this possible? Well first let's quickly explain some language:&lt;/p&gt;
&lt;h2&gt;Elliptic Curves&lt;/h2&gt;
&lt;p&gt;When it comes to cryptography, it's highly likely that those with a computer science background will be familiar with ideas such as key-exchange and private-key signing. The stand-in asymmetric cipher in a typical computer science curriculum is typically RSA. However, the heyday of Elliptic Curve ciphers has well and truly arrived, and their operation seems no less mystical than when they were just a toy for academia.&lt;/p&gt;
&lt;p&gt;The word 'Elliptic' may seem to imply continuous mathematics. As a useful cryptographic problem, we fundamentally are just interested with the algebraic properties of these curves, whose points are elements of a &lt;a href="https://en.wikipedia.org/wiki/Finite_field"&gt;finite field&lt;/a&gt;. Irrespective of the underlying finite field, the algebraic properties of the elliptic curve group can be shown to exist by an application of &lt;a href="https://en.wikipedia.org/wiki/Bézout%27s_theorem#:~:text=Bézout%27s%20theorem%20is%20a%20statement,the%20degrees%20of%20the%20polynomials."&gt;Bézout's Theorem&lt;/a&gt;. The &lt;a href="https://en.wikipedia.org/wiki/Algebraic_group"&gt;group operator&lt;/a&gt; on points on an elliptic curve for a particular choice of field involves the intersection of lines intersecting either once, twice or thrice with the curve, granting notions of addition and doubling for the points of intersection, and giving the 'point at infinity' as the group identity. A closed form exists for computing a point double/addition in arbitrary fields (different closed forms can apply, but determined by the field's &lt;a href="https://en.wikipedia.org/wiki/Characteristic_(algebra)"&gt;characteristic&lt;/a&gt;, and the same closed form applies for all large prime fields).&lt;/p&gt;
&lt;p&gt;Our algorithm uses a field of the form &lt;span class="math"&gt;\(\mathbb{F}_p\)&lt;/span&gt;, that is the &lt;a href="https://en.wikipedia.org/wiki/Finite_field#Existence_and_uniqueness"&gt;unique&lt;/a&gt; field with &lt;span class="math"&gt;\(p\)&lt;/span&gt; (a prime) elements. The most straightforward construction of this field is arithmetic modulo &lt;span class="math"&gt;\(p\)&lt;/span&gt;. The other finite fields used in practise in ECC are of the form &lt;span class="math"&gt;\(\mathbb{F}_{2^m}\)&lt;/span&gt; and are sometimes called 'binary fields' (representible as polynomials with binary coefficients). Their field structure is also used in AES through byte substitution, implemented by inversion modulo &lt;span class="math"&gt;\(\mathbb{F}_{2^8}\)&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;From a performance perspective, great optimisations can be made by implementing efficient fixed-point arithmetic specialised to modulo by single prime constant, &lt;span class="math"&gt;\(p\)&lt;/span&gt;. From here on out, I'll be speaking from this abstraction layer alone.&lt;/p&gt;
&lt;h2&gt;Limbs&lt;/h2&gt;
&lt;p&gt;We wish to multiply two &lt;span class="math"&gt;\(m\)&lt;/span&gt;-bit numbers, each of which represented with &lt;span class="math"&gt;\(n\)&lt;/span&gt; 64-bit machine words in some way. Let's suppose just for now that &lt;span class="math"&gt;\(n\)&lt;/span&gt; divides &lt;span class="math"&gt;\(m\)&lt;/span&gt; neatly, then the quotient &lt;span class="math"&gt;\(d\)&lt;/span&gt; is the minimum number of bits in each machine word that will be required for representing our number. Suppose we use the straightforward representation whereby the least significant &lt;span class="math"&gt;\(d\)&lt;/span&gt; bits are used for storing parts of our number, which we better call &lt;span class="math"&gt;\(x\)&lt;/span&gt; because this is crypto and descriptive variable names are considered harmful (apparently).&lt;/p&gt;
&lt;div class="math"&gt;$$x = \sum_{k = 0}^{n-1} 2^{dk} l_k$$&lt;/div&gt;
&lt;p&gt;If we then drop the requirement for each of our &lt;span class="math"&gt;\(n\)&lt;/span&gt; machine words (also referred to as a 'limb' from hereon out) to have no more than the least significant &lt;span class="math"&gt;\(d\)&lt;/span&gt; bits populated, we say that such an implementation uses 'redundant limbs', meaning that the &lt;span class="math"&gt;\(k\)&lt;/span&gt;-th limb has high bits which overlap with the place values represented in the &lt;span class="math"&gt;\((k+1)\)&lt;/span&gt;-th limb.&lt;/p&gt;
&lt;h2&gt;Multiplication (mod p)&lt;/h2&gt;
&lt;p&gt;The fundamental difficulty with making modulo arithmetic fast is to do with the following property of multiplication.&lt;/p&gt;
&lt;p&gt;Let &lt;span class="math"&gt;\(a\)&lt;/span&gt; and &lt;span class="math"&gt;\(b\)&lt;/span&gt; be &lt;span class="math"&gt;\(m\)&lt;/span&gt;-bit numbers, then &lt;span class="math"&gt;\(0 \leq a &amp;lt; 2^m\)&lt;/span&gt; and &lt;span class="math"&gt;\(0 \leq b &amp;lt; 2^m\)&lt;/span&gt;, but critically we cannot say the same about &lt;span class="math"&gt;\(ab\)&lt;/span&gt;. Instead, the best we can say is that &lt;span class="math"&gt;\(0 \leq ab &amp;lt; 2^{2m}\)&lt;/span&gt;. Multiplication can in the worst case double the number of bits that must be stored, unless we can reduce modulo our prime.&lt;/p&gt;
&lt;p&gt;If we begin with non-redundant, 56-bit limbs, then for &lt;span class="math"&gt;\(a\)&lt;/span&gt; and &lt;span class="math"&gt;\(b\)&lt;/span&gt; not too much larger than &lt;span class="math"&gt;\(2^{384} &amp;gt; p_{384}\)&lt;/span&gt; that are 'reduced sufficiently' then we can multiply our limbs in the following ladder, so long as we are capable of storing the following sums without overflow.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="cm"&gt;/* and so on ... */&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;uint128_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="cm"&gt;/* ... and so forth */&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is possible, if we back each of the 56-bit limbs with a 64-bit machine word, with products being stored in 128-bit machine words. The numbers &lt;span class="math"&gt;\(a\)&lt;/span&gt; and &lt;span class="math"&gt;\(b\)&lt;/span&gt; were able to be stored with 7 limbs, whereas we use 13 limbs for storing the product. If &lt;span class="math"&gt;\(a\)&lt;/span&gt; and &lt;span class="math"&gt;\(b\)&lt;/span&gt; were stored non-redundantly, than each of the output (redundant) limbs must contain values less than &lt;span class="math"&gt;\(6 \cdot 2^{56} \cdot 2^{56} &amp;lt; 2^{115}\)&lt;/span&gt;, so there is no possibility of overflow in 128 bits. We even have room spare to do some additions/subtractions in cheap, redundant limb arithmetic.&lt;/p&gt;
&lt;p&gt;But we can't keep doing our sums in redundant limb arithmetic forever, we must eventually reduce. Doing so may be expensive, and so we would rather reduce only when strictly necessary!&lt;/p&gt;
&lt;h2&gt;Solinas-ish Reduction&lt;/h2&gt;
&lt;p&gt;Our prime is a &lt;em&gt;Solinas&lt;/em&gt; (&lt;em&gt;Pseudo/Generalised-Mersenne&lt;/em&gt;) &lt;em&gt;Prime&lt;/em&gt;. Mersenne Primes are primes expressible as &lt;span class="math"&gt;\(2^m - 1\)&lt;/span&gt;. This can be generalised to low-degree polynomials in &lt;span class="math"&gt;\(2^m\)&lt;/span&gt;. For example, another NIST curve uses &lt;span class="math"&gt;\(p_{224} = 2^{224} - 2^{96} + 1\)&lt;/span&gt; (a 224-bit number) where &lt;span class="math"&gt;\(p_{224} = f(2^{32})\)&lt;/span&gt; for &lt;span class="math"&gt;\(f(t) = t^7 - t^3 + 1\)&lt;/span&gt;. The simpler the choice of polynomial, the simpler the modular reduction logic.&lt;/p&gt;
&lt;p&gt;Our choice of &lt;span class="math"&gt;\(t\)&lt;/span&gt; is &lt;span class="math"&gt;\(2^{56}\)&lt;/span&gt;. &lt;a href="https://en.wikipedia.org/wiki/Solinas_prime#Modular_reduction_algorithm"&gt;Wikipedia&lt;/a&gt; the ideal case for Solinas reduction where the bitwidth of the prime is divisible by &lt;span class="math"&gt;\(\log_2{t}\)&lt;/span&gt;, but that is not our scenario. We choose 56-bits for some pretty simple realities of hardware. 56 is less than 64 (standard machine word size) but not by too much, and the difference is byte-addressible (&lt;span class="math"&gt;\(64-56=8\)&lt;/span&gt;). Let me explain:&lt;/p&gt;
&lt;h2&gt;Just the Right Amount of Reduction (mod p)&lt;/h2&gt;
&lt;p&gt;Let's first describe the actual prime that is our modulus.&lt;/p&gt;
&lt;div class="math"&gt;$$p_{384} = 2^{384} - 2^{128} - 2^{96} + 2^{32} - 1$$&lt;/div&gt;
&lt;p&gt;Yuck. This number is so yuck in fact, that noone has so far managed to upstream a Solinas' reduction method for it in OpenSSL, in spite of &lt;code&gt;secp384r1&lt;/code&gt; being the preferred curve for ECDH (Elliptic Curve Diffie-Hellman key exchange) and ECDSA (Elliptic Curve Digital Signature Algorithm) by NIST.&lt;/p&gt;
&lt;p&gt;In 56-bit limbs, we would express this number so:&lt;/p&gt;
&lt;p&gt;Let &lt;span class="math"&gt;\(f(t) = 2^{48} t^6 - 2^{16} t^2 - 2^{40} t + (2^{32} - 1)\)&lt;/span&gt;, then observe that all coefficients are smaller than &lt;span class="math"&gt;\(2^{56}\)&lt;/span&gt;, and that &lt;span class="math"&gt;\(p_{384} = f(2^{56})\)&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;Now let &lt;span class="math"&gt;\(\delta(t) = 2^{16} t^2 + 2^{40} t - 2^{32} + 1\)&lt;/span&gt;, consider that &lt;span class="math"&gt;\(p_{384} = 2^{384} - \delta(2^{56})\)&lt;/span&gt;, and thus &lt;span class="math"&gt;\(2^{384} \equiv \delta(2^{56}) \mod{p_{384}}\)&lt;/span&gt;. From now on let's call &lt;span class="math"&gt;\(\delta(2^{56})\)&lt;/span&gt; just &lt;span class="math"&gt;\(\delta\)&lt;/span&gt;. Thus, 'reduction' can be achieved as follows for suitable &lt;span class="math"&gt;\(X\)&lt;/span&gt; and &lt;span class="math"&gt;\(Y\)&lt;/span&gt;:&lt;/p&gt;
&lt;div class="math"&gt;$$ab = X + 2^{384} Y \equiv X + \delta Y \mod{p_{384}}$$&lt;/div&gt;
&lt;h3&gt;Calculating &lt;span class="math"&gt;\(\delta Y\)&lt;/span&gt;&lt;/h3&gt;
&lt;h4&gt;First Substitution&lt;/h4&gt;
&lt;p&gt;First make a choice of &lt;span class="math"&gt;\(X\)&lt;/span&gt; and &lt;span class="math"&gt;\(Y\)&lt;/span&gt;. The first thing to observe here is that this can actually be made a large number of ways! We choose:&lt;/p&gt;
&lt;div class="math"&gt;$$X_1 = \sum_{k=0}^8\texttt{in[k]} t^k$$&lt;/div&gt;
&lt;div class="math"&gt;$$Y_1 = 2^8 t^2 \sum_{k=9}^{12}\texttt{in[k]} t^{k-9} = 2^8 \sum_{k=9}^{12}\texttt{in[k]} t^{k-7}$$&lt;/div&gt;
&lt;p&gt;'Where does the &lt;span class="math"&gt;\(2^8 t^{2}\)&lt;/span&gt; come from?' I hear you ask. See &lt;span class="math"&gt;\(t^9 = t^2 \cdot t^7 = t^2 (2^8 \cdot 2^{384}) \equiv (2^8 t^2) \delta \mod{f(t)}\)&lt;/span&gt;. It's clear to see that the place value of &lt;code&gt;in[9] ... in[12]&lt;/code&gt; is greater than &lt;span class="math"&gt;\(2^{384}\)&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;I'm using the subscripts here because we're in fact going to do a series of these reductions to reach a suitably small answer. That's because our equation for reducing &lt;span class="math"&gt;\(t^7\)&lt;/span&gt; terms is as follows:&lt;/p&gt;
&lt;div class="math"&gt;$$t^7 \equiv 2^8\delta \equiv 2^{24} t^2 + 2^{48} t + (-2^{40} + 2^8) \mod{f(t)}$$&lt;/div&gt;
&lt;p&gt;Thus reducing &lt;code&gt;in[12]&lt;/code&gt; involves computing:&lt;/p&gt;
&lt;div class="math"&gt;$$\texttt{in[12]} t^{12} = \texttt{in[12]} (t^5)(t^7) \equiv 2^8\delta \cdot \texttt{in[12]} t^5 \mod{f(t)}$$&lt;/div&gt;
&lt;p&gt;But &lt;span class="math"&gt;\(\delta\)&lt;/span&gt; is a degree two polynomial, and so our numbers can still have two more limbs than we would want them to have. To be safe, let's store &lt;span class="math"&gt;\(X_1 + \delta Y_1\)&lt;/span&gt; in accumulator limbs &lt;code&gt;acc[0] ... acc[8]&lt;/code&gt; (this will at first appear to be one more limb than necessary), then we can eliminate &lt;code&gt;in[12]&lt;/code&gt; with the following logic.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="cm"&gt;/* assign accumulators to begin */&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="cm"&gt;/* X += 2^128 Y */&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xffffffff&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="cm"&gt;/* X += 2^96 Y */&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xff&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;48&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="cm"&gt;/* X += (-2^32 + 1) Y */&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xffff&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;40&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;48&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xffffffffffff&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Notice that for each term in &lt;span class="math"&gt;\(\delta = 2^{128} + 2^{96} + (2^{32} - 1)\)&lt;/span&gt; we do two additions/subtractions. This is in order to split up operands in order to minimise the final size of numbers and prevent over/underflows. Consequently, we need an &lt;code&gt;acc[8]&lt;/code&gt; to receive the high bits of our &lt;code&gt;in[12]&lt;/code&gt; substitution given above.&lt;/p&gt;
&lt;h4&gt;Second Substitution&lt;/h4&gt;
&lt;p&gt;Let's try and now eliminate through substitution &lt;code&gt;acc[7]&lt;/code&gt; and &lt;code&gt;acc[8]&lt;/code&gt;. Let&lt;/p&gt;
&lt;div class="math"&gt;$$X_2 = \sum^{6}_{k=0}\texttt{acc[k]}t^k $$&lt;/div&gt;
&lt;div class="math"&gt;$$Y_2 = 2^8(\texttt{acc[7]} t^7 + \texttt{acc[8]} t^8)$$&lt;/div&gt;
&lt;p&gt;But this time, &lt;span class="math"&gt;\(\delta Y_2\)&lt;/span&gt; is a number that comfortably can take up just five limbs, so we can update &lt;code&gt;acc[0], ..., acc[5]&lt;/code&gt; comfortably in-place.&lt;/p&gt;
&lt;h4&gt;Third Substitution&lt;/h4&gt;
&lt;p&gt;Finally, let's reduce all the high bits of &lt;code&gt;in[6]&lt;/code&gt;. Since &lt;code&gt;in[6]&lt;/code&gt; has place value &lt;span class="math"&gt;\(t^6 = 2^{336}\)&lt;/span&gt;, thus we wish to reduce all but the least significant &lt;span class="math"&gt;\(384 - 336 = 48\)&lt;/span&gt; bits.&lt;/p&gt;
&lt;p&gt;A goal in designing this algorithm is to ensure that &lt;code&gt;acc[6]&lt;/code&gt; has as tight a bound as reasonably possible. Intuitively, if we can cause &lt;code&gt;acc[6]&lt;/code&gt; to be as large as possible by absorbing the high bits of lower limbs, we reduce the number of bits that must be carried forward later on. As such, we perform a carry of the high-bits of &lt;code&gt;acc[4]&lt;/code&gt;, &lt;code&gt;acc[5]&lt;/code&gt; into &lt;code&gt;acc[6]&lt;/code&gt; before we begin our substitution.&lt;/p&gt;
&lt;p&gt;Again, let&lt;/p&gt;
&lt;div class="math"&gt;$$X_3 = \sum^{5}_{k=0}\texttt{acc[k]}t^k + (\texttt{acc[6]} \text{(low bits)})t^6$$&lt;/div&gt;
&lt;div class="math"&gt;$$Y_3 = 2^{48}(\texttt{acc[6]} \text{(high bits, right shifted)}) t^6$$&lt;/div&gt;
&lt;p&gt;The equation for eliminating &lt;span class="math"&gt;\(2^{48}t^6\)&lt;/span&gt; is pretty straightforward:&lt;/p&gt;
&lt;div class="math"&gt;$$2^{384} = 2^{48}t^6 \equiv 2^{16}t^2 + 2^{40}t + (-2^{32} + 1) \mod{f(t)}$$&lt;/div&gt;
&lt;h4&gt;Carries&lt;/h4&gt;
&lt;p&gt;Finally, as each of &lt;code&gt;acc[0], ..., acc[6]&lt;/code&gt; can contain values larger than &lt;span class="math"&gt;\(2^{56}\)&lt;/span&gt;, we carry their respective high bits into &lt;code&gt;acc[6]&lt;/code&gt; so as to remove any redundancy. Conveniently, our preemptive carrying before the third substitution has granted us a pretty tight bound on our final calculation - the final reduced number has the range &lt;span class="math"&gt;\([0, 2^{384}]\)&lt;/span&gt;.&lt;/p&gt;
&lt;h4&gt;Canonicalisation&lt;/h4&gt;
&lt;p&gt;This is 'just the right amount of reduction' but not &lt;em&gt;canonicalisation&lt;/em&gt;. That is, since &lt;span class="math"&gt;\(0 &amp;lt; p_{384} &amp;lt; 2^{384}\)&lt;/span&gt;, there can be multiple possible reduced values for a given congruence class. &lt;code&gt;felem_contract&lt;/code&gt; is a method which uses the fact that &lt;span class="math"&gt;\(0  \leq x &amp;lt; 2 p_{384}\)&lt;/span&gt; to further reduce the output of &lt;code&gt;felem_reduce&lt;/code&gt; into the range &lt;span class="math"&gt;\([0, p_{384})\)&lt;/span&gt; in constant time.&lt;/p&gt;
&lt;p&gt;This code has many more dragons I won't explain here, but the basic premise to the calculations performed there is as follows:&lt;/p&gt;
&lt;p&gt;Given a 385 bit input, checking whether our input (expressed as a concatenation of bits) &lt;span class="math"&gt;\(b_{384}b_{383} \ldots b_1b_0\)&lt;/span&gt; is greater than or equal to &lt;span class="math"&gt;\(p_{384}\)&lt;/span&gt; whose bits we denote &lt;span class="math"&gt;\(q_{384}, \ldots, q_0\)&lt;/span&gt; (&lt;span class="math"&gt;\(q_{384} = 0\)&lt;/span&gt;) is determined by the following logical predicate (&lt;span class="math"&gt;\(G(384)\)&lt;/span&gt;):&lt;/p&gt;
&lt;div class="math"&gt;$$G(k) \equiv (b_k \land \lnot q_k) \lor ((b_k = q_k) \land G(k-1))$$&lt;/div&gt;
&lt;div class="math"&gt;$$G(0) \equiv b_k = q_k$$&lt;/div&gt;
&lt;p&gt;With &lt;span class="math"&gt;\(p_{384}\)&lt;/span&gt; being a Solinas'/Pseudo-Mersenne Prime, it has a large number of contiguous runs of repeated bits, so we can of course use this to massively simplify our predicate. Doing this in constant time involves some interesting bit-shifting/masking schenanigans. Essentially, you want a bit vector of all ones/zeros depending on the value of &lt;span class="math"&gt;\(G(384)\)&lt;/span&gt;, we then logically 'and' with this bitmask to 'conditionally' subtract &lt;span class="math"&gt;\(p_{384}\)&lt;/span&gt; from our result.&lt;/p&gt;
&lt;h3&gt;A Side Note about the Weird Constants&lt;/h3&gt;
&lt;p&gt;Okay so we're implementing our modular arithmetic with unsigned integer limbs that together represent a number of the following form:&lt;/p&gt;
&lt;div class="math"&gt;$$x = \sum_{k = 0}^{n-1} 2^{dk} l_k$$&lt;/div&gt;
&lt;p&gt;How do we then do subtractions in a way which will make overflow impossible? Well computing &lt;span class="math"&gt;\(a - b\)&lt;/span&gt; is really straightforward if every limb of &lt;span class="math"&gt;\(a\)&lt;/span&gt; is larger than every limb of &lt;span class="math"&gt;\(b\)&lt;/span&gt;. We then add a suitable multiple of &lt;span class="math"&gt;\(p_{384}\)&lt;/span&gt; to &lt;span class="math"&gt;\(a\)&lt;/span&gt; that causes each limb of &lt;span class="math"&gt;\(a\)&lt;/span&gt; to be sufficiently large.&lt;/p&gt;
&lt;p&gt;Thankfully, with redundant-limb arithmetic, we can do this easily by means of &lt;em&gt;telescopic sums&lt;/em&gt;. For example, in &lt;code&gt;felem_reduce&lt;/code&gt; we wanted all limbs of our &lt;span class="math"&gt;\(p_{384}\)&lt;/span&gt; multiple to be sufficiently large. We overshot any requirement and provided such a multiple which gives a lower bound &lt;span class="math"&gt;\(2^{123}\)&lt;/span&gt;. We first scale our prime accordingly so that its 'lead term' (speaking in the polynomial representation) is &lt;span class="math"&gt;\(2^{124}\)&lt;/span&gt;.&lt;/p&gt;
&lt;div class="math"&gt;$$2^{76} f(t) = 2^{124} t^6 - 2^{92} t^2 - 2^{116} t + (2^{108} - 2^{76}) t^0$$&lt;/div&gt;
&lt;p&gt;Notice that most limbs of this multiple (the limbs will be the coefficients) are either too small or negative. We then transform this expression into a suitable telescopic sum. Observe that when &lt;span class="math"&gt;\(t = 2^{56}\)&lt;/span&gt;, &lt;span class="math"&gt;\(2^{124} t^k = 2^{124-56}t^{k+1} = 2^{68} t^{k+1}\)&lt;/span&gt;, and so simply introduce into each limb where required a &lt;span class="math"&gt;\(2^{124}\)&lt;/span&gt; term by means of addition, subtracting the same number from a higher limb.&lt;/p&gt;
&lt;div class="math"&gt;$$
\begin{align*}
2^{76} f(t) &amp;amp;= (2^{124} - 2^{68}) t^6 \\
            &amp;amp;+ (2^{124} - 2^{68}) t^5 \\
            &amp;amp;+ (2^{124} - 2^{68}) t^4 \\
            &amp;amp;+ (2^{124} - 2^{68}) t^3 \\
            &amp;amp;+ (2^{124} - 2^{92} - 2^{68}) t^2 \\
            &amp;amp;+ (2^{124} - 2^{116} - 2^{68}) t \\
            &amp;amp;+ (2^{124} + 2^{108} - 2^{76})
\end{align*}
$$&lt;/div&gt;
&lt;p&gt;We can then subtract values whose limbs are no larger than the least of these limbs above without fear of underflows providing us with an incorrect result. In our case, that upper bound for limb value is &lt;span class="math"&gt;\(2^{124} - 2^{116} - 2^{68} &amp;gt; 2^{123}\)&lt;/span&gt;. Very comfortable.&lt;/p&gt;
&lt;h2&gt;Concerning Timing Side-Channels&lt;/h2&gt;
&lt;p&gt;Cryptographic routines must perform all of their calculations in constant time. More specifically, it is important that timing cryptography code should not reveal any private keys or random nonces used during computation. Ultimately, all of our work so far has been to speed up field arithmetic in the modulo field with prime &lt;span class="math"&gt;\(p_{384}\)&lt;/span&gt;. But this is done in order to facilitate calculations in the secp384r1 elliptic curve, and ECDSA/ECDH each depend on being able to perform scalar 'point multiplication' (repeat application of the group operator). Since such an operation is inherently iterative, it presents the greatest potential for timing attacks.&lt;/p&gt;
&lt;p&gt;We implement constant-time multiplication with the &lt;em&gt;wNAF&lt;/em&gt; ladder method. This relies on pre-computing a window of multiples of the group generator, and then scaling and selectively adding multiples when required. &lt;a href="https://en.wikipedia.org/wiki/Elliptic_curve_point_multiplication#Point_multiplication"&gt;Wikipedia&lt;/a&gt; provides a helpful primer to this method  by cumulatively building upon more naive approaches.&lt;/p&gt;
&lt;h2&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;While the resulting code borrows from and uses common language of Solinas reduction, ultimately there are a number of implementation decisions that were guided by heuristic - going from theory to implementation was far from cut-and-dry. The limb size, carry order, choice of substitutions as well as pre and post conditions made here are ultimately arbitrary. You could easily imagine there being further refinements obtaining a better result. For now, I hope this post serves to demystify the inner workings of ECC implementations in OpenSSL. These algorithms, although particular and sophisticated, need not be immutable.&lt;/p&gt;
&lt;script type="text/javascript"&gt;if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
    var align = "center",
        indent = "0em",
        linebreak = "false";

    if (false) {
        align = (screen.width &lt; 768) ? "left" : align;
        indent = (screen.width &lt; 768) ? "0em" : indent;
        linebreak = (screen.width &lt; 768) ? 'true' : linebreak;
    }

    var mathjaxscript = document.createElement('script');
    mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
    mathjaxscript.type = 'text/javascript';
    mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';

    var configscript = document.createElement('script');
    configscript.type = 'text/x-mathjax-config';
    configscript[(window.opera ? "innerHTML" : "text")] =
        "MathJax.Hub.Config({" +
        "    config: ['MMLorHTML.js']," +
        "    TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
        "    jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
        "    extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
        "    displayAlign: '"+ align +"'," +
        "    displayIndent: '"+ indent +"'," +
        "    showMathMenu: true," +
        "    messageStyle: 'normal'," +
        "    tex2jax: { " +
        "        inlineMath: [ ['\\\\(','\\\\)'] ], " +
        "        displayMath: [ ['$$','$$'] ]," +
        "        processEscapes: true," +
        "        preview: 'TeX'," +
        "    }, " +
        "    'HTML-CSS': { " +
        "        availableFonts: ['STIX', 'TeX']," +
        "        preferredFont: 'STIX'," +
        "        styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
        "        linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
        "    }, " +
        "}); " +
        "if ('default' !== 'default') {" +
            "MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
                "var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
                "VARIANT['normal'].fonts.unshift('MathJax_default');" +
                "VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
                "VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
                "VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
            "});" +
            "MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
                "var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
                "VARIANT['normal'].fonts.unshift('MathJax_default');" +
                "VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
                "VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
                "VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
            "});" +
        "}";

    (document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
    (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
&lt;/script&gt;&lt;script type='text/javascript'&gt;if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
    var align = "center",
        indent = "0em",
        linebreak = "false";

    if (false) {
        align = (screen.width &lt; 768) ? "left" : align;
        indent = (screen.width &lt; 768) ? "0em" : indent;
        linebreak = (screen.width &lt; 768) ? 'true' : linebreak;
    }

    var mathjaxscript = document.createElement('script');
    mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
    mathjaxscript.type = 'text/javascript';
    mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';

    var configscript = document.createElement('script');
    configscript.type = 'text/x-mathjax-config';
    configscript[(window.opera ? "innerHTML" : "text")] =
        "MathJax.Hub.Config({" +
        "    config: ['MMLorHTML.js']," +
        "    TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
        "    jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
        "    extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
        "    displayAlign: '"+ align +"'," +
        "    displayIndent: '"+ indent +"'," +
        "    showMathMenu: true," +
        "    messageStyle: 'normal'," +
        "    tex2jax: { " +
        "        inlineMath: [ ['\\\\(','\\\\)'] ], " +
        "        displayMath: [ ['$$','$$'] ]," +
        "        processEscapes: true," +
        "        preview: 'TeX'," +
        "    }, " +
        "    'HTML-CSS': { " +
        "        availableFonts: ['STIX', 'TeX']," +
        "        preferredFont: 'STIX'," +
        "        styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
        "        linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
        "    }, " +
        "}); " +
        "if ('default' !== 'default') {" +
            "MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
                "var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
                "VARIANT['normal'].fonts.unshift('MathJax_default');" +
                "VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
                "VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
                "VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
            "});" +
            "MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
                "var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
                "VARIANT['normal'].fonts.unshift('MathJax_default');" +
                "VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
                "VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
                "VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
            "});" +
        "}";

    (document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
    (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
&lt;/script&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Rohan McLure</dc:creator><pubDate>Mon, 07 Aug 2023 12:00:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2023-08-07:/blog/2023/08/07/going-out-on-a-limb-efficient-elliptic-curve-arithmetic-in-openssl/</guid><category>Cryptography</category></item><item><title>Quirks of parsing SSH configs</title><link>https://sthbrx.github.io/blog/2023/08/04/quirks-of-parsing-ssh-configs/</link><description>&lt;h2&gt;Introduction&lt;/h2&gt;
&lt;p&gt;I've been using the VSCodium 
&lt;a href="https://open-vsx.org/extension/jeanp413/open-remote-ssh"&gt;Open Remote - SSH&lt;/a&gt;
extension recently to great results. I can treat everything as a single
environment, without any worry about syncing between my local development files
and the remote. This is very different to mounting the remote as a network drive
and opening a local instance of VSCodium on it: in addition to crippling latency
on every action, a locally mounted drive doesn't bring the build context that
tools like &lt;code&gt;clangd&lt;/code&gt; require (e.g., system headers).&lt;/p&gt;
&lt;p&gt;Instead, the remote extension runs a server on the remote that performs most
actions, and the local VSCodium instance acts as a client that buffers and
caches data seamlessly, so the experience is nearly as good as developing
locally. &lt;/p&gt;
&lt;p&gt;For example, a project wide file search on a network drive is unusably slow
because every file and directory read requires a round trip back to the remote,
and the latency is just too large to finish getting results back in a reasonable
time. But with the client-server approach, the client just sends the search
request to the server for it to fulfil, and all the server has to do is send the
matches back. This eliminates nearly all the latency effects, except for the
initial request and receiving any results.&lt;/p&gt;
&lt;p&gt;However there has been one issue with using this for everything: the extension
failed to connect when I wasn't on the same network as the host machine. So I
wasn't able to use it when working from home over a VPN. In this post we find
out why this happened, and in the process look at some of the weird quirks of
parsing an SSH config.&lt;/p&gt;
&lt;h2&gt;The issue&lt;/h2&gt;
&lt;p&gt;As above, I wasn't able to connect to my remote machines when working from home.
The extension would abort with the following error:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[Error  - 00:23:10.592] Error resolving authority
Error: getaddrinfo ENOTFOUND remotename.ozlabs.ibm.com
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:109:26)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;So it's a DNS issue. This would make sense, as the remote machine is not exposed
to the internet, and must instead be accessed through a proxy. What's weird is
that the integrated terminal in VSCodium has no problem connecting to the
remote. So the extension seems to be doing something different than just a plain
SSH connection.&lt;/p&gt;
&lt;p&gt;You might think that the extension is not reading the SSH config. But the
extension panel lists all the host aliases I've declared in the config, so it's
clearly aware of the config at least. Possibly it doesn't understand the proxy
config correctly? If it was trying to connect directly from the host, it would
make sense to fail a DNS lookup.&lt;/p&gt;
&lt;h2&gt;Investigating&lt;/h2&gt;
&lt;p&gt;Enough theorising, time to debug the extension as it tries to connect.&lt;/p&gt;
&lt;p&gt;From the error above, the string &lt;code&gt;"Error resolving authority"&lt;/code&gt; looks like
something I can search for. This takes me to the
&lt;a href="https://github.com/jeanp413/open-remote-ssh/blob/521098e24f48b4b9e04d476895f9097b03f8c984/src/authResolver.ts#L226"&gt;&lt;code&gt;catch&lt;/code&gt; case for a large try-catch block&lt;/a&gt;.
It could be annoying to narrow down which part of the block
throws the exception, but fortunately debugging is as easy as installing the
dependencies and running the pre-configured 'Extension' debug target. This opens
a new window with the local copy of the extension active, and I can debug it in
the original window.&lt;/p&gt;
&lt;p&gt;In this block, there is a conditional statement on whether the &lt;code&gt;ProxyJump&lt;/code&gt; field
is present in the config. This is a good place to break on and see what the
computed config looks like. If it doesn't find a proxy then of course it's going
to run everything on the host.&lt;/p&gt;
&lt;p&gt;And indeed, it doesn't think there is a proxy. This is progress, but why does
the extension's view of the config not match up with what SSH does? After all,
invoking SSH directly connects properly. Tracing back the source of the config
in the extension, it ultimately comes from manually reading in and parsing the
SSH config. When resolving the host argument it manually computes the config as
per &lt;a href="https://man7.org/linux/man-pages/man5/ssh_config.5.html"&gt;&lt;code&gt;ssh_config(5)&lt;/code&gt;&lt;/a&gt;.
Yet somewhere it makes a mistake, because it doesn't include the &lt;code&gt;ProxyJump&lt;/code&gt;
field.&lt;/p&gt;
&lt;h2&gt;Parsing SSH config&lt;/h2&gt;
&lt;p&gt;To get to the bottom of this, we need to know the rules behind parsing SSH
configs. The &lt;code&gt;ssh_config(5)&lt;/code&gt; manpage does a pretty decent job of explaining
this, but I'm going to go over the relevant information here. I reckon most
people have a vague idea of how it works, and can write enough to meet their
needs, but have never looked deeper into the actual rules behind how SSH parses
the config.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;For starters, the config is parsed line by line. Leading whitespace (i.e.,
   indentation) is ignored. So, while indentation makes it look like you are
   configuring properties for a particular host, this isn't quite correct.
   Instead, the &lt;code&gt;Host&lt;/code&gt; and &lt;code&gt;Match&lt;/code&gt; lines are special statements that enable or
   disable all subsequent lines until the next &lt;code&gt;Host&lt;/code&gt; or &lt;code&gt;Match&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;There is no backtracking; previous conditions and lines are not re-evaluated
after learning more about the config later on.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;When a config line is seen, and is active thanks to the most recent &lt;code&gt;Host&lt;/code&gt; or
   &lt;code&gt;Match&lt;/code&gt; succeeding, its value is selected if it is the first of that config
   to be selected. So the earliest place a value is set takes priority; this may
   be a little counterintuitive if you are used to having the latest value be
   picked, like enable/disable command line flags tend to work.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;When &lt;code&gt;HostName&lt;/code&gt; is set, it replaces the &lt;code&gt;host&lt;/code&gt; value in &lt;code&gt;Match&lt;/code&gt; matches. It
   is also used as the &lt;code&gt;Host&lt;/code&gt; value during a final pass (if requested).&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;The last behaviour of interest is the &lt;code&gt;Match final&lt;/code&gt; rule. There are several
   conditions a &lt;code&gt;Match&lt;/code&gt; statement can have, and the &lt;code&gt;final&lt;/code&gt; rule says make this
   active on the final pass over the config.&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Wait, final pass? Multiple passes? Yes. If &lt;code&gt;final&lt;/code&gt; is a condition on a &lt;code&gt;Match&lt;/code&gt;,
SSH will do another pass over the entire config, following all the rules above.
Except this time all the configs we read on the first pass are still active (and
can't be changed). But all the &lt;code&gt;Host&lt;/code&gt; and &lt;code&gt;Matches&lt;/code&gt; are re-evaluated, allowing
other configs to potentially be set. I guess that means rule (1) ought to have a
big asterisk next to it.&lt;/p&gt;
&lt;p&gt;Together, these rules can lead to some quirky behaviours. Consider the following
config&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Match host=&amp;quot;*.ozlabs.ibm.com&amp;quot;
    ProxyJump proxy

Host example
    HostName example.ozlabs.ibm.com
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If I run &lt;code&gt;ssh example&lt;/code&gt; on the command line, will it use the proxy?&lt;/p&gt;
&lt;p&gt;By rule (1), no. When testing the first &lt;code&gt;Match host&lt;/code&gt; condition, our host value
is currently &lt;code&gt;example&lt;/code&gt;. It is not until we reach the &lt;code&gt;HostName&lt;/code&gt; config that we
start using &lt;code&gt;example.ozlabs.ibm.com&lt;/code&gt; for these matches.&lt;/p&gt;
&lt;p&gt;But by rule (4), the answer turns into &lt;em&gt;maybe&lt;/em&gt;. If we end up doing a second pass
over the config thanks to a &lt;code&gt;Match final&lt;/code&gt; that could be &lt;em&gt;anywhere&lt;/em&gt; else, we
would now be matching &lt;code&gt;example.ozlabs.ibm.com&lt;/code&gt; against the first line on the
second go around. This will pass, and, since nothing has set &lt;code&gt;ProxyJump&lt;/code&gt; yet, we
would gain the proxy.&lt;/p&gt;
&lt;p&gt;You may think, yes, but we don't have a &lt;code&gt;Match final&lt;/code&gt; in that example. But if
you thought that, then you forgot about the system config.&lt;/p&gt;
&lt;p&gt;The system config is effectively appended to the user config, to allow any
system wide settings. Most of the time this isn't an issue because of the
first-come-first-served rule with config matches (rule 2). But if the system
config includes a &lt;code&gt;Match final&lt;/code&gt;, it will trigger the entire config to be
re-parsed, including the user section. And it so happens that, at least on
Fedora with the &lt;code&gt;openssh-clients&lt;/code&gt; package installed, the system config does
contain a &lt;code&gt;Match final&lt;/code&gt; (see &lt;code&gt;/etc/ssh/ssh_config.d&lt;/code&gt;).&lt;/p&gt;
&lt;p&gt;But wait, there's more! If we want to specify a custom SSH config file, then we
can use &lt;code&gt;-F path/to/config&lt;/code&gt; in the command line. But this disables loading a
system config, so we would no longer get the proxy!&lt;/p&gt;
&lt;p&gt;To sum up, for the above config:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;ssh example&lt;/code&gt; doesn't have a proxy&lt;/li&gt;
&lt;li&gt;...unless a system config contains &lt;code&gt;Match final&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;...but invoking it as &lt;code&gt;ssh -F ~/.ssh/config example&lt;/code&gt; definitely won't have
   the proxy&lt;/li&gt;
&lt;li&gt;...but if a subprocess invokes &lt;code&gt;ssh example&lt;/code&gt; while trying to resolve another
   host, it'll probably not add the &lt;code&gt;-F ~/.ssh/config&lt;/code&gt;, so we might get the
   proxy again (in the child process).&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Wait, how did that last one slip in? Well, unlike environment variables, it's a
lot harder for processes to propagate command line flags correctly. If resolving
the config involves running a script that itself tries to run SSH, chances are
the &lt;code&gt;-F&lt;/code&gt; flag won't be propagated and you'll see some weird behaviour.&lt;/p&gt;
&lt;p&gt;I swear that's all for now, you've probably learned more about SSH configs than
you will ever need to care about.&lt;/p&gt;
&lt;h2&gt;Back to VSCodium&lt;/h2&gt;
&lt;p&gt;Alright, armed now with this knowledge on SSH config parsing, we can work out
what's going on with the extension. It ends up being a simple issue: it doesn't
apply rules (3) and (4), so all &lt;code&gt;Host&lt;/code&gt; matches are done against the original
host name.&lt;/p&gt;
&lt;p&gt;In my case, there are several machines behind the proxy, but they all share a
common suffix, so I had a &lt;code&gt;Host *.ozlabs.ibm.com&lt;/code&gt; rule to apply the proxy. I
also use aliases to refer to the machines without the &lt;code&gt;.ozlabs.ibm.com&lt;/code&gt; suffix,
so failing to follow rule (3) lead to the situation where the extension didn't
think there was a proxy.&lt;/p&gt;
&lt;p&gt;However, even if this were to be fixed, it still doesn't respect rule (4), or
most complex match logic in general. If the hostname bug is fixed then my setup
would work, but it's less than ideal to keep playing whack-a-mole with parsing
bugs. It would be a lot easier if there was a way to just ask SSH for the config
that a given host name resolves to.&lt;/p&gt;
&lt;p&gt;Enter &lt;code&gt;ssh -G&lt;/code&gt;. The &lt;code&gt;-G&lt;/code&gt; flag asks SSH to dump the complete resolved config,
without actually opening the connection (it may execute arbitrary code while
resolving the config however!). So to fix the extension once and for all, we
could swap the manual parser to just invoking &lt;code&gt;ssh -G example&lt;/code&gt;, and parsing the
output as the final config. No &lt;code&gt;Host&lt;/code&gt; or &lt;code&gt;Match&lt;/code&gt; or &lt;code&gt;HostName&lt;/code&gt; or &lt;code&gt;Match final&lt;/code&gt;
quirks to worry about.&lt;/p&gt;
&lt;p&gt;Sure enough, if we replace the config backend with this 'native' resolver, we
can connect to all the machines with no problem. Hopefully the
&lt;a href="https://github.com/jeanp413/open-remote-ssh/pull/103"&gt;pull request&lt;/a&gt; to add this
support will get accepted, and I can stop running my locally patched copy of the
extension.&lt;/p&gt;
&lt;p&gt;In general, I'd suggest avoiding any dependency on a second pass being done on
the config. Resolve your aliases early, so that the rest of your matches work
against the full hostname. If you later need to match against the name passed in
the command line, you can use &lt;code&gt;Match originalhost=example&lt;/code&gt;. The example above
should always be written as&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Host example
    HostName example.ozlabs.ibm.com

Match host=&amp;quot;*.ozlabs.ibm.com&amp;quot;
    ProxyJump proxy
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;even if the reversed order might appear to work thanks to the weird interactions
described above. And after learning these parser quirks, I find the idea of
using &lt;code&gt;Host&lt;/code&gt; match statements unreliable; that they may or may not be run
against the &lt;code&gt;HostName&lt;/code&gt; value allows for truely strange bugs to appear. Maybe you
should remove this uncertainty by starting your config with &lt;code&gt;Match final&lt;/code&gt; to at
least always be parsed the same way.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Benjamin Gray</dc:creator><pubDate>Fri, 04 Aug 2023 18:00:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2023-08-04:/blog/2023/08/04/quirks-of-parsing-ssh-configs/</guid><category>Development</category><category>ssh</category></item><item><title>Detecting rootless Docker</title><link>https://sthbrx.github.io/blog/2023/04/05/detecting-rootless-docker/</link><description>&lt;h2&gt;Trying to do some fuzzing...&lt;/h2&gt;
&lt;p&gt;The other day, for the first time in a while, I wanted to do something with &lt;a href="https://github.com/google/syzkaller"&gt;syzkaller&lt;/a&gt;, a system call fuzzer that has been used to find literally thousands of kernel bugs. As it turns out, since the last time I had done any work on syzkaller, I switched to a new laptop, and so I needed to set up a few things in my development environment again.&lt;/p&gt;
&lt;p&gt;While I was doing this, I took a look at the syzkaller source again and found a neat little script called &lt;a href="https://github.com/google/syzkaller/blob/master/tools/syz-env"&gt;&lt;code&gt;syz-env&lt;/code&gt;&lt;/a&gt;, which uses a Docker image to provide you with a standardised environment that has all the necessary tools and dependencies preinstalled.&lt;/p&gt;
&lt;p&gt;I decided to give it a go, and then realised I hadn't actually installed Docker since getting my new laptop. So I went to do that, and along the way I discovered &lt;a href="https://docs.docker.com/engine/security/rootless/"&gt;rootless mode&lt;/a&gt;, and decided to give it a try.&lt;/p&gt;
&lt;h2&gt;What's rootless mode?&lt;/h2&gt;
&lt;p&gt;As of relatively recently, Docker supports rootless mode, which allows you to run your &lt;code&gt;dockerd&lt;/code&gt; as a non-root user. This is helpful for security, as traditional "rootful" Docker can trivially be used to obtain root privileges outside of a container. Rootless Docker is implemented using &lt;a href="https://github.com/rootless-containers/rootlesskit"&gt;RootlessKit&lt;/a&gt; (a fancy replacement for &lt;a href="https://wiki.debian.org/FakeRoot"&gt;fakeroot&lt;/a&gt; that uses user namespaces) to create a new user namespace that maps the UID of the user running &lt;code&gt;dockerd&lt;/code&gt; to 0.&lt;/p&gt;
&lt;p&gt;You can find more information, including details of the various restrictions that apply to rootless setups, &lt;a href="https://docs.docker.com/engine/security/rootless/"&gt;in the Docker documentation&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;The problem&lt;/h2&gt;
&lt;p&gt;I ran &lt;code&gt;tools/syz-env make&lt;/code&gt; to test things out. It pulled the container image, then gave me some strange errors:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ajd@jarvis-debian:~/syzkaller$ tools/syz-env make NCORES=1
gcr.io/syzkaller/env:latest
warning: Not a git repository. Use --no-index to compare two paths outside a working tree
usage: git diff --no-index [&amp;lt;options&amp;gt;] &amp;lt;path&amp;gt; &amp;lt;path&amp;gt;

    ...

fatal: detected dubious ownership in repository at &amp;#39;/syzkaller/gopath/src/github.com/google/syzkaller&amp;#39;
To add an exception for this directory, call:

        git config --global --add safe.directory /syzkaller/gopath/src/github.com/google/syzkaller
fatal: detected dubious ownership in repository at &amp;#39;/syzkaller/gopath/src/github.com/google/syzkaller&amp;#39;
To add an exception for this directory, call:

        git config --global --add safe.directory /syzkaller/gopath/src/github.com/google/syzkaller
go list -f &amp;#39;{{.Stale}}&amp;#39; ./sys/syz-sysgen | grep -q false || go install ./sys/syz-sysgen
error obtaining VCS status: exit status 128
        Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
        Use -buildvcs=false to disable VCS stamping.
make: *** [Makefile:155: descriptions] Error 1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After a bit of digging, I found that &lt;code&gt;syz-env&lt;/code&gt; mounts the syzkaller source directory inside the container as a volume. &lt;code&gt;make&lt;/code&gt; was running with UID 1000, while the files in the mounted volume appeared to be owned by root.&lt;/p&gt;
&lt;p&gt;Reading the script, it turns out that &lt;code&gt;syz-env&lt;/code&gt; invokes &lt;code&gt;docker run&lt;/code&gt; with the &lt;code&gt;--user&lt;/code&gt; option to set the UID inside the container to match the user's UID outside the container, to ensure that file ownership and permissions behave as expected.&lt;/p&gt;
&lt;p&gt;This works in rootful Docker, where files appear inside the container to be owned by the same UID as they are outside the container. However, it breaks in rootless mode: due to the way RootlessKit sets up the namespaces, the user's UID is mapped to 0, causing the files to appear to be owned by root.&lt;/p&gt;
&lt;p&gt;The workaround seemed pretty obvious: just skip the &lt;code&gt;--user&lt;/code&gt; flag if running rootless.&lt;/p&gt;
&lt;h2&gt;How can you check whether your Docker daemon is running in rootless mode?&lt;/h2&gt;
&lt;p&gt;It took me quite a while, as a total Docker non-expert, to figure out how to definitively check whether the Docker daemon is running rootless or not. There's a variety of ways you could do this, such as checking the name of the current Docker context to see if it's called &lt;code&gt;rootless&lt;/code&gt; (as used by the Docker rootless setup scripts), but I think the approach I settled on is the most correct one.&lt;/p&gt;
&lt;p&gt;If you want to check whether your Docker daemon is running in rootless mode, use &lt;code&gt;docker info&lt;/code&gt; to query the daemon's security options, and check for the &lt;code&gt;rootless&lt;/code&gt; option.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;docker info -f &amp;quot;{{println .SecurityOptions}}&amp;quot; | grep rootless
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If this prints something like:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[name=seccomp,profile=builtin name=rootless name=cgroupns]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;then you're running rootless.&lt;/p&gt;
&lt;p&gt;If not, then you're running the traditional rootful.&lt;/p&gt;
&lt;p&gt;Easy! (And I sent a fix which is now &lt;a href="https://github.com/google/syzkaller/commit/340a1b9094e4b3fad232c98c62de653ec48954ab"&gt;merged into syzkaller!&lt;/a&gt;)&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Donnellan</dc:creator><pubDate>Wed, 05 Apr 2023 13:00:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2023-04-05:/blog/2023/04/05/detecting-rootless-docker/</guid><category>Development</category><category>Docker</category><category>syzkaller</category></item><item><title>Dumb bugs: the PCI device that wasn't</title><link>https://sthbrx.github.io/blog/2023/04/04/dumb-bugs-the-pci-device-that-wasnt/</link><description>&lt;p&gt;I was happily minding my own business one fateful afternoon when I received the following kernel bug report:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;BUG: KASAN: slab-out-of-bounds in vga_arbiter_add_pci_device+0x60/0xe00
Read of size 4 at addr c000000264c26fdc by task swapper/0/1

Call Trace:
dump_stack_lvl+0x1bc/0x2b8 (unreliable)
print_report+0x3f4/0xc60
kasan_report+0x244/0x698
__asan_load4+0xe8/0x250
vga_arbiter_add_pci_device+0x60/0xe00
pci_notify+0x88/0x444
notifier_call_chain+0x104/0x320
blocking_notifier_call_chain+0xa0/0x140
device_add+0xac8/0x1d30
device_register+0x58/0x80
vio_register_device_node+0x9ac/0xce0
vio_bus_scan_register_devices+0xc4/0x13c
__machine_initcall_pseries_vio_device_init+0x94/0xf0
do_one_initcall+0x12c/0xaa8
kernel_init_freeable+0xa48/0xba8
kernel_init+0x64/0x400
ret_from_kernel_thread+0x5c/0x64
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;OK, so &lt;a href="https://www.kernel.org/doc/html/latest/dev-tools/kasan.html"&gt;KASAN&lt;/a&gt; has helpfully found an out-of-bounds access in &lt;code&gt;vga_arbiter_add_pci_device()&lt;/code&gt;.  What the heck is that?&lt;/p&gt;
&lt;h2&gt;Why does my VGA require arbitration?&lt;/h2&gt;
&lt;p&gt;I'd never heard of the &lt;a href="https://en.wikipedia.org/wiki/VGA_connector"&gt;VGA&lt;/a&gt; arbiter in the kernel (do kids these days know what VGA is?), or &lt;code&gt;vgaarb&lt;/code&gt; as it's called.  What it does is irrelevant to this bug, but I found the history pretty interesting!  &lt;a href="https://lists.freedesktop.org/archives/xorg/2005-March/006663.html"&gt;Benjamin Herrenschmidt proposed VGA arbitration back in 2005&lt;/a&gt; as a way of resolving conflicts between multiple legacy VGA devices that want to use the same address assignments.  This was previously handled in userspace by the X server, but issues arose with multiple X servers on the same machine.  Plus, it's probably not a good idea for this kind of thing to be handled by userspace.  &lt;a href="https://docs.kernel.org/gpu/vgaarbiter.html"&gt;You can read more about the VGA arbiter in the kernel docs&lt;/a&gt;, but it's probably not something anyone has thought much about in a long time.&lt;/p&gt;
&lt;h2&gt;The bad access&lt;/h2&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;bool&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;vga_arbiter_add_pci_device&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;pci_dev&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;pdev&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;vga_device&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;vgadev&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;pci_bus&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;bus&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;pci_dev&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;bridge&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;u16&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="cm"&gt;/* Only deal with VGA class devices */&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;pdev&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;class&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PCI_CLASS_DISPLAY_VGA&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;false&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;We're blowing up on the read to &lt;code&gt;pdev-&amp;gt;class&lt;/code&gt;, and it's not something like the data being uninitialised, it's out-of-bounds.  If we look back at the call trace:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;vga_arbiter_add_pci_device+0x60/0xe00
pci_notify+0x88/0x444
notifier_call_chain+0x104/0x320
blocking_notifier_call_chain+0xa0/0x140
device_add+0xac8/0x1d30
device_register+0x58/0x80
vio_register_device_node+0x9ac/0xce0
vio_bus_scan_register_devices+0xc4/0x13c
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This thing is a VIO device, not a PCI device!  Let's jump into the caller, &lt;code&gt;pci_notify()&lt;/code&gt;, to find out how we got our &lt;code&gt;pdev&lt;/code&gt;.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;pci_notify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;notifier_block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;nb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;action&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;                      &lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;device&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;pci_dev&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;pdev&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to_pci_dev&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;So &lt;code&gt;pci_notify()&lt;/code&gt; gets called with our VIO device (somehow), and we're converting that &lt;code&gt;struct device&lt;/code&gt; into a &lt;code&gt;struct pci_dev&lt;/code&gt; with no error checking.  We could solve this particular bug by just checking that our device is &lt;em&gt;actually&lt;/em&gt; a PCI device before we proceed - but we're in a function called &lt;code&gt;pci_notify&lt;/code&gt;, we're expecting a PCI device to come in, so this would just be a bandaid.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;to_pci_dev()&lt;/code&gt; works like other struct containers in the kernel - &lt;code&gt;struct pci_dev&lt;/code&gt; contains a &lt;code&gt;struct device&lt;/code&gt; as a member, so the &lt;code&gt;container_of()&lt;/code&gt; function returns an address based on where a &lt;code&gt;struct pci_dev&lt;/code&gt; would have to be if the given &lt;code&gt;struct device&lt;/code&gt; was actually a PCI device.  Since we know it's not actually a PCI device and this &lt;code&gt;struct device&lt;/code&gt; does not actually sit inside a &lt;code&gt;struct pci_dev&lt;/code&gt;, our &lt;code&gt;pdev&lt;/code&gt; is now pointing to some random place in memory, hence our access to a member like &lt;code&gt;class&lt;/code&gt; is caught by KASAN.&lt;/p&gt;
&lt;p&gt;Now we know why and how we're blowing up, but we still don't understand how we got here, so let's back up further.&lt;/p&gt;
&lt;h2&gt;Notifiers&lt;/h2&gt;
&lt;p&gt;The kernel's device subsystem allows consumers to register callbacks so that they can be notified of a given event.  I'm not going to go into a ton of detail on how they work, because I don't fully understand myself, and there's a lot of internals of the device subsystem involved.
The best references I could find for this are &lt;a href="https://elixir.bootlin.com/linux/latest/source/include/linux/notifier.h"&gt;notifier.h&lt;/a&gt;, and for our purposes here, &lt;a href="https://elixir.bootlin.com/linux/latest/source/include/linux/device/bus.h#L260"&gt;the register notifier functions in bus.h&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Something's clearly gone awry if we can end up in a function named &lt;code&gt;pci_notify()&lt;/code&gt; without passing it a PCI device.  We find where the notifier is registered in &lt;code&gt;vgaarb.c&lt;/code&gt; here:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;notifier_block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_notifier&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;notifier_call&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_notify&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;

&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__init&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;vga_arb_device_init&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="cm"&gt;/* some stuff removed here... */&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;bus_register_notifier&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;pci_bus_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;pci_notifier&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This all looks sane.  A blocking notifier is registered so that &lt;code&gt;pci_notify()&lt;/code&gt; gets called whenever there's a notification going out to PCI buses.  Our VIO device is distinctly &lt;em&gt;not&lt;/em&gt; on a PCI bus, and in my debugging I couldn't find any potential causes of such confusion, so how on earth is a notification for PCI buses being applied to our non-PCI device?&lt;/p&gt;
&lt;p&gt;Deep in the guts of the device subsystem, if we have a look at &lt;code&gt;device_add()&lt;/code&gt; we find the following:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;device_add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;device&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="cm"&gt;/* lots of device init stuff... */&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;bus&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="n"&gt;blocking_notifier_call_chain&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;bus&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;bus_notifier&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;                                             &lt;/span&gt;&lt;span class="n"&gt;BUS_NOTIFY_ADD_DEVICE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If the device we're initialising is attached to a bus, then we call the bus notifier of that bus with the &lt;code&gt;BUS_NOTIFY_ADD_DEVICE&lt;/code&gt; notification, and the device in question.  So we're going through the process of adding a VIO device, and somehow calling into a notifier that's only registered for PCI devices.  I did a bunch of debugging to see if our VIO device was somehow malformed and pointing to a PCI bus, or the &lt;code&gt;struct subsys_private&lt;/code&gt; (that's the &lt;code&gt;bus-&amp;gt;p&lt;/code&gt; above) was somehow pointing to the wrong place, but everything seemed sane.  My thesis of there being confusion while matching devices to buses was getting harder to justify - everything still looked sane.&lt;/p&gt;
&lt;h2&gt;Debuggers&lt;/h2&gt;
&lt;p&gt;I do not like debuggers.  I am an avid &lt;code&gt;printk()&lt;/code&gt; enthusiast.  There's no real justification for this, a bunch of my problems could almost certainly be solved easier by using actual tools, but my brain seemingly enjoys the routine of printing and building and running until I figure out what's going on.  It was becoming increasingly obvious, however, that &lt;code&gt;printk&lt;/code&gt; could not save me here, and we needed to go deeper.&lt;/p&gt;
&lt;p&gt;Very thankfully for me, even though this bug was discovered on real hardware, it reproduces easily in &lt;a href="https://www.qemu.org"&gt;QEMU&lt;/a&gt;, making iteration easy.  With &lt;a href="https://qemu-project.gitlab.io/qemu/system/gdb.html"&gt;GDB attached to QEMU&lt;/a&gt;, it's time to dive in to the guts of this issue and figure out what's happening.&lt;/p&gt;
&lt;p&gt;Somehow, VIO buses are ending up with &lt;code&gt;pci_notify()&lt;/code&gt; in their &lt;code&gt;bus_notifier&lt;/code&gt; list.  Let's break down the data structures here with a look at &lt;code&gt;struct notifier_block&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;notifier_block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;notifier_fn_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;notifier_call&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;notifier_block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__rcu&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;next&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;priority&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;So notifier chains are &lt;a href="https://en.wikipedia.org/wiki/Linked_list#Singly_linked_list"&gt;singly linked lists&lt;/a&gt;.  Callbacks are registered through functions like &lt;code&gt;bus_register_notifier()&lt;/code&gt;, then after a long chain of breadcrumbs we reach &lt;a href="https://elixir.bootlin.com/linux/latest/source/kernel/notifier.c#L22"&gt;&lt;code&gt;notifier_chain_register()&lt;/code&gt;&lt;/a&gt; which walks the list of &lt;code&gt;-&amp;gt;next&lt;/code&gt; pointers until it reaches &lt;code&gt;NULL&lt;/code&gt;, at which point it sets &lt;code&gt;-&amp;gt;next&lt;/code&gt; of the tail node to the &lt;code&gt;struct notifier_block&lt;/code&gt; that was passed in.  It's very important to note here that the data being appended to the list here is &lt;em&gt;not just the callback function&lt;/em&gt; (i.e. &lt;code&gt;pci_notify()&lt;/code&gt;), but the &lt;code&gt;struct notifier_block&lt;/code&gt; itself (i.e. &lt;code&gt;struct notifier_block pci_notifier&lt;/code&gt; from earlier).  There's no new data being initialised, just updating a pointer to the object that was passed by the caller.&lt;/p&gt;
&lt;p&gt;If you've guessed what our bug is at this point, great job!  If the same &lt;code&gt;struct notifier_block&lt;/code&gt; gets registered to two different bus types, then both of their &lt;code&gt;bus_notifier&lt;/code&gt; fields will point to the &lt;em&gt;same memory&lt;/em&gt;, and any further notifiers registered to either bus will end up being referenced by both since they walk through the same node.&lt;/p&gt;
&lt;p&gt;So we bust out the debugger and start looking at what ends up in &lt;code&gt;bus_notifier&lt;/code&gt; for PCI and VIO buses with breakpoints and watchpoints.&lt;/p&gt;
&lt;h2&gt;Candidates&lt;/h2&gt;
&lt;p&gt;Walking the &lt;code&gt;bus_notifier&lt;/code&gt; list gave me the following:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;__gcov_.perf_trace_module_free
fail_iommu_bus_notify
isa_bridge_notify
ppc_pci_unmap_irq_line
eeh_device_notifier
iommu_bus_notifier
tce_iommu_bus_notifier
pci_notify
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Time to find out if our assumption is correct - the same &lt;code&gt;struct notifier_block&lt;/code&gt; is being registered to both bus types.  Let's start going through them!&lt;/p&gt;
&lt;p&gt;First up, we have &lt;code&gt;__gcov_.perf_trace_module_free&lt;/code&gt;.  Thankfully, I recognised this as complete bait.  Trying to figure out what gcov and perf are doing here is going to be its own giant rabbit hole, and unless building without gcov makes our problem disappear, we skip this one and keep on looking.  Rabbit holes in the kernel never end, we have to be strategic with our time!&lt;/p&gt;
&lt;p&gt;Next, we reach &lt;code&gt;fail_iommu_bus_notify&lt;/code&gt;, so let's take a look at that.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;notifier_block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fail_iommu_bus_notifier&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;notifier_call&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fail_iommu_bus_notify&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;

&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__init&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;fail_iommu_setup&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="cp"&gt;#ifdef CONFIG_PCI&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;bus_register_notifier&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;pci_bus_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;fail_iommu_bus_notifier&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="cp"&gt;#endif&lt;/span&gt;
&lt;span class="cp"&gt;#ifdef CONFIG_IBMVIO&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;bus_register_notifier&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;vio_bus_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;fail_iommu_bus_notifier&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="cp"&gt;#endif&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Sure enough, here's our bug.  The same node is being registered to two different bus types:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;+------------------+
| PCI bus_notifier \
+------------------+\
                     \+-------------------------+    +-----------------+    +------------+
                      | fail_iommu_bus_notifier |----| PCI + VIO stuff |----| pci_notify |
                     /+-------------------------+    +-----------------+    +------------+
+------------------+/
| VIO bus_notifier /
+------------------+
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;when it should be like:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;+------------------+    +-----------------------------+    +-----------+    +------------+
| PCI bus_notifier |----| fail_iommu_pci_bus_notifier |----| PCI stuff |----| pci_notify |
+------------------+    +-----------------------------+    +-----------+    +------------+

+------------------+    +-----------------------------+    +-----------+
| VIO bus_notifier |----| fail_iommu_vio_bus_notifier |----| VIO stuff |
+------------------+    +-----------------------------+    +-----------+
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2&gt;The fix&lt;/h2&gt;
&lt;p&gt;Ultimately, the fix turned out to be pretty simple:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Author: Russell Currey &amp;lt;ruscur@russell.cc&amp;gt;
Date:   Wed Mar 22 14:37:42 2023 +1100

&lt;span class="w"&gt; &lt;/span&gt;   powerpc/iommu: Fix notifiers being shared by PCI and VIO buses

&lt;span class="w"&gt; &lt;/span&gt;   fail_iommu_setup() registers the fail_iommu_bus_notifier struct to both
&lt;span class="w"&gt; &lt;/span&gt;   PCI and VIO buses.  struct notifier_block is a linked list node, so this
&lt;span class="w"&gt; &lt;/span&gt;   causes any notifiers later registered to either bus type to also be
&lt;span class="w"&gt; &lt;/span&gt;   registered to the other since they share the same node.

&lt;span class="w"&gt; &lt;/span&gt;   This causes issues in (at least) the vgaarb code, which registers a
&lt;span class="w"&gt; &lt;/span&gt;   notifier for PCI buses.  pci_notify() ends up being called on a vio
&lt;span class="w"&gt; &lt;/span&gt;   device, converted with to_pci_dev() even though it&amp;#39;s not a PCI device,
&lt;span class="w"&gt; &lt;/span&gt;   and finally makes a bad access in vga_arbiter_add_pci_device() as
&lt;span class="w"&gt; &lt;/span&gt;   discovered with KASAN:

&lt;span class="w"&gt; &lt;/span&gt;   [stack trace redacted, see above]

&lt;span class="w"&gt; &lt;/span&gt;   Fix this by creating separate notifier_block structs for each bus type.

&lt;span class="w"&gt; &lt;/span&gt;   Fixes: d6b9a81b2a45 (&amp;quot;powerpc: IOMMU fault injection&amp;quot;)
&lt;span class="w"&gt; &lt;/span&gt;   Reported-by: Nageswara R Sastry &amp;lt;rnsastry@linux.ibm.com&amp;gt;
&lt;span class="w"&gt; &lt;/span&gt;   Signed-off-by: Russell Currey &amp;lt;ruscur@russell.cc&amp;gt;

&lt;span class="gh"&gt;diff --git a/arch/powerpc/kernel/iommu.c b/arch/powerpc/kernel/iommu.c&lt;/span&gt;
&lt;span class="gh"&gt;index ee95937bdaf1..6f1117fe3870 100644&lt;/span&gt;
&lt;span class="gd"&gt;--- a/arch/powerpc/kernel/iommu.c&lt;/span&gt;
&lt;span class="gi"&gt;+++ b/arch/powerpc/kernel/iommu.c&lt;/span&gt;
&lt;span class="gu"&gt;@@ -171,17 +171,26 @@ static int fail_iommu_bus_notify(struct notifier_block *nb,&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;        return 0;
&lt;span class="w"&gt; &lt;/span&gt;}

&lt;span class="gd"&gt;-static struct notifier_block fail_iommu_bus_notifier = {&lt;/span&gt;
&lt;span class="gi"&gt;+/*&lt;/span&gt;
&lt;span class="gi"&gt;+ * PCI and VIO buses need separate notifier_block structs, since they&amp;#39;re linked&lt;/span&gt;
&lt;span class="gi"&gt;+ * list nodes.  Sharing a notifier_block would mean that any notifiers later&lt;/span&gt;
&lt;span class="gi"&gt;+ * registered for PCI buses would also get called by VIO buses and vice versa.&lt;/span&gt;
&lt;span class="gi"&gt;+ */&lt;/span&gt;
&lt;span class="gi"&gt;+static struct notifier_block fail_iommu_pci_bus_notifier = {&lt;/span&gt;
&lt;span class="gi"&gt;+        .notifier_call = fail_iommu_bus_notify&lt;/span&gt;
&lt;span class="gi"&gt;+};&lt;/span&gt;
&lt;span class="gi"&gt;+&lt;/span&gt;
&lt;span class="gi"&gt;+static struct notifier_block fail_iommu_vio_bus_notifier = {&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;        .notifier_call = fail_iommu_bus_notify
&lt;span class="w"&gt; &lt;/span&gt;};

&lt;span class="w"&gt; &lt;/span&gt;static int __init fail_iommu_setup(void)
&lt;span class="w"&gt; &lt;/span&gt;{
&lt;span class="w"&gt; &lt;/span&gt;#ifdef CONFIG_PCI
&lt;span class="gd"&gt;-        bus_register_notifier(&amp;amp;pci_bus_type, &amp;amp;fail_iommu_bus_notifier);&lt;/span&gt;
&lt;span class="gi"&gt;+        bus_register_notifier(&amp;amp;pci_bus_type, &amp;amp;fail_iommu_pci_bus_notifier);&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;#endif
&lt;span class="w"&gt; &lt;/span&gt;#ifdef CONFIG_IBMVIO
&lt;span class="gd"&gt;-        bus_register_notifier(&amp;amp;vio_bus_type, &amp;amp;fail_iommu_bus_notifier);&lt;/span&gt;
&lt;span class="gi"&gt;+        bus_register_notifier(&amp;amp;vio_bus_type, &amp;amp;fail_iommu_vio_bus_notifier);&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;#endif

&lt;span class="w"&gt; &lt;/span&gt;        return 0;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Easy!  Problem solved.  The &lt;a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d6b9a81b2a45"&gt;commit that introduced this bug back in 2012&lt;/a&gt; was written by the legendary &lt;a href="http://antonblanchardfacts.com"&gt;Anton Blanchard&lt;/a&gt;, so it's always a treat to discover an Anton bug.  Ultimately this bug is of little consequence, but it's always fun to catch dormant issues with powerful tools like KASAN.&lt;/p&gt;
&lt;h2&gt;In conclusion&lt;/h2&gt;
&lt;p&gt;I think this bug provides a nice window into what kernel debugging can be like.  Thankfully, things are made easier by not dealing with any specific hardware and being easily reproducible in QEMU.&lt;/p&gt;
&lt;p&gt;Bugs like this have an absurd amount of underlying complexity, but you rarely need to understand all of it to comprehend the situation and discover the issue.  I spent way too much time digging into device subsystem internals, when the odds of the issue lying within were quite low - the combination of IBM VIO devices and VGA arbitration isn't exactly common, so searching for potential issues within the guts of a heavily utilised subsystem isn't going to yield results very often.&lt;/p&gt;
&lt;p&gt;Is there something haunted in the device subsystem?  Is there something haunted inside the notifier handlers?  It's possible, but assuming the core guts of the kernel have a baseline level of sanity helps to let you stay focused on the parts more likely to be relevant.&lt;/p&gt;
&lt;p&gt;Finally, the process was made much easier by having good code navigation.  A ludicrous amount of kernel developers still use plain vim or Emacs, maybe with tags if you're lucky, and get by on &lt;code&gt;git grep&lt;/code&gt; (not even ripgrep!) and memory.  Sort yourselves out and get yourself an editor with LSP support.  I personally use &lt;a href="https://github.com/doomemacs/doomemacs"&gt;Doom Emacs&lt;/a&gt; with &lt;a href="https://clangd.llvm.org/"&gt;clangd&lt;/a&gt;, and with the amount of jumping around the kernel I had to do to solve this bug, it would've been a much bigger ordeal without that power.&lt;/p&gt;
&lt;p&gt;If you enjoyed the read, why not follow me on &lt;a href="https://ozlabs.house/@ruscur"&gt;Mastodon&lt;/a&gt; or checkout &lt;a href="https://sthbrx.github.io/blog/2023/03/24/dumb-bugs-when-a-date-breaks-booting-the-kernel/"&gt;Ben's recount of another cursed bug!&lt;/a&gt;  Thanks for stopping by.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Russell Currey</dc:creator><pubDate>Tue, 04 Apr 2023 15:55:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2023-04-04:/blog/2023/04/04/dumb-bugs-the-pci-device-that-wasnt/</guid><category>Development</category><category>linux</category></item><item><title>Dumb bugs: When a date breaks booting the kernel</title><link>https://sthbrx.github.io/blog/2023/03/24/dumb-bugs-when-a-date-breaks-booting-the-kernel/</link><description>&lt;h2&gt;The setup&lt;/h2&gt;
&lt;p&gt;I've recently been working on internal CI infrastructure for testing kernels before sending them to the mailing list. As part of this effort, I became interested in &lt;a href="https://reproducible-builds.org/"&gt;reproducible builds&lt;/a&gt;. Minimising the changing parts outside of the source tree itself could improve consistency and ccache hits, which is great for trying to make the CI faster and more reproducible across different machines. This means removing 'external' factors like timestamps from the build process, because the time changes every build and means the results between builds of the same tree are no longer identical binaries. This also prevents using previously cached results, potentially slowing down builds (though it turns out the kernel does a good job of limiting the scope of where timestamps appear in the build).&lt;/p&gt;
&lt;p&gt;As part of this effort, I came across the &lt;code&gt;KBUILD_BUILD_TIMESTAMP&lt;/code&gt; environment variable. This variable is used to set the kernel timestamp, which is primarily for any users who want to know when their kernel was built. That's mostly irrelevant for our work, so an easy &lt;code&gt;KBUILD_BUILD_TIMESTAMP=0&lt;/code&gt; later and... it still uses the current date.&lt;/p&gt;
&lt;p&gt;Ok, checking &lt;a href="https://docs.kernel.org/kbuild/kbuild.html#kbuild-build-timestamp"&gt;the documentation&lt;/a&gt; it says&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Setting this to a date string overrides the timestamp used in the UTS_VERSION definition (uname -v in the running kernel). The value has to be a string that can be passed to date -d. The default value is the output of the date command at one point during build.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;So it looks like the timestamp variable is actually expected to be a date format. To make it obvious that it's not a 'real' date, let's set &lt;code&gt;KBUILD_BUILD_TIMESTAMP=0000-01-01&lt;/code&gt;. A bunch of zeroes (and the ones to make it a valid month and day) should tip off anyone to the fact it's invalid.&lt;/p&gt;
&lt;p&gt;As an aside, this is a different date to what I tried to set it to earlier; a 'timestamp' typically refers to the number of seconds since the UNIX epoch (1970), so my first attempt would have corresponded to 1970-01-01. But given we're passing a date, not a timestamp, there should be no problem setting it back to the year 0. And I like the aesthetics of 0000 over 1970.&lt;/p&gt;
&lt;p&gt;Building and booting the kernel, we see &lt;code&gt;#1 SMP 0000-01-01&lt;/code&gt; printed as the build timestamp. Success! After confirming everything works, I set the environment variable in the CI jobs and call it a day.&lt;/p&gt;
&lt;h2&gt;An unexpected error&lt;/h2&gt;
&lt;p&gt;A few days later I need to run the CI to test my patches, and something strange happens. It builds fine, but the boot tests that load a root disk image fail inexplicably: there is a kernel panic saying "VFS: Unable to mount root fs on unknown-block(253,2)".&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[    0.909648][    T1] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(253,2)
[    0.909797][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.3.0-rc2-g065ffaee7389 #8
[    0.909880][    T1] Hardware name: IBM pSeries (emulated by qemu) POWER8 (raw) 0x4d0200 0xf000004 of:SLOF,HEAD pSeries
[    0.910044][    T1] Call Trace:
[    0.910107][    T1] [c000000003643b00] [c000000000fb6f9c] dump_stack_lvl+0x70/0xa0 (unreliable)
[    0.910378][    T1] [c000000003643b30] [c000000000144e34] panic+0x178/0x424
[    0.910423][    T1] [c000000003643bd0] [c000000002005144] mount_block_root+0x1d0/0x2bc
[    0.910457][    T1] [c000000003643ca0] [c000000002005720] prepare_namespace+0x1d4/0x22c
[    0.910487][    T1] [c000000003643d20] [c000000002004b04] kernel_init_freeable+0x36c/0x3bc
[    0.910517][    T1] [c000000003643df0] [c000000000013830] kernel_init+0x30/0x1a0
[    0.910549][    T1] [c000000003643e50] [c00000000000df94] ret_from_kernel_thread+0x5c/0x64
[    0.910587][    T1] --- interrupt: 0 at 0x0
[    0.910794][    T1] NIP:  0000000000000000 LR: 0000000000000000 CTR: 0000000000000000
[    0.910828][    T1] REGS: c000000003643e80 TRAP: 0000   Not tainted  (6.3.0-rc2-g065ffaee7389)
[    0.910883][    T1] MSR:  0000000000000000 &amp;lt;&amp;gt;  CR: 00000000  XER: 00000000
[    0.910990][    T1] CFAR: 0000000000000000 IRQMASK: 0
[    0.910990][    T1] GPR00: 0000000000000000 c000000003644000 0000000000000000 0000000000000000
[    0.910990][    T1] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    0.910990][    T1] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    0.910990][    T1] GPR12: 0000000000000000 0000000000000000 c000000000013808 0000000000000000
[    0.910990][    T1] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    0.910990][    T1] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    0.910990][    T1] GPR24: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    0.910990][    T1] GPR28: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    0.911371][    T1] NIP [0000000000000000] 0x0
[    0.911397][    T1] LR [0000000000000000] 0x0
[    0.911427][    T1] --- interrupt: 0
qemu-system-ppc64: OS terminated: OS panic: VFS: Unable to mount root fs on unknown-block(253,2)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Above the panic was some more context, saying&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[    0.906194][    T1] Warning: unable to open an initial console.
...
[    0.908321][    T1] VFS: Cannot open root device &amp;quot;vda2&amp;quot; or unknown-block(253,2): error -2
[    0.908356][    T1] Please append a correct &amp;quot;root=&amp;quot; boot option; here are the available partitions:
[    0.908528][    T1] 0100           65536 ram0
[    0.908657][    T1]  (driver?)
[    0.908735][    T1] 0101           65536 ram1
[    0.908744][    T1]  (driver?)
...
[    0.909216][    T1] 010f           65536 ram15
[    0.909226][    T1]  (driver?)
[    0.909265][    T1] fd00         5242880 vda
[    0.909282][    T1]  driver: virtio_blk
[    0.909335][    T1]   fd01            4096 vda1 d1f35394-01
[    0.909364][    T1]
[    0.909401][    T1]   fd02         5237760 vda2 d1f35394-02
[    0.909408][    T1]
[    0.909441][    T1] fd10             366 vdb
[    0.909446][    T1]  driver: virtio_blk
[    0.909479][    T1] 0b00         1048575 sr0
[    0.909486][    T1]  driver: sr
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is even more baffling: if it's unable to open a console, then what am I reading these messages on? And error &lt;code&gt;-2&lt;/code&gt;, or ENOENT, on opening 'vda2' implies that no such file or directory exists. But it then lists vda2 as a present drive with a known driver? So is vda2 missing or not?&lt;/p&gt;
&lt;h2&gt;Living in denial&lt;/h2&gt;
&lt;p&gt;As you've read the title of this article, you can probably guess as to what changed to cause this error. But at the time I had no idea what could have been the cause. I'd already confirmed that a kernel with a set timestamp can boot to userspace, and there was another (seemingly) far more likely candidate for the failure: as part of the CI design, patches are extracted from the submitted branch and rebased onto the maintainer's tree. This is great from a convenience perspective, because you don't need to worry about forgetting to rebase your patches before testing and submission. But if the maintainer has synced their branch with Linus' tree it means there could be a lot of things changed in the source tree between runs, even if they were only a few days apart.&lt;/p&gt;
&lt;p&gt;So, when you're faced with a working test on one commit and a broken test on another commit, it's time to break out the &lt;code&gt;git bisect&lt;/code&gt;. Downloading the kernel images from the relevant CI jobs, I confirmed that indeed one was working while the other was broken. So I bisected the relevant commits, and... everything kept working. Each step I would build and boot the kernel, and each step would reach userspace just fine. I was getting suspicious at this point, so skipped ahead to the known bad commit and built and tested it locally. It &lt;em&gt;also worked&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;This was highly confusing, because it meant there was something fishy going on. Some kind of state outside of the kernel tree. Could it be... surely not...&lt;/p&gt;
&lt;p&gt;Comparing the boot logs of the two CI kernels, I see that the working one indeed uses an actual timestamp, and the broken one uses the 0000-01-01 fixed date. Oh no. Setting the timestamp with a local build, I can now reproduce the boot panic with a kernel I built myself.&lt;/p&gt;
&lt;h2&gt;But... why?&lt;/h2&gt;
&lt;p&gt;OK, so it's obvious at this point that the timestamp is affecting loading a root disk somehow. But why? The obvious answer is that it's before the UNIX epoch. Something in the build process is turning the date into an actual timestamp, and going wrong when that timestamp gets used for something.&lt;/p&gt;
&lt;p&gt;But it's not like there was a build error complaining about it. As best I could tell, the kernel doesn't try to parse the date anywhere, besides passing it to &lt;code&gt;date&lt;/code&gt; during the build. And if &lt;code&gt;date&lt;/code&gt; had an issue with it, it would have broken the &lt;em&gt;build&lt;/em&gt;. Not &lt;em&gt;booting&lt;/em&gt; the kernel. There's no &lt;code&gt;date&lt;/code&gt; utility being invoked during kernel boot!&lt;/p&gt;
&lt;p&gt;Regardless, I set about tracing the usage of &lt;code&gt;KBUILD_BUILD_TIMESTAMP&lt;/code&gt; inside the kernel. The stacktrace in the panic gave the end point of the search; the function &lt;code&gt;mount_block_root()&lt;/code&gt; wasn't happy. So all I had to do was work out at which point &lt;code&gt;mount_block_root()&lt;/code&gt; tried to access the &lt;code&gt;KBUILD_BUILD_TIMESTAMP&lt;/code&gt; value.&lt;/p&gt;
&lt;p&gt;In short, that went nowhere.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;mount_block_root()&lt;/code&gt; effectively just tries to open a file in the filesystem. There's massive amounts of code handling this, and any part could have had the undocumented dependency on &lt;code&gt;KBUILD_BUILD_TIMESTAMP&lt;/code&gt;. Approaching from the other direction, &lt;code&gt;KBUILD_BUILD_TIMESTAMP&lt;/code&gt; is turned into &lt;code&gt;build-timestamp&lt;/code&gt; inside a Makefile, which is in turn related to a file &lt;code&gt;include/generated/utsversion.h&lt;/code&gt;. This file &lt;code&gt;#define&lt;/code&gt;s &lt;code&gt;UTS_VERSION&lt;/code&gt; equal to the &lt;code&gt;KBUILD_BUILD_TIMESTAMP&lt;/code&gt; value. Searching the kernel for &lt;code&gt;UTS_VERSION&lt;/code&gt;, we hit &lt;code&gt;init/version-timestamp.c&lt;/code&gt; which stores it in a struct with other build information:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;uts_namespace&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;init_uts_ns&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ns&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;REFCOUNT_INIT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sysname&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;UTS_SYSNAME&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;nodename&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;UTS_NODENAME&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;release&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;UTS_RELEASE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;UTS_VERSION&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;machine&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;UTS_MACHINE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;domainname&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;UTS_DOMAINNAME&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;user_ns&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;init_user_ns&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ns&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inum&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PROC_UTS_INIT_INO&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="cp"&gt;#ifdef CONFIG_UTS_NS&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ns&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ops&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;utsns_operations&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="cp"&gt;#endif&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is where the trail goes cold: I don't know if you've ever tried this, but searching for &lt;code&gt;.version&lt;/code&gt; in the kernel's codebase is not a very fruitful endeavor when you're interested in a specific kind of version.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ rg &amp;quot;(\.|\-&amp;gt;)version\b&amp;quot; | wc -l
5718
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I tried tracing the usage of &lt;code&gt;init_uts_ns&lt;/code&gt;, but didn't get very far.&lt;/p&gt;
&lt;p&gt;By now I'd already posted this in chat and another developer, &lt;a href="https://shenki.github.io/"&gt;Joel Stanley&lt;/a&gt;, was also investigating this bizarre bug. They had been testing different timestamp values and made the horrifying discovery that the bug sticks around after a rebuild. So you could start with a broken build, set the timestamp back to the correct value, rebuild, and the resulting kernel would &lt;em&gt;still be broken&lt;/em&gt;. The boot log would report the correct time, but the root disk mounter panicked all the same.&lt;/p&gt;
&lt;h2&gt;Getting sidetracked&lt;/h2&gt;
&lt;p&gt;I wasn't prepared to investigate the boot panic directly until the persistence bug was fixed. Having to run &lt;code&gt;make clean&lt;/code&gt; and rebuild everything would take an annoyingly long time, even with ccache. Fortunately, I had a plan. All I had to do was work out which generated files are different between a broken and working build, and binary search by deleting half of them until deleting only one made the difference between the bug persisting or not. We can use &lt;code&gt;diff&lt;/code&gt; for this. Running the initial diff we get&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ diff -q --exclude System.map --exclude .tmp_vmlinux* --exclude tools broken/ working/
Common subdirectories: broken/arch and working/arch
Common subdirectories: broken/block and working/block
Files broken/built-in.a and working/built-in.a differ
Common subdirectories: broken/certs and working/certs
Common subdirectories: broken/crypto and working/crypto
Common subdirectories: broken/drivers and working/drivers
Common subdirectories: broken/fs and working/fs
Common subdirectories: broken/include and working/include
Common subdirectories: broken/init and working/init
Common subdirectories: broken/io_uring and working/io_uring
Common subdirectories: broken/ipc and working/ipc
Common subdirectories: broken/kernel and working/kernel
Common subdirectories: broken/lib and working/lib
Common subdirectories: broken/mm and working/mm
Common subdirectories: broken/net and working/net
Common subdirectories: broken/scripts and working/scripts
Common subdirectories: broken/security and working/security
Common subdirectories: broken/sound and working/sound
Common subdirectories: broken/usr and working/usr
Files broken/.version and working/.version differ
Common subdirectories: broken/virt and working/virt
Files broken/vmlinux and working/vmlinux differ
Files broken/vmlinux.a and working/vmlinux.a differ
Files broken/vmlinux.o and working/vmlinux.o differ
Files broken/vmlinux.strip.gz and working/vmlinux.strip.gz differ
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Hmm, OK so only some top level files are different. Deleting all the different files doesn't fix the persistence bug though, and I know that a proper &lt;code&gt;make clean&lt;/code&gt; does fix it, so what could possibly be the difference when all the remaining files are identical?&lt;/p&gt;
&lt;p&gt;Oh wait. &lt;code&gt;man diff&lt;/code&gt; reports that &lt;code&gt;diff&lt;/code&gt; only compares the top level folder entries by default. So it was literally just telling me "yes, both the broken and working builds have a folder named X". How GNU of it. Re-running the diff command with actually useful options, we get a more promising story&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ diff -qr --exclude System.map --exclude .tmp_vmlinux* --exclude tools build/broken/ build/working/
Files build/broken/arch/powerpc/boot/zImage and build/working/arch/powerpc/boot/zImage differ
Files build/broken/arch/powerpc/boot/zImage.epapr and build/working/arch/powerpc/boot/zImage.epapr differ
Files build/broken/arch/powerpc/boot/zImage.pseries and build/working/arch/powerpc/boot/zImage.pseries differ
Files build/broken/built-in.a and build/working/built-in.a differ
Files build/broken/include/generated/utsversion.h and build/working/include/generated/utsversion.h differ
Files build/broken/init/built-in.a and build/working/init/built-in.a differ
Files build/broken/init/utsversion-tmp.h and build/working/init/utsversion-tmp.h differ
Files build/broken/init/version.o and build/working/init/version.o differ
Files build/broken/init/version-timestamp.o and build/working/init/version-timestamp.o differ
Files build/broken/usr/built-in.a and build/working/usr/built-in.a differ
Files build/broken/usr/initramfs_data.cpio and build/working/usr/initramfs_data.cpio differ
Files build/broken/usr/initramfs_data.o and build/working/usr/initramfs_data.o differ
Files build/broken/usr/initramfs_inc_data and build/working/usr/initramfs_inc_data differ
Files build/broken/.version and build/working/.version differ
Files build/broken/vmlinux and build/working/vmlinux differ
Files build/broken/vmlinux.a and build/working/vmlinux.a differ
Files build/broken/vmlinux.o and build/working/vmlinux.o differ
Files build/broken/vmlinux.strip.gz and build/working/vmlinux.strip.gz differ
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There are some new entries here: notably &lt;code&gt;init/version*&lt;/code&gt; and &lt;code&gt;usr/initramfs*&lt;/code&gt;. Binary searching these files results in a single culprit: &lt;code&gt;usr/initramfs_data.cpio&lt;/code&gt;. This is quite fitting, as the &lt;code&gt;.cpio&lt;/code&gt; file is an archive defining a filesystem layout, &lt;a href="https://docs.kernel.org/filesystems/ramfs-rootfs-initramfs.html?highlight=initramfs#why-cpio-rather-than-tar"&gt;much like &lt;code&gt;.tar&lt;/code&gt; files&lt;/a&gt;. This file is actually embedded into the kernel image, and loaded as a bare-bones shim filesystem when the user doesn't provide their own initramfs&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1"&gt;1&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;So it would make sense that if the CPIO archive wasn't being rebuilt, then the initial filesystem wouldn't change. And it would make sense for the initial filesystem to be causing mount issues of the proper root disk filesystem.&lt;/p&gt;
&lt;p&gt;This just leaves the question of how &lt;code&gt;KBUILD_BUILD_TIMESTAMP&lt;/code&gt; is breaking the CPIO archive. And it's around this time that a third developer, &lt;a href="https://twitter.com/ajdlinux"&gt;Andrew&lt;/a&gt;, who I'd roped into this bug hunt for having the (mis)fortune to sit next to me, pointed out that the generator script for this CPIO archive was passing the &lt;code&gt;KBUILD_BUILD_TIMESTAMP&lt;/code&gt; to &lt;code&gt;date&lt;/code&gt;. Whoop, we've found the murder weapon&lt;sup id="fnref:2"&gt;&lt;a class="footnote-ref" href="#fn:2"&gt;2&lt;/a&gt;&lt;/sup&gt;!&lt;/p&gt;
&lt;p&gt;The persistence bug could be explained now: because the script was only using &lt;code&gt;KBUILD_BUILD_TIMESTAMP&lt;/code&gt; internally, &lt;code&gt;make&lt;/code&gt; had no way of knowing that the archive generation depended on this variable. So even when I changed the variable to a valid value, &lt;code&gt;make&lt;/code&gt; didn't know to rebuild the corrupt archive. Let's now get back to the main issue: why boot panics.&lt;/p&gt;
&lt;h2&gt;Solving the case&lt;/h2&gt;
&lt;p&gt;Following along the CPIO generation script, the &lt;code&gt;KBUILD_BUILD_TIMESTAMP&lt;/code&gt; variable is turned into a timestamp by &lt;code&gt;date -d"$KBUILD_BUILD_TIMESTAMP" +%s&lt;/code&gt;. Testing this in the shell with &lt;code&gt;0000-01-01&lt;/code&gt; we get this (somewhat amusing, but also painful) result&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;date -d&amp;quot;$KBUILD_BUILD_TIMESTAMP&amp;quot; +%s
-62167255492
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This timestamp is then passed to a C program that assigns it to a variable &lt;code&gt;default_mtime&lt;/code&gt;. Looking over the source, it seems this variable is used to set the &lt;code&gt;mtime&lt;/code&gt; field on the files in the CPIO archive. The timestamp is stored as a &lt;code&gt;time_t&lt;/code&gt;, which is an alias for &lt;code&gt;int64_t&lt;/code&gt;. That's 64 bits of data, up to 16 hexadecimal characters. And yes, that's relevant: CPIO stores the &lt;code&gt;mtime&lt;/code&gt; (and all other numerical fields) as 32 bit unsigned integers represented by ASCII hexadecimal characters. The &lt;code&gt;sprintf()&lt;/code&gt; call that ultimately embeds the timestamp uses the &lt;code&gt;%08lX&lt;/code&gt; format specifier. This formats a &lt;code&gt;long&lt;/code&gt; as hexadecimal, padded to at least 8 characters. Hang on... &lt;strong&gt;&lt;em&gt;at least&lt;/em&gt;&lt;/strong&gt; 8 characters? What if our timestamp happens to be more?&lt;/p&gt;
&lt;p&gt;It turns out that large timestamps are already guarded against. The program will error during build if the date is later than 2106-02-07 (maximum unsigned 8 hex digit timestamp).&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cm"&gt;/*&lt;/span&gt;
&lt;span class="cm"&gt; * Timestamps after 2106-02-07 06:28:15 UTC have an ascii hex time_t&lt;/span&gt;
&lt;span class="cm"&gt; * representation that exceeds 8 chars and breaks the cpio header&lt;/span&gt;
&lt;span class="cm"&gt; * specification.&lt;/span&gt;
&lt;span class="cm"&gt; */&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;default_mtime&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xffffffff&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;fprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;ERROR: Timestamp too large for cpio format&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;But we are using an &lt;code&gt;int64_t&lt;/code&gt;. What would happen if one were to provide a negative timestamp?&lt;/p&gt;
&lt;p&gt;Well, &lt;code&gt;sprintf()&lt;/code&gt; happily spits out &lt;code&gt;FFFFFFF1868AF63C&lt;/code&gt; when we pass in our negative timestamp representing &lt;code&gt;0000-01-01&lt;/code&gt;. That's 16 characters, 8 too many for the CPIO header&lt;sup id="fnref:3"&gt;&lt;a class="footnote-ref" href="#fn:3"&gt;3&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;So at last we've found the cause of the panic: the timestamp is being formatted too long, which breaks the CPIO header and the kernel doesn't create an initial filesystem correctly. This includes the &lt;code&gt;/dev&lt;/code&gt; folder (which surprisingly is not hardcoded into kernel, but must be declared by the initramfs). So when the root disk mounter tries to open &lt;code&gt;/dev/vda2&lt;/code&gt;, it correctly complains that it failed to create a device in the non-existent &lt;code&gt;/dev&lt;/code&gt;.&lt;/p&gt;
&lt;h2&gt;Postmortem&lt;/h2&gt;
&lt;p&gt;After discovering all this, I sent in a couple of patches to fix &lt;a href="https://lore.kernel.org/all/20230320040839.660475-1-bgray@linux.ibm.com/"&gt;the CPIO generation&lt;/a&gt; and &lt;a href="https://lore.kernel.org/all/20230320040839.660475-2-bgray@linux.ibm.com/"&gt;rebuild logic&lt;/a&gt;. They were not complicated fixes, but wow were they time consuming to track down. I didn't see the error initially because I typically only boot with my own initramfs over the embedded one, and not with the intent to load a root disk. Then the panic itself was quite far away from the real issue, and there were many dead ends to explore.&lt;/p&gt;
&lt;p&gt;I also got curious as to why the kernel didn't complain about a corrupt initramfs earlier. A brief investigation showed a streaming parser that is &lt;em&gt;extremely&lt;/em&gt; fault tolerant, silently skipping invalid entries (like ones missing or having too long a name). The corrupted header was being interpreted as an entry with an empty name and 2 gigabyte body contents, which meant that (1) the kernel skipped inserting it due to the empty name, and (2) the kernel skipped the rest of the initramfs because it thought that up to 2 GB of the remaining content was part of that first entry.&lt;/p&gt;
&lt;p&gt;Perhaps this could be improved to require that all input is consumed without unexpected EOF, such as how the userspace &lt;code&gt;cpio&lt;/code&gt; tool works (which, by the way, recognises the corrupt archive as such and refuses to decompress it). The parsing logic is mostly from the before-times though (i.e., pre initial git commit), so it's difficult to distinguish intentional leniency and bugs.&lt;/p&gt;
&lt;h2&gt;Afterword&lt;/h2&gt;
&lt;p&gt;Incidentally, in investigating this I came across another bug. There is a helper function &lt;code&gt;panic_show_mem()&lt;/code&gt; in the initramfs that's meant to dump memory information and then call &lt;code&gt;panic()&lt;/code&gt;. It takes in standard &lt;code&gt;printf()&lt;/code&gt; style format string and arguments, and tries to forward them to &lt;code&gt;panic()&lt;/code&gt; which ultimately prints them.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;panic_show_mem&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;...)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kt"&gt;va_list&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;show_mem&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;va_start&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;panic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;va_end&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;panic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;...);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;But variadic arguments don't quite work this way: instead of forwarding the list &lt;code&gt;args&lt;/code&gt; as intended, &lt;code&gt;panic()&lt;/code&gt; will instead interpret &lt;code&gt;args&lt;/code&gt; as a single argument for the format string &lt;code&gt;fmt&lt;/code&gt;. Standard library functions address this by providing &lt;code&gt;v*&lt;/code&gt; variants of &lt;code&gt;printf()&lt;/code&gt; and friends. For example,&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;...);&lt;/span&gt;

&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;vprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;va_list&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;We might create a &lt;code&gt;vpanic()&lt;/code&gt; function in the kernel that follows this style, but it seems easier to just make &lt;code&gt;panic_show_mem()&lt;/code&gt; a macro and 'forward' the arguments in the source code&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;#define panic_show_mem(fmt, ...) \&lt;/span&gt;
&lt;span class="cp"&gt;    ({ show_mem(0, NULL); panic(fmt, ##__VA_ARGS__); })&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://lore.kernel.org/all/20230320230534.50174-1-bgray@linux.ibm.com/"&gt;Patch sent&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;And that's where I've left things. Big thanks to Joel and Andrew for helping me with this bug. It was certainly a trip.&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;initramfs, or initrd for the older format, are specific kinds of CPIO archives. The initramfs is intended to be loaded as the initial filesystem of a booted kernel, typically in preparation for loading your normal root filesystem. It might contain modules necessary to mount the disk for example.&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:2"&gt;
&lt;p&gt;Hindsight again would suggest it was obvious to look here because it shows up when searching for &lt;code&gt;KBUILD_BUILD_TIMESTAMP&lt;/code&gt;. I unfortunately wasn't familiar with the &lt;code&gt;usr/&lt;/code&gt; source folder initially, and focused on the core kernel components too much earlier. Oh well, we found it eventually.&amp;#160;&lt;a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:3"&gt;
&lt;p&gt;I almost missed this initially. Thanks to the ASCII header format, &lt;code&gt;strings&lt;/code&gt; was able to print the headers without any CPIO specific tooling. I did a double take when I noticed the headers for the broken CPIO were a little longer than the headers in the working one.&amp;#160;&lt;a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Benjamin Gray</dc:creator><pubDate>Fri, 24 Mar 2023 00:00:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2023-03-24:/blog/2023/03/24/dumb-bugs-when-a-date-breaks-booting-the-kernel/</guid><category>Development</category><category>linux</category></item><item><title>What distro options are there for POWER8 in 2022?</title><link>https://sthbrx.github.io/blog/2022/11/16/what-distro-options-are-there-for-power8-in-2022/</link><description>&lt;p&gt;If you have POWER8 systems that you want to keep alive, what are your options in 2022?  You can keep using the legacy distribution you're still using as long as it's still supported, but if you want some modernisation, that might not be the best option for you.  Here's the current landscape of POWER8 support in major distributions, and hopefully it helps you out!&lt;/p&gt;
&lt;p&gt;Please note that I am entirely focused on what runs and keeps getting new packages, not what companies will officially support.  &lt;a href="https://www.ibm.com/docs/en/linux-on-systems?topic=lpo-supported-linux-distributions-virtualization-options-power8-power9-linux-power-systems"&gt;IBM provides documentation for that.&lt;/a&gt;  I'm also mostly focused on OpenPOWER and not what's supported under IBM PowerVM.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;RHEL-compatible&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Things aren't too great on the RHEL-compatible side.  RHEL 9 is compiled with P9 instructions, removing support for P8.  This includes compatible distributions, like CentOS Stream and Rocky Linux.&lt;/p&gt;
&lt;p&gt;You can continue to use RHEL 8 for a long time.  Unfortunately, Rocky Linux only has a Power release for EL9 and not EL8, and CentOS Stream 8 hits EOL May 31st, 2024 - a bit too soon for my liking.  If you're a RHEL customer though, you're set.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Fedora&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Fedora seems like a great option - the latest versions still support P8 and there's no immediate signs of that changing.  The issue is that Fedora could change this with relatively little warning (and their big brother RHEL already has), Fedora doesn't provide LTS versions that will stay supported if this happens, and any options you could migrate to would be very different from what you're using.&lt;/p&gt;
&lt;p&gt;For that reason, I don't recommend using Fedora on POWER8 if you intend to keep it around for a while.  If you want something modern for a short-term project, go right ahead!  Otherwise, I'd avoid it.  If you're still keeping POWER8 systems alive, you probably want something more set-and-forget than Fedora anyway.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Ubuntu&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Ubuntu is a mixed bag.  The good news is that Ubuntu 20.04 LTS is supported until mid-2025, and if you give Canonical money, that support can extend through 2030.  Ubuntu 20.04 LTS is my personal pick for the best distro to install on POWER8 systems that you want to have somewhat modern software but without the risks of future issues.&lt;/p&gt;
&lt;p&gt;The bad news is that POWER8 support went away in Ubuntu 22.04, which is extremely unfortunate.  Missing an LTS cycle is one thing, but &lt;em&gt;not having a pathway from 21.10 is another&lt;/em&gt;.  If you were on 20.10/21.04/21.10, you are completely boned, because they're all out of support and 22.04 and later don't support POWER8.  You're going to have to reinstall 20.04.&lt;/p&gt;
&lt;p&gt;If I sound salty, it's because I had to do this for a few machines.  Hopefully you're not in that situation.  20.04 is going to be around for a good while longer, with a lot of modern creature comforts you'd miss on an EL8-compatible distro, so it's my pick for now.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;OpenSUSE&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;I'm pretty ignorant when it comes to chameleon-flavoured distros, so take this with a grain of salt as most of it is from some quick searching.  OpenSUSE Leap follows SLES, but without extended support lifetimes for older major versions.  From what I can tell, the latest release (15.4) still includes POWER8 support (and adds Power10 support!), but similar to Fedora, that looks rather prone to a new version dropping P8 support to me.&lt;/p&gt;
&lt;p&gt;If the 15.x series stayed alive after 16 came out, you might be good, but it doesn't seem like there's a history of that happening.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Debian&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Debian 11 "bullseye" came out in 2021, supports POWER8, and is likely to be supported until around 2026.  I can't really chime in on more than that because I am a certified Debian hater (even newer releases feel outdated to me), but that looks like a pretty good deal.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Other options&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Those are just some major distros, there's plenty of others, including some Power-specific ones from the community.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Conclusion&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;POWER8's getting old, but is still plenty capable.  Make sure your distro still remembers to send your POWER8 a birthday card each year and you'll have plenty more good times to come.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Russell Currey</dc:creator><pubDate>Wed, 16 Nov 2022 17:30:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2022-11-16:/blog/2022/11/16/what-distro-options-are-there-for-power8-in-2022/</guid><category>OpenPOWER</category><category>linux</category><category>power8</category><category>distro</category></item><item><title>Power kernel hardening features in Linux 6.1</title><link>https://sthbrx.github.io/blog/2022/10/26/power-kernel-hardening-features-in-linux-61/</link><description>&lt;p&gt;Linux 6.1-rc1 was tagged on October 16th, 2022 and includes a bunch of nice things from my team that I want to highlight.  Our goal is to make the Linux kernel running on IBM's Power CPUs more secure, and landed a few goodies upstream in 6.1 to that end.&lt;/p&gt;
&lt;p&gt;Specifically, Linux 6.1 on Power will include &lt;a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7e92e01b724526b98cbc7f03dd4afa0295780d56"&gt;a complete system call infrastructure rework with security &lt;em&gt;and&lt;/em&gt; performance benefits&lt;/a&gt;, &lt;a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a5edf9815dd739fce660b4c8658f61b7d2517042"&gt;support for KFENCE (a low-overhead memory safety error detector)&lt;/a&gt;, and &lt;a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=395cac7752b905318ae454a8b859d4c190485510"&gt;execute-only memory (XOM) support on the Radix MMU&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The syscall work from Rohan McLure and Andrew Donnellan replaces arch/powerpc's legacy infrastructure with the syscall wrapper shared between architectures.  This was a significant overhaul of a lot of legacy code impacting all of powerpc's many platforms, including multiple different ABIs and 32/64bit compatibility infrastructure.  Rohan's series started at &lt;a href="http://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=302791&amp;amp;state=*"&gt;v1 with 6 patches&lt;/a&gt; and ended at &lt;a href="http://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=319348&amp;amp;state=*"&gt;v6 with 25 patches&lt;/a&gt;, and he's done an incredible job at adopting community feedback and handling new problems.&lt;/p&gt;
&lt;p&gt;Big thanks to Christophe Leroy, Arnd Bergmann, Nick Piggin, Michael Ellerman and others for their reviews, and of course Andrew for providing a lot of review and feedback (and prototyping the syscall wrapper in the first place).  Our syscalls have entered the modern era, we can zeroise registers to improve security (but don't yet due to some ongoing discussion around compatibility and making it optional, look out for Linux 6.2), and gain a nice little performance boost by avoiding the allocation of a kernel stack frame.  For more detail, see &lt;a href="http://patchwork.ozlabs.org/project/linuxppc-dev/cover/20220921065605.1051927-1-rmclure@linux.ibm.com/"&gt;Rohan's cover letter&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Next, we have Nicholas Miehlbradt's implementation of &lt;a href="https://www.kernel.org/doc/html/latest/dev-tools/kfence.html"&gt;Kernel Electric Fence (KFENCE)&lt;/a&gt; (and &lt;code&gt;DEBUG_PAGEALLOC&lt;/code&gt;) for 64-bit Power, including the Hash and Radix MMUs.  Christophe Leroy has already implemented KFENCE for 32-bit powerpc upstream and a series adding support for 64-bit was posted by Jordan Niethe last year, but couldn't proceed due to locking issues.  Those issues have since been resolved, and after fixing a previously unknown and very obscure MM issue, Nick's KFENCE patches have been merged.&lt;/p&gt;
&lt;p&gt;KFENCE is a low-overhead alternative to memory detectors like KASAN (&lt;a href="https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?id=41b7a347bf1491e7300563bb224432608b41f62a"&gt;which we implemented for Radix earlier this year, thanks to Daniel Axtens and Paul Mackerras&lt;/a&gt;), which you probably wouldn't want to run in production.  If you're chasing a memory corruption bug that doesn't like to present itself, KFENCE can help you do that for out-of-bounds accesses, use-after-frees, double frees etc without significantly impacting performance.&lt;/p&gt;
&lt;p&gt;Finally, I wired up execute-only memory (XOM) for the Radix MMU.  XOM is a niche feature that lets users map pages with &lt;code&gt;PROT_EXEC&lt;/code&gt; only, creating a page that can't be read or written to, but still executed.  This is primarily useful for defending against code reuse attacks like ROP, but has other uses such as JIT/sandbox environments.  Power8 and later CPUs running the Hash MMU already had this capability through protection keys (pkeys), my implementation for Radix uses the native execute permission bit of the Radix MMU instead.&lt;/p&gt;
&lt;p&gt;This basically took me an afternoon to wire up after I had the idea and I roped in Nicholas Miehlbradt to contribute a &lt;a href="https://github.com/torvalds/linux/blob/master/tools/testing/selftests/powerpc/mm/exec_prot.c"&gt;selftest&lt;/a&gt;, which ended up being a more significant engineering effort than the feature implementation itself.  We now have a comprehensive test for XOM that runs on both Hash and Radix for all possible combinations of R/W/X upstream.&lt;/p&gt;
&lt;p&gt;Anyway, that's all I have - this is my first time writing a post like this, so let me know what you think!  A lot of our work doesn't result in upstream patches so we're not always going to have kernel releases as eventful as this, but we can post summaries every once in a while if there's interest.  Thanks for reading!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Russell Currey</dc:creator><pubDate>Wed, 26 Oct 2022 16:30:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2022-10-26:/blog/2022/10/26/power-kernel-hardening-features-in-linux-61/</guid><category>Development</category><category>linux</category><category>kernel</category><category>hardening</category></item><item><title>Fuzzing grub, part 2: going faster</title><link>https://sthbrx.github.io/blog/2021/06/14/fuzzing-grub-part-2-going-faster/</link><description>&lt;p&gt;Recently a set of 8 vulnerabilities were disclosed for the &lt;a href="https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/GRUB2SecureBootBypass2021"&gt;grub bootloader&lt;/a&gt;. I
found 2 of them (CVE-2021-20225 and CVE-2021-20233), and contributed a number of
other fixes for crashing bugs which we don't believe are exploitable. I found
them by applying fuzz testing to grub. Here's how.&lt;/p&gt;
&lt;p&gt;This is a multi-part series: I think it will end up being 4 posts. I'm hoping to
cover:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="/blog/2021/03/04/fuzzing-grub-part-1"&gt;Part 1: getting started with fuzzing grub&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Part 2 (this post): going faster by doing lots more work&lt;/li&gt;
&lt;li&gt;Part 3: fuzzing filesystems and more&lt;/li&gt;
&lt;li&gt;Part 4: potential next steps and avenues for further work&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;We've been looking at fuzzing &lt;code&gt;grub-emu&lt;/code&gt;, which is basically most parts of grub
built into a standard userspace program. This includes all the script parsing
logic, fonts, graphics, partition tables, filesystems and so on - just not
platform specific driver code or the ability to actually load and boot a kernel.&lt;/p&gt;
&lt;p&gt;Previously, we talked about some issues building grub with AFL++'s
instrumentation:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:::&lt;span class="nv"&gt;text&lt;/span&gt;
.&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;configure&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="nv"&gt;with&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;platform&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;emu&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="nv"&gt;disable&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;grub&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;emu&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;sdl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;CC&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mh"&gt;$AF&lt;/span&gt;&lt;span class="nv"&gt;L_PATH&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;afl&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;cc&lt;/span&gt;
...
&lt;span class="nv"&gt;checking&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;whether&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;target&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;compiler&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;working&lt;/span&gt;...&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;no&lt;/span&gt;
&lt;span class="nv"&gt;configure&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;error&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;cannot&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;compile&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It also doesn't work with &lt;code&gt;afl-gcc&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;We tried to trick configure:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:::shell
./configure --with-platform=emu --disable-grub-emu-sdl CC=clang CXX=clang++
make CC=&amp;quot;$AFL_PATH/afl-cc&amp;quot; 
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Sadly, things still break:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;:::&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nl"&gt;ld:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;disk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;module&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="n"&gt;bss&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;multiple&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;definition&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;`__afl_global_area_ptr&lt;/span&gt;&lt;span class="p"&gt;&amp;#39;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nl"&gt;exec:&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="n"&gt;bss&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0&lt;/span&gt;&lt;span class="n"&gt;xe078&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;first&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;defined&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;here&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nl"&gt;ld:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;regexp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;module&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="n"&gt;bss&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x70&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;multiple&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;definition&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;`__afl_global_area_ptr&lt;/span&gt;&lt;span class="p"&gt;&amp;#39;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nl"&gt;exec:&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="n"&gt;bss&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0&lt;/span&gt;&lt;span class="n"&gt;xe078&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;first&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;defined&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;here&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nl"&gt;ld:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;blocklist&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;module&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="n"&gt;bss&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x28&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;multiple&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;definition&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;`__afl_global_area_ptr&lt;/span&gt;&lt;span class="p"&gt;&amp;#39;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nl"&gt;exec:&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="n"&gt;bss&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0&lt;/span&gt;&lt;span class="n"&gt;xe078&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;first&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;defined&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;here&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The problem is the module linkage that I talked about in &lt;a href="/blog/2021/03/04/fuzzing-grub-part-1"&gt;part 1&lt;/a&gt;.
There is a link stage of sorts for the kernel (&lt;code&gt;kernel.exec&lt;/code&gt;) and each module
(e.g. &lt;code&gt;disk.module&lt;/code&gt;), so some AFL support code gets linked into each of
those. Then there's another link stage for &lt;code&gt;grub-emu&lt;/code&gt; itself, which also tries
to bring in the same support code. The linker doesn't like the symbols being in
multiple places, which is fair enough.&lt;/p&gt;
&lt;p&gt;There are (at least) 3 ways you could solve this. I'm going to call them the hard way, and the ugly way and the easy way.&lt;/p&gt;
&lt;h2&gt;The hard way: messing with makefiles&lt;/h2&gt;
&lt;p&gt;We've been looking at fuzzing &lt;code&gt;grub-emu&lt;/code&gt;. Building &lt;code&gt;grub-emu&lt;/code&gt; links
&lt;code&gt;kernel.exec&lt;/code&gt; and almost every &lt;code&gt;.module&lt;/code&gt; file that grub produces into the
final binary. Maybe we could avoid our duplicate symbol problems entirely by
changing how we build things?&lt;/p&gt;
&lt;p&gt;I didn't do this in my early work because, to be honest, I don't like working
with build systems and I'm not especially good at it. grub's build system is
based on autotools but is even more quirky than usual: rather than just having a
&lt;code&gt;Makefile.am&lt;/code&gt;, we have &lt;code&gt;Makefile.core.def&lt;/code&gt; which is used along with other things
to generate &lt;code&gt;Makefile.am&lt;/code&gt;. It's a pretty cool system for making modules, but
it's not my idea of fun to work with.&lt;/p&gt;
&lt;p&gt;But, for the sake of completeness, I tried again.&lt;/p&gt;
&lt;p&gt;It gets unpleasant quickly. The generated &lt;code&gt;grub-core/Makefile.core.am&lt;/code&gt; adds each module to &lt;code&gt;platform_PROGRAMS&lt;/code&gt;, and then each is built with &lt;code&gt;LDFLAGS_MODULE = $(LDFLAGS_PLATFORM) -nostdlib $(TARGET_LDFLAGS_OLDMAGIC) -Wl,-r,-d&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Basically, in the makefile this ends up being (e.g.):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="err"&gt;:::make&lt;/span&gt;
&lt;span class="nf"&gt;tar.module$(EXEEXT)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nv"&gt;tar_module_OBJECTS&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt; &lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nv"&gt;tar_module_DEPENDENCIES&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt; &lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nv"&gt;EXTRA_tar_module_DEPENDENCIES&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt; 
&lt;span class="w"&gt;    &lt;/span&gt;@rm&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;tar.module&lt;span class="k"&gt;$(&lt;/span&gt;EXEEXT&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;AM_V_CCLD&lt;span class="k"&gt;)$(&lt;/span&gt;tar_module_LINK&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;tar_module_OBJECTS&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;tar_module_LDADD&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;LIBS&lt;span class="k"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Ideally I don't want them to be linked at all; there's no benefit if
they're just going to be linked again.&lt;/p&gt;
&lt;p&gt;You can't just collect the sources and build them into &lt;code&gt;grub-emu&lt;/code&gt; - they all
have to built with different &lt;code&gt;CFLAGS&lt;/code&gt;! So instead I spent some hours messing
around with the build system. Given some changes to the python script that
converts the &lt;code&gt;Makefile.*.def&lt;/code&gt; files into &lt;code&gt;Makefile.am&lt;/code&gt; files, plus some other
bits and pieces, we can build &lt;code&gt;grub-emu&lt;/code&gt; by linking the object files rather than
the more-processed modules.&lt;/p&gt;
&lt;p&gt;The build dies immediately after linking &lt;code&gt;grub-emu&lt;/code&gt; in other components, and it
requires a bit of manual intervention to get the right things built in the right
order, but with all of those caveats, it's enough. It works, and you can turn on
things like ASAN, but getting there was hard, unrewarding and unpleasant. Let's
consider alternative ways to solve this problem.&lt;/p&gt;
&lt;h2&gt;The ugly way: patching AFL&lt;/h2&gt;
&lt;p&gt;What I did when finding the bugs was to observe that we only wanted AFL to link
in its extra instrumentation at certain points of the build process. So I
patched AFL to add an environment variable &lt;code&gt;AFL_DEFER_LIB&lt;/code&gt; - which prevented AFL
adding its own instrumentation library when being called as a linker. I combined
this with the older CFG instrumentation, as the PCGUARD instrumentation brought
in a bunch of symbols from LLVM which I didn't want to also figure out how to
guard.&lt;/p&gt;
&lt;p&gt;I then wrapped this in a horrifying script that basically built bits and pieces
of grub with the environment variable on or off, in order to at least get the
userspace tools and &lt;code&gt;grub-emu&lt;/code&gt; built. Basically it set &lt;code&gt;AFL_DEFER_LIB&lt;/code&gt; when
building all the modules and turned it off when building the userspace tools
and &lt;code&gt;grub-emu&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;This worked and it's what I used to find most of my bugs. But I'd probably not
recommend it, and I'm not sharing the source: it's extremely fragile and
brittle, the hard way is more generally applicable, and the easy way is nicer.&lt;/p&gt;
&lt;h2&gt;The easy way: adjusting linker flags&lt;/h2&gt;
&lt;p&gt;After posting part 1 of this series, I had a fascinating twitter DM conversation
with &lt;a href="https://twitter.com/hackerschoice"&gt;@hackerschoice&lt;/a&gt;, who pointed me to some
new work that had been done in AFL++ between when I started and when I published
part 1.&lt;/p&gt;
&lt;p&gt;AFL++ now has the ability to dynamically detect some duplicate symbols, allowing
it to support plugins and modules better. This isn't directly applicable because
we link all the modules in at build time, but in the conversation I was pointed
to a linker flag which instructs the linker to ignore the symbol duplication
rather than error out. This provides a significantly simpler way to instrument
&lt;code&gt;grub-emu&lt;/code&gt;, avoiding all the issues I'd previously been fighting so hard to
address.&lt;/p&gt;
&lt;p&gt;So, with a modern AFL++, and the patch from &lt;a href="https://github.com/daxtens/grub/tree/fuzzing-pt2"&gt;part 1&lt;/a&gt;,
you can sort out this entire process like this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:::shell
./bootstrap
./configure --with-platform=emu CC=clang CXX=clang++ --disable-grub-emu-sdl
make CC=/path/to/afl-clang-fast LDFLAGS=&amp;quot;-Wl,--allow-multiple-definition&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Eventually it will error out, but &lt;code&gt;./grub-core/grub-emu&lt;/code&gt; should be successfully
built first.&lt;/p&gt;
&lt;p&gt;(Why not just build &lt;code&gt;grub-emu&lt;/code&gt; directly? It gets built by &lt;code&gt;grub-core/Makefile&lt;/code&gt;,
but depends on a bunch of things made by the top-level makefile and doesn't
express its dependencies well. So you can try to build all the things that you
need separately and then &lt;code&gt;cd grub-core; make ...flags... grub-emu&lt;/code&gt; if you want -
but it's way more complicated to do it that way!)&lt;/p&gt;
&lt;h1&gt;Going extra fast: &lt;code&gt;__AFL_INIT&lt;/code&gt;&lt;/h1&gt;
&lt;p&gt;Now that we can compile with instrumentation, we can use &lt;code&gt;__AFL_INIT&lt;/code&gt;. I'll
leave the precise details of how this works to the AFL docs, but in short it
allows us to do a bunch of early setup only once, and just fork the process
after the setup is done.&lt;/p&gt;
&lt;p&gt;There's a patch that inserts a call to &lt;code&gt;__AFL_INIT&lt;/code&gt; in the &lt;code&gt;grub-emu&lt;/code&gt; start path
in &lt;a href="https://github.com/daxtens/grub/tree/fuzzing-pt2"&gt;my GitHub repo&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;All up, this can lead to a 2x-3x speedup over the figures I saw in &lt;a href="/blog/2021/03/04/fuzzing-grub-part-1"&gt;part 1&lt;/a&gt;.
In part 1 we saw around 244 executions per second at this point - now we're over
500:&lt;/p&gt;
&lt;p&gt;&lt;img alt="afl-fuzz fuzzing grub, showing fuzzing happening" src="/images/dja/grub-fuzzing-pt2.png"&gt;&lt;/p&gt;
&lt;h1&gt;Finding more bugs with sanitisers&lt;/h1&gt;
&lt;p&gt;A 'sanitizer' refers to a set of checks inserted by a compiler at build time to
detect various runtime issues that might not cause a crash or otherwise be
detected. A particularly common and useful sanitizer is ASAN, the
&lt;a href="https://clang.llvm.org/docs/AddressSanitizer.html"&gt;AddressSanitizer&lt;/a&gt;, which
detects out-of-bounds memory accesses, use-after-frees and other assorted memory
bugs. Other sanitisers can check for
&lt;a href="https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html"&gt;undefined behaviour&lt;/a&gt;,
&lt;a href="https://clang.llvm.org/docs/MemorySanitizer.html"&gt;uninitialised memory reads&lt;/a&gt;
or even breaches of
&lt;a href="https://releases.llvm.org/12.0.0/tools/clang/docs/ControlFlowIntegrity.html"&gt;control flow integrity&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;ASAN is particularly popular for fuzzing. In theory, compiling with AFL++ and
LLVM makes it really easy to compile with ASAN. Setting &lt;code&gt;AFL_USE_ASAN=1&lt;/code&gt; should
be sufficient.&lt;/p&gt;
&lt;p&gt;However, in practice, it's quite fragile for grub. I believe I had it all
working, and then I upgraded my distro, LLVM and AFL++ versions, and everything
stopped working. (It's possible that I hadn't sufficiently cleaned my source
tree and I was still building based on the hard way? Who knows.)&lt;/p&gt;
&lt;p&gt;I spent quite a while fighting "truncated relocations". ASAN instrumentation was
bloating the binaries, and the size of all the &lt;code&gt;*.module&lt;/code&gt; files was over 512MB,
which I suspect was causing the issues. (Without ASAN, it only comes to 35MB.)&lt;/p&gt;
&lt;p&gt;I tried &lt;code&gt;afl-clang-lto&lt;/code&gt;: I installed &lt;code&gt;lld&lt;/code&gt;, rebuilt AFL++, and managed to
segfault the linker while building grub. So I wrote that off. Changing the
instrumentation type to classic didn't help either.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://www.technovelty.org/c/relocation-truncated-to-fit-wtf.html"&gt;Some googling&lt;/a&gt;
suggested GCC's &lt;code&gt;-mmodel&lt;/code&gt;, which in Clang seems to be &lt;code&gt;-mcmodel&lt;/code&gt;, but
&lt;code&gt;CFLAGS="-mcmodel=large"&lt;/code&gt; didn't get me any further either: it's already added
in a few different links.&lt;/p&gt;
&lt;p&gt;My default llvm is llvm-12, so I tried building with llvm-9 and llvm-11 in case
that helped. Both built a binary, but it would fail to start:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;:::&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;
&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="mi"&gt;375638&lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="n"&gt;AddressSanitizer&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;CHECK&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;failed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;build&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;llvm&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;toolchain&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="n"&gt;fovFY&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;llvm&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;toolchain&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;9.0&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;compiler&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rt&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;lib&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;sanitizer_common&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;sanitizer_common_libcdep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cc&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;23&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;((SoftRssLimitExceededCallback)) == ((nullptr))&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mh"&gt;0x423660&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The same happens if I build with llvm-12 and &lt;code&gt;afl-clang&lt;/code&gt;, the old-style
instrumentation.&lt;/p&gt;
&lt;p&gt;I spun up a Ubuntu 20.04 VM and build there with LLVM 10 and the latest stable
AFL++. That didn't work either.&lt;/p&gt;
&lt;p&gt;I had much better luck using GCC's and GCC's ASAN implementation, either with
the old-school &lt;code&gt;afl-gcc&lt;/code&gt; or the newer GCC plugin-based &lt;code&gt;afl-gcc-fast&lt;/code&gt;. (I have
some hypotheses around shared library vs static library ASAN, but having spent
way more work time on this than was reasonable, I was disinclined to debug it
further.) Here's what worked for me:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;:::&lt;/span&gt;&lt;span class="n"&gt;shell&lt;/span&gt;
&lt;span class="o"&gt;./&lt;/span&gt;&lt;span class="n"&gt;configure&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="n"&gt;with&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;platform&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;emu&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="n"&gt;disable&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;grub&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;emu&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;sdl&lt;/span&gt;
&lt;span class="c1"&gt;# the ASAN option is required because one of the tools leaks memory and&lt;/span&gt;
&lt;span class="c1"&gt;# that breaks the generation of documentation.&lt;/span&gt;
&lt;span class="c1"&gt;# -Wno-nested-extern makes __AFL_INIT work on gcc&lt;/span&gt;
&lt;span class="n"&gt;ASAN_OPTIONS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;detect_leaks&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;AFL_USE_ASAN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;make&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;CC&lt;/span&gt;&lt;span class="o"&gt;=/&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;afl&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;gcc&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;fast&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;LDFLAGS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-Wl,--allow-multiple-definition&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;CFLAGS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-Wno-nested-extern&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;GCC doesn't support as many sanitisers as LLVM, but happily it does support
ASAN. AFL++'s GCC plugin mode should get us most of the speed we would get from
LLVM, and indeed the speed - even with ASAN - is quite acceptable.&lt;/p&gt;
&lt;p&gt;If you persist, you should be able to find some more bugs: for example there's a
very boring global array out-of-bounds read when parsing config files.&lt;/p&gt;
&lt;p&gt;That's all for part 2. In part 3 we'll look at fuzzing filesystems and
more. Hopefully there will be a quicker turnaround between part 2 and part 3
than there was between part 1 and part 2!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Mon, 14 Jun 2021 17:10:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2021-06-14:/blog/2021/06/14/fuzzing-grub-part-2-going-faster/</guid><category>Development</category><category>testing</category></item><item><title>Fuzzing grub: part 1</title><link>https://sthbrx.github.io/blog/2021/03/04/fuzzing-grub-part-1/</link><description>&lt;p&gt;Recently a set of 8 vulnerabilities were disclosed for the &lt;a href="https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/GRUB2SecureBootBypass2021"&gt;grub bootloader&lt;/a&gt;. I
found 2 of them (CVE-2021-20225 and CVE-2021-20233), and contributed a number of
other fixes for crashing bugs which we don't believe are exploitable. I found
them by applying fuzz testing to grub. Here's how.&lt;/p&gt;
&lt;p&gt;This is a multi-part series: I think it will end up being 4 posts. I'm hoping to
cover:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Part 1 (this post): getting started with fuzzing grub&lt;/li&gt;
&lt;li&gt;Part 2: &lt;a href="/blog/2021/06/14/fuzzing-grub-part-2-going-faster/"&gt;going faster by doing lots more work&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Part 3: fuzzing filesystems and more&lt;/li&gt;
&lt;li&gt;Part 4: potential next steps and avenues for further work&lt;/li&gt;
&lt;/ul&gt;
&lt;h1&gt;Fuzz testing&lt;/h1&gt;
&lt;p&gt;Let's begin with part one: getting started with fuzzing grub.&lt;/p&gt;
&lt;p&gt;One of my all-time favourite techniques for testing programs, especially
programs that handle untrusted input, and especially-especially programs written
in C that parse untrusted input, is fuzz testing. Fuzz testing (or fuzzing) is
the process of repeatedly throwing randomised data at your program under test
and seeing what it does.&lt;/p&gt;
&lt;p&gt;(For the secure boot threat model, untrusted input is anything not validated by
a cryptographic signature - so config files are untrusted for our purposes, but
grub modules can only be loaded if they are signed, so they are trusted.)&lt;/p&gt;
&lt;p&gt;Fuzzing has a long history and has recently received a new lease on life with
coverage-guided fuzzing tools like &lt;a href="https://lcamtuf.coredump.cx/afl/"&gt;AFL&lt;/a&gt; and
more recently &lt;a href="https://aflplus.plus/"&gt;AFL++&lt;/a&gt;.&lt;/p&gt;
&lt;h1&gt;Building grub for AFL++&lt;/h1&gt;
&lt;p&gt;AFL++ is extremely easy to use ... if your program:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;is built as a single binary with a regular tool-chain&lt;/li&gt;
&lt;li&gt;runs as a regular user-space program on Linux&lt;/li&gt;
&lt;li&gt;reads a small input files from disk and then exits&lt;/li&gt;
&lt;li&gt;doesn't do anything fancy with threads or signals&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Beyond that, it gets a bit more complex.&lt;/p&gt;
&lt;p&gt;On the face of it, grub fails 3 of these 4 criteria:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;grub is a highly modular program: it loads almost all of its functionality as
   modules which are linked as separate ELF relocatable files. (Not runnable
   programs, but not shared libraries either.)&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;grub usually runs as a bootloader, not as a regular app.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;grub reads all sorts of things, ranging in size from small files to full
   disks. After loading most things, it returns to a command prompt rather than
   exiting.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Fortunately, these problems are not insurmountable.&lt;/p&gt;
&lt;p&gt;We'll start with the 'running as a bootloader' problem. Here, grub helps us out
a bit, because it provides an 'emulator' target, which runs most of grub
functionality as a userspace program. It doesn't support actually booting
anything (unsurprisingly) but it does support most other modules, including
things like the config file parser.&lt;/p&gt;
&lt;p&gt;We can configure grub to build the emulator. We disable the graphical frontend
for now.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:::shell
./bootstrap
./configure --with-platform=emu --disable-grub-emu-sdl
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;At this point in building a fuzzing target, we'd normally try to configure with
&lt;code&gt;afl-cc&lt;/code&gt; to get the instrumentation that makes AFL(++) so powerful. However, the
grub configure script is not a fan:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:::&lt;span class="nv"&gt;text&lt;/span&gt;
.&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;configure&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="nv"&gt;with&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;platform&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;emu&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="nv"&gt;disable&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;grub&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;emu&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;sdl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;CC&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mh"&gt;$AF&lt;/span&gt;&lt;span class="nv"&gt;L_PATH&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;afl&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;cc&lt;/span&gt;
...
&lt;span class="nv"&gt;checking&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;whether&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;target&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;compiler&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;working&lt;/span&gt;...&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;no&lt;/span&gt;
&lt;span class="nv"&gt;configure&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;error&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;cannot&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;compile&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It also doesn't work with &lt;code&gt;afl-gcc&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Hmm, ok, so what if we just... lie a bit?&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:::shell
./configure --with-platform=emu --disable-grub-emu-sdl
make CC=&amp;quot;$AFL_PATH/afl-gcc&amp;quot; 
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;(Normally I'd use &lt;code&gt;CC=clang&lt;/code&gt; and &lt;code&gt;afl-cc&lt;/code&gt;, but clang support is slightly broken
upstream at the moment.)&lt;/p&gt;
&lt;p&gt;After a small fix for gcc-10 compatibility, we get the userspace tools
(potentially handy!) but a bunch of link errors for &lt;code&gt;grub-emu&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;:::&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nl"&gt;ld:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;disk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;module&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="n"&gt;bss&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;multiple&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;definition&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;`__afl_global_area_ptr&lt;/span&gt;&lt;span class="p"&gt;&amp;#39;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nl"&gt;exec:&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="n"&gt;bss&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0&lt;/span&gt;&lt;span class="n"&gt;xe078&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;first&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;defined&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;here&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nl"&gt;ld:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;regexp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;module&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="n"&gt;bss&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x70&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;multiple&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;definition&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;`__afl_global_area_ptr&lt;/span&gt;&lt;span class="p"&gt;&amp;#39;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nl"&gt;exec:&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="n"&gt;bss&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0&lt;/span&gt;&lt;span class="n"&gt;xe078&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;first&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;defined&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;here&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nl"&gt;ld:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;blocklist&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;module&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="n"&gt;bss&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x28&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;multiple&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;definition&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;`__afl_global_area_ptr&lt;/span&gt;&lt;span class="p"&gt;&amp;#39;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nl"&gt;exec:&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="n"&gt;bss&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0&lt;/span&gt;&lt;span class="n"&gt;xe078&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;first&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;defined&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;here&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The problem is the module linkage that I talked about earlier: because there is
a link stage of sorts for each module, some AFL support code gets linked in to
both the grub kernel (&lt;code&gt;kernel.exec&lt;/code&gt;) and each module (here &lt;code&gt;disk.module&lt;/code&gt;,
&lt;code&gt;regexp.module&lt;/code&gt;, ...). The linker doesn't like it being in both, which is fair
enough.&lt;/p&gt;
&lt;p&gt;To get started, let's instead take advantage of the smarts of AFL++ using Qemu
mode instead. This builds a specially instrumented qemu user-mode emulator
that's capable of doing coverage-guided fuzzing on uninstrumented binaries at
the cost of a significant performance penalty.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:::shell
make clean
make
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now we have a grub-emu binary. If you run it directly, you'll pick up your
system boot configuration, but the &lt;code&gt;-d&lt;/code&gt; option can point it to a directory of
your choosing. Let's set up one for fuzzing:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:::shell
mkdir stage
echo &amp;quot;echo Hello sthbrx readers&amp;quot; &amp;gt; stage/grub.cfg
cd stage
../grub-core/grub-emu -d .
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You probably won't see the message because the screen gets blanked at the end of
running the config file, but if you pipe it through &lt;code&gt;less&lt;/code&gt; or something you'll
see it.&lt;/p&gt;
&lt;h1&gt;Running the fuzzer&lt;/h1&gt;
&lt;p&gt;So, that seems to work - let's create a test input and try fuzzing:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:::&lt;span class="nv"&gt;shell&lt;/span&gt;
&lt;span class="nv"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;..
&lt;span class="nv"&gt;mkdir&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;in&lt;/span&gt;
&lt;span class="nv"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;echo hi&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;in&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;echo&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;hi&lt;/span&gt;

&lt;span class="nv"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;stage&lt;/span&gt;
#&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;Q&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;qemu&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;mode&lt;/span&gt;
#&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;M&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;main&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;fuzzer&lt;/span&gt;
#&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;d&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;don&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;t do deterministic steps (too slow for a text format)&lt;/span&gt;
&lt;span class="err"&gt;# -f create file grub.cfg&lt;/span&gt;
&lt;span class="err"&gt;$AFL_PATH/afl-fuzz -Q -i ../in -o ../out -M main -d -- ../grub-core/grub-emu -d .&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Sadly:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c"&gt;:::text&lt;/span&gt;
&lt;span class="k"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;-&lt;/span&gt;&lt;span class="k"&gt;]&lt;/span&gt;&lt;span class="c"&gt; The program took more than 1000 ms to process one of the initial test cases&lt;/span&gt;&lt;span class="nt"&gt;.&lt;/span&gt;
&lt;span class="c"&gt;    This is bad news; raising the limit with the &lt;/span&gt;&lt;span class="nb"&gt;-&lt;/span&gt;&lt;span class="c"&gt;t option is possible&lt;/span&gt;&lt;span class="nt"&gt;,&lt;/span&gt;&lt;span class="c"&gt; but&lt;/span&gt;
&lt;span class="c"&gt;    will probably make the fuzzing process extremely slow&lt;/span&gt;&lt;span class="nt"&gt;.&lt;/span&gt;

&lt;span class="c"&gt;    If this test case is just a fluke&lt;/span&gt;&lt;span class="nt"&gt;,&lt;/span&gt;&lt;span class="c"&gt; the other option is to just avoid it&lt;/span&gt;
&lt;span class="c"&gt;    altogether&lt;/span&gt;&lt;span class="nt"&gt;,&lt;/span&gt;&lt;span class="c"&gt; and find one that is less of a CPU hog&lt;/span&gt;&lt;span class="nt"&gt;.&lt;/span&gt;

&lt;span class="k"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;-&lt;/span&gt;&lt;span class="k"&gt;]&lt;/span&gt;&lt;span class="c"&gt; PROGRAM ABORT : Test case &amp;#39;id:000000&lt;/span&gt;&lt;span class="nt"&gt;,&lt;/span&gt;&lt;span class="c"&gt;time:0&lt;/span&gt;&lt;span class="nt"&gt;,&lt;/span&gt;&lt;span class="c"&gt;orig:echo&lt;/span&gt;&lt;span class="nb"&gt;-&lt;/span&gt;&lt;span class="c"&gt;hi&amp;#39; results in a timeout&lt;/span&gt;
&lt;span class="c"&gt;         Location : perform_dry_run()&lt;/span&gt;&lt;span class="nt"&gt;,&lt;/span&gt;&lt;span class="c"&gt; src/afl&lt;/span&gt;&lt;span class="nb"&gt;-&lt;/span&gt;&lt;span class="c"&gt;fuzz&lt;/span&gt;&lt;span class="nb"&gt;-&lt;/span&gt;&lt;span class="c"&gt;init&lt;/span&gt;&lt;span class="nt"&gt;.&lt;/span&gt;&lt;span class="c"&gt;c:866&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;What we're seeing here (and indeed what you can observe if you run &lt;code&gt;grub-emu&lt;/code&gt;
directly) is that &lt;code&gt;grub-emu&lt;/code&gt; isn't exiting when it's done. It's waiting for more
input, and will keep waiting for input until it's killed by &lt;code&gt;afl-fuzz&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;We need to patch grub to sort that out. It's on &lt;a href="https://github.com/daxtens/grub/commit/ad2e84224e674eb1f9dcd8efc3d8efe78ed62bec"&gt;my GitHub&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Apply that, rebuild with &lt;code&gt;FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION&lt;/code&gt;, and voila:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:::shell
cd ..
make CFLAGS=&amp;quot;-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION&amp;quot;
cd stage
$AFL_PATH/afl-fuzz -Q -i ../in -o ../out -M main -d -f grub.cfg -- ../grub-core/grub-emu -d .
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And fuzzing is happening!&lt;/p&gt;
&lt;p&gt;&lt;img alt="afl-fuzz fuzzing grub, showing fuzzing happening" src="/images/dja/grub-fuzzing-pt1.png"&gt;&lt;/p&gt;
&lt;p&gt;This is enough to find some of the (now-fixed) bugs in the grub config file
parsing!&lt;/p&gt;
&lt;h1&gt;Fuzzing beyond the config file&lt;/h1&gt;
&lt;p&gt;You can also extend this to fuzzing other things that don't require the
graphical UI, such as grub's transparent decompression support:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:::shell
cd ..
rm -rf in out stage
mkdir in stage
echo hi &amp;gt; in/hi
gzip in/hi
cd stage
echo &amp;quot;cat thefile&amp;quot; &amp;gt; grub.cfg
$AFL_PATH/afl-fuzz -Q -i ../in -o ../out -M main -f thefile -- ../grub-core/grub-emu -d .
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You should be able to find a hang pretty quickly with this, an as-yet-unfixed
bug where grub will print output forever from a corrupt file: (your mileage may
vary, as will the paths.)&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;shell&lt;/span&gt;
&lt;span class="nx"&gt;cp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;main&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;hangs&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;000000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;src&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;000000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;time&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;43383&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;op&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;havoc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;rep&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;thefile&lt;/span&gt;
&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;grub&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;core&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;grub&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;emu&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;d&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;less&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;observe&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;this&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;going&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;forever&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;zcat&lt;/code&gt;, on the other hand, reports it as simply corrupt:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:::text
$ zcat thefile

gzip: thefile: invalid compressed data--format violated
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;(Feel free to fix that and send a patch to the list!)&lt;/p&gt;
&lt;p&gt;That wraps up part 1. Eventually I'll be back with part 2, where I explain the
hoops to jump through to go faster with the &lt;code&gt;afl-cc&lt;/code&gt; instrumentation.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Thu, 04 Mar 2021 17:10:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2021-03-04:/blog/2021/03/04/fuzzing-grub-part-1/</guid><category>Development</category><category>testing</category></item><item><title>linux.conf.au 2020 recap</title><link>https://sthbrx.github.io/blog/2020/01/22/linuxconfau-2020-recap/</link><description>&lt;p&gt;It's that time of year again. Most of OzLabs headed up to the Gold Coast for linux.conf.au 2020.&lt;/p&gt;
&lt;p&gt;linux.conf.au is one of the longest-running community-led Linux and Free Software events in the world, and attracts a crowd from Australia, New Zealand and much further afield. OzLabbers have been involved in LCA since the very beginning and this year was no exception with myself running the Kernel Miniconf and several others speaking.&lt;/p&gt;
&lt;p&gt;The list below contains some of our highlights that we think you should check out. This is just a few of the talks that we managed to make it to - there's plenty more worthwhile stuff on the &lt;a href="https://www.youtube.com/user/linuxconfau2019/featured"&gt;linux.conf.au YouTube channel&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;We'll see you all at LCA2021 right here in Canberra...&lt;/p&gt;
&lt;h1&gt;Keynotes&lt;/h1&gt;
&lt;p&gt;A couple of the keynotes really stood out:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=Yv4tI6939q0"&gt;Sean Brady - Drop Your Tools - Does Expertise have a Dark Side?&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Sean is a forensic structural engineer who shows us a variety of examples, from structural collapses and firefighting disasters, where trained professionals were blinded by their expertise and couldn't bring themselves to do things that were obvious.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=xfpy5c3thQo"&gt;Vanessa Teague - Who cares about Democracy?&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There's nothing quite like cryptography proofs presented to a keynote audience at 9:30 in the morning. Vanessa goes over the issues with electronic voting systems in Australia, and especially internet voting as used in NSW, including flaws in their implementation of cryptographic algorithms. There continues to be no good way to do internet voting, but with developments in methodologies like risk-limiting audits there may be reasonably safe ways to do in-person electronic voting.&lt;/p&gt;
&lt;h1&gt;OpenPOWER&lt;/h1&gt;
&lt;p&gt;There was an OpenISA miniconf, co-organised by none other than Hugh Blemings of the OpenPOWER Foundation.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=RU6RPYAqFzE"&gt;Michael Neuling / Anton Blanchard - Power OpenISA and Microwatt Introduction&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Anton (on Mikey's behalf) introduces the Power OpenISA and the Microwatt FPGA core which has been released to go with it.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=g3slH03MCmo"&gt;Anton Blanchard - Build your own Open Hardware CPU in 25 minutes or less&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Anton live demos Microwatt in simulation, and also tries to synthesise it for his FPGA but runs out of time...&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=JkDx_y0onSk"&gt;Paul Mackerras - Microwatt Microarchitecture&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Paul presents an in-depth overview of the design of the Microwatt core.&lt;/p&gt;
&lt;h1&gt;Kernel&lt;/h1&gt;
&lt;p&gt;There were quite a few kernel talks, both in the &lt;a href="http://lca-kernel.ozlabs.org"&gt;Kernel Miniconf&lt;/a&gt; and throughout the main conference. These are just some of them:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=ggD-eb3yPVs"&gt;Aleksa Sarai - Designing Extensible Syscalls&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There's been many cases where we've introduced a syscall only to find out later on that we need to add some new parameters - how do we make our syscalls extensible so we can add new parameters later on without needing to define a whole new syscall, while maintaining both forward and backward compatibility? It turns out it's pretty simple but needs a few more kernel helpers.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=heib48KG-YQ"&gt;Russell Currey - Kernel hacking like it's 2020&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There are a bunch of tools out there which you can use to make your kernel hacking experience much more pleasant. You should use them.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=tGseJW_uBB8"&gt;Aleksa Sarai - Securing Container Runtimes - How Hard Can It Be?&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Among other security issues with container runtimes, using procfs to setup security controls during the startup of a container is fraught with hilarious problems, because procfs and the Linux filesystem API aren't really designed to do this safely, and also have a bunch of amusing bugs.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=0Bj6W7qrOOI"&gt;Kees Cook - Control Flow Integrity in the Linux Kernel&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Control Flow Integrity is a technique for restricting exploit techniques that hijack a program's control flow (e.g. by overwriting a return address on the stack (ROP), or overwriting a function pointer that's used in an indirect jump). Kees goes through the current state of CFI supporting features in hardware and what is currently available to enable CFI in the kernel.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=p5u-vbwu3Fs"&gt;Matthew Wilcox - Large Pages in Linux&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Linux has supported huge pages for many years, which has significantly improved CPU performance. However, the huge page mechanism was driven by hardware advancements and is somewhat inflexible, and it's just as important to consider software overhead. Matthew has been working on supporting more flexible "large pages" in the page cache to do just that.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=9Fzd6MapG3Y"&gt;Russell Currey - The magical fantasy land of Linux kernel testing&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Spoiler: the magical fantasy land is a trap.&lt;/p&gt;
&lt;h1&gt;Community&lt;/h1&gt;
&lt;p&gt;Lots of community and ethics discussion this year - one talk which stood out to me:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=n55WClalwHo"&gt;Bradley M. Kuhn / Karen Sandler - Open Source Won, but Software Freedom Hasn't Yet: A Guide &amp;amp; Commisseration Session for FOSS activists&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Bradley and Karen argue that while open source has "won", software freedom has regressed in recent years, and present their vision for what modern, pragmatic Free Software activism should look like.&lt;/p&gt;
&lt;h1&gt;Other&lt;/h1&gt;
&lt;p&gt;Among the variety of other technical talks at LCA...&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=L2P501Iy6J8"&gt;Matthew Treinish - Building a Compiler for Quantum Computers&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Quantum compilers are not really like regular classical compilers (indeed, they're really closer to FPGA synthesis tools). Matthew talks through how quantum compilers map a program on to IBM's quantum hardware and the types of optimisations they apply.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=Dk6ZuydQt9I"&gt;Fraser Tweedale - Clevis and Tang: securing your secrets at rest&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Clevis and Tang provide an implementation of "network bound encryption", allowing you to magically decrypt your secrets when you are on a secure network with access to the appropriate Tang servers. This talk outlines use cases and provides a demonstration.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=uBBaVtHkiOI"&gt;Christoph Lameter - How to capture 100G Ethernet traffic at wire speed to local disk&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Christoph discusses how to deal with the hardware and software limitations that make it difficult to capture traffic at wire speed on fast fibre networks.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Donnellan</dc:creator><pubDate>Wed, 22 Jan 2020 16:00:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2020-01-22:/blog/2020/01/22/linuxconfau-2020-recap/</guid><category>Education</category><category>conferences</category></item><item><title>rfid and hrfid</title><link>https://sthbrx.github.io/blog/2019/12/18/rfid-and-hrfid/</link><description>&lt;p&gt;I was staring at some assembly recently, and for not the first time encountered &lt;em&gt;rfid&lt;/em&gt; and &lt;em&gt;hrfid&lt;/em&gt;, two instructions that we use when doing things like returning to userspace, returning from OPAL to the kernel, or from a host kernel into a guest.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;rfid&lt;/em&gt; copies various bits from the register SRR1 (Machine Status Save/Restore Register 1) into the MSR (Machine State Register), and then jumps to an address given in SRR0 (Machine Status Save/Restore Register 0). &lt;em&gt;hrfid&lt;/em&gt; does something similar, using HSRR0 and HSRR1 (Hypervisor Machine Status Save/Restore Registers 0/1), and slightly different handling of MSR bits.&lt;/p&gt;
&lt;p&gt;The various Save/Restore Registers are used to preserve the state of the CPU before jumping to an interrupt handler, entering the kernel, etc, and are set up as part of instructions like &lt;em&gt;sc&lt;/em&gt; (System Call), by the interrupt mechanism, or manually (using instructions like &lt;em&gt;mtsrr1&lt;/em&gt;).&lt;/p&gt;
&lt;p&gt;Anyway, the way in which &lt;em&gt;rfid&lt;/em&gt; and &lt;em&gt;hrfid&lt;/em&gt; restores MSR bits is documented somewhat obtusely in the ISA (if you don't believe me, look it up), and I was annoyed by this, so here, have a more useful definition. Leave a comment if I got something wrong.&lt;/p&gt;
&lt;h1&gt;rfid - Return From Interrupt Doubleword&lt;/h1&gt;
&lt;h2&gt;Machine State Register&lt;/h2&gt;
&lt;p&gt;Copy all bits (except some reserved bits) from SRR1 into the MSR, with the following exceptions:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;MSR_3 (HV, Hypervisor State) = MSR_3 &amp;amp; SRR1_3&lt;br&gt;
[We won't put the thread into hypervisor state if we're not already in hypervisor state]&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;If MSR_29:31 != 0b010 [Transaction State Suspended, TM not available], or SRR1_29:31 != 0b000 [Transaction State Non-transactional, TM not available] then:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;MSR_29:30 (TS, Transaction State) = SRR1_29:30&lt;/li&gt;
&lt;li&gt;MSR_31 (TM, Transactional Memory Available) = SRR1_31&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;[See the ISA description for explanation on how rfid interacts with TM and resulting interrupts]&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;MSR_48 (EE, External Interrupt Enable) = SRR1_48 | SRR1_49 (PR, Problem State)&lt;br&gt;
[If going into problem state, external interrupts will be enabled]&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;MSR_51 (ME, Machine Check Interrupt Enable) = (MSR_3 (HV, Hypervisor State) &amp;amp; SRR1_51) | ((! MSR_3) &amp;amp; MSR_51)&lt;br&gt;
[If we're not already in hypervisor state, we won't alter ME]&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;MSR_58 (IR, Instruction Relocate) = SRR1_58 | SRR1_49 (PR, Problem State)&lt;br&gt;
[If going into problem state, relocation will be enabled]&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;MSR_59 (DR, Data Relocate) = SRR1_59 | SRR1_49 (PR, Problem State)&lt;br&gt;
[If going into problem state, relocation will be enabled]&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Next Instruction Address&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;NIA = SRR0_0:61 || 0b00&lt;br&gt;
[Jump to SRR0, set last 2 bits to zero to ensure address is aligned to 4 bytes]&lt;/li&gt;
&lt;/ul&gt;
&lt;h1&gt;hrfid - Hypervisor Return From Interrupt Doubleword&lt;/h1&gt;
&lt;h2&gt;Machine State Register&lt;/h2&gt;
&lt;p&gt;Copy all bits (except some reserved bits) from HSRR1 into the MSR, with the following exceptions:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;If MSR_29:31 != 0b010 [Transaction State Suspended, TM not available], or HSRR1_29:31 != 0b000 [Transaction State Non-transactional, TM not available] then:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;MSR_29:30 (TS, Transaction State) = HSRR1_29:30&lt;/li&gt;
&lt;li&gt;MSR_31 (TM, Transactional Memory Available) = HSRR1_31&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;[See the ISA description for explanation on how rfid interacts with TM and resulting interrupts]&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;MSR_48 (EE, External Interrupt Enable) = HSRR1_48 | HSRR1_49 (PR, Problem State)&lt;br&gt;
[If going into problem state, external interrupts will be enabled]&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;MSR_58 (IR, Instruction Relocate) = HSRR1_58 | HSRR1_49 (PR, Problem State)&lt;br&gt;
[If going into problem state, relocation will be enabled]&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;MSR_59 (DR, Data Relocate) = HSRR1_59 | HSRR1_49 (PR, Problem State)&lt;br&gt;
[If going into problem state, relocation will be enabled]&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Next Instruction Address&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;NIA = HSRR0_0:61 || 0b00&lt;br&gt;
[Jump to HSRR0, set last 2 bits to zero to ensure address is aligned to 4 bytes]&lt;/li&gt;
&lt;/ul&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Donnellan</dc:creator><pubDate>Wed, 18 Dec 2019 17:30:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2019-12-18:/blog/2019/12/18/rfid-and-hrfid/</guid><category>OpenPOWER</category><category>Instruction Set Architecture</category><category>openpower</category></item><item><title>TEN THOUSAND DISKS</title><link>https://sthbrx.github.io/blog/2019/06/18/ten-thousand-disks/</link><description>&lt;p&gt;In OpenPOWER land we have a project called &lt;a href="https://github.com/open-power/op-test-framework/"&gt;op-test-framework&lt;/a&gt; which (for all its strengths and weaknesses) allows us to test firmware on a variety of different hardware platforms and even emulators like Qemu.&lt;/p&gt;
&lt;p&gt;Qemu is a fantasic tool allowing us to relatively quickly test against an emulated POWER model, and of course is a critical part of KVM virtual machines running natively on POWER hardware. However the default POWER model in Qemu is based on the "pseries" machine type, which models something closer to a virtual machine or a PowerVM partition rather than a "bare metal" machine.&lt;/p&gt;
&lt;p&gt;Luckily we have &lt;a href="https://github.com/legoater"&gt;Cédric Le Goater&lt;/a&gt; who is &lt;a href="https://github.com/legoater/qemu"&gt;developing and maintaining&lt;/a&gt; a Qemu "powernv" machine type which more accurately models running directly on an OpenPOWER machine. It's an unwritten rule that if you're using Qemu in op-test, you've compiled this version of Qemu!&lt;/p&gt;
&lt;h2&gt;Teething Problems&lt;/h2&gt;
&lt;p&gt;Because the "powernv" type does more accurately model the physical system some extra care needs to be taken when setting it up. In particular at one point we noticed that the pretend CDROM and disk drive we attached to the model were.. not being attached. &lt;a href="https://github.com/open-power/op-test-framework/commit/1a97bc92c6029cfda91f565face260b806c04d86"&gt;This commit&lt;/a&gt; took care of that; the problem was that the &lt;a href="https://github.com/legoater/qemu/wiki/PowerNV#complex-configuration"&gt;PCI topology&lt;/a&gt; defined by the layout required us to be more exact about where PCI devices were to be added. By default only three spare PCI "slots" are available but as the commit says, "This can be expanded by adding bridges"...&lt;/p&gt;
&lt;h2&gt;More Slots!&lt;/h2&gt;
&lt;p&gt;Never one to stop at a just-enough solution, I wondered how easy it would be to add an extra PCI bridge or two to give the Qemu model more available slots for PCI devices. It turns out, easy enough once you know the correct invocation. For example, adding a PCI bridge in the first slot of the first default PHB is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;device&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;pci&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;bridge&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="m m-Double"&gt;.3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;bus&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="m m-Double"&gt;.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="kd"&gt;addr&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And inserting a device in that bridge just requires us to specify the bus and slot:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;device&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;virtio&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;blk&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;pci&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;drive&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;cdrom01&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;virtio02&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;bus&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="m m-Double"&gt;.4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="kd"&gt;addr&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Great! Each bridge provides 31 slots, so now we have plenty of room for extra devices.&lt;/p&gt;
&lt;h2&gt;Why Stop There?&lt;/h2&gt;
&lt;p&gt;We have three free slots, and we don't have a strict requirement on where devices are plugged in, so lets just plug a bridge into each of those slots while we're here:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;device&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;pci&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;bridge&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="m m-Double"&gt;.3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;bus&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="m m-Double"&gt;.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="kd"&gt;addr&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;device&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;pci&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;bridge&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="m m-Double"&gt;.4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;bus&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="m m-Double"&gt;.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="kd"&gt;addr&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;device&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;pci&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;bridge&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="m m-Double"&gt;.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;bus&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;pcie&lt;/span&gt;&lt;span class="m m-Double"&gt;.2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="kd"&gt;addr&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;What happens if we insert a new PCI bridge into another PCI bridge? Aside from stressing out our PCI developers, a bunch of extra slots! And then we could plug bridges into those bridges and then..&lt;/p&gt;
&lt;iframe src="https://giphy.com/embed/VB5WwlZIt8eRy" width="480" height="199" frameBorder="0" class="giphy-embed" allowFullScreen&gt;&lt;/iframe&gt;
&lt;p&gt;&lt;br&gt;&lt;/p&gt;
&lt;p&gt;Thus was born &lt;a href="https://github.com/open-power/op-test-framework/commit/6e78eea78a2f2cf37ebdd2cccbd5b18a7050c1eb"&gt;"OpTestQemu: Add PCI bridges to support more devices."&lt;/a&gt; and the testcase &lt;a href="https://github.com/open-power/op-test-framework/blob/master/testcases/Petitboot10000Disks.py"&gt;"Petitboot10000Disks"&lt;/a&gt;.
The changes to the Qemu model setup fill up each PCI bridge as long as we have devices to add, but reserve the first slot to add another bridge if we run out of room... and so on..&lt;/p&gt;
&lt;p&gt;Officially this is to support adding interesting disk topologies to test Pettiboot use cases, stress test device handling, and so on, but while we're here... what happens with 10,000 temporary disks?&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;======================================================================&lt;/span&gt;
&lt;span class="nl"&gt;ERROR&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;testListDisks&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;testcases&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Petitboot10000Disks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ConfigEditorTestCase&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;----------------------------------------------------------------------&lt;/span&gt;
&lt;span class="n"&gt;Traceback&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;most&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;recent&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;call&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;last&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="k"&gt;File&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;&amp;quot;/home/sam/git/op-test-framework/testcases/Petitboot10000Disks.py&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;27&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;setUp&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;system&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;goto_state&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;OpSystemState&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;PETITBOOT_SHELL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="k"&gt;File&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;&amp;quot;/home/sam/git/op-test-framework/common/OpTestSystem.py&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;366&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;goto_state&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stateHandlers&lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;self.state&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;state&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="k"&gt;File&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;&amp;quot;/home/sam/git/op-test-framework/common/OpTestSystem.py&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;695&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;run_IPLing&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;raise&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;my_exception&lt;/span&gt;
&lt;span class="nl"&gt;UnknownStateTransition&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Something&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;happened&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;system&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;state&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="ss"&gt;&amp;quot;2&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;and&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;we&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;transitioned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;UNKNOWN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;Review&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;following&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;more&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;details&lt;/span&gt;
&lt;span class="n"&gt;Message&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="ss"&gt;&amp;quot;OpTestSystem in run_IPLing and the Exception=&lt;/span&gt;
&lt;span class="ss"&gt;&amp;quot;&lt;/span&gt;&lt;span class="n"&gt;filedescriptor&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;range&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;select&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="ss"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="ss"&gt; caused the system to go to UNKNOWN_BAD and the system will be stopping.&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Yeah that's probably to be expected without some more massaging. What about a more modest 512?&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Resetting&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHBs&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;and&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;training&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;links&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;55.293343496&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PCI&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Probing&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;slots&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;56.364337089&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0000&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;56.364973775&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0000&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;57.127964432&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0000&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;03&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;57.128545637&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0000&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;03&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;57.395489618&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0000&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;04&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;57.396048285&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0000&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;04&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;58.145944205&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0000&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;05&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;58.146465795&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0000&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;05&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;58.404954853&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0000&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;06&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;58.405485438&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0000&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;06&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;60.178957315&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;60.179524173&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;60.198502097&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;02.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;60.198982582&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;02.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;60.435096197&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;03&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;60.435634380&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;03&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;61.171512439&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;04&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;61.172029071&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;04&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;61.425416049&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;05&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;61.425934524&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;05&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;62.172664549&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;06&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;62.173186458&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;06&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;63.434516732&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0002&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;63.435062124&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0002&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;64.177567772&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0002&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;03&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;64.178099773&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0002&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;03&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;64.431763989&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0002&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;04&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;64.432285000&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0002&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;04&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;65.180506790&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0002&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;05&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;65.181049905&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0002&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;05&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;65.432105600&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0002&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;06&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;65.432654326&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;0002&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;06&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_find_ecap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;(That isn't good)&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   66.177240655,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PCI&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;Summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   66.177906083,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB#0000&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;00.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;ROOT&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1014&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;03&lt;/span&gt;&lt;span class="n"&gt;dc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;R&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;C&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;060400&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;B&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01..07&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   66.178760724,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB#0000&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;01&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;00.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;ETOX&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;b36&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;000&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;R&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;C&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;060400&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;B&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;02..07&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   66.179501494,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB#0000&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;ETOX&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;b36&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;000&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;R&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;C&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;060400&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;B&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;03..07&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   66.180227773,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB#0000&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;03&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;ETOX&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;b36&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;000&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;R&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;C&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;060400&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;B&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;04..07&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   66.180953149,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB#0000&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;04&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;ETOX&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;b36&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;000&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;R&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;C&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;060400&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;B&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;05..07&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   66.181673576,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB#0000&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;05&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;ETOX&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;b36&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;000&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;R&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;C&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;060400&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;B&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;06..07&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   66.182395253,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB#0000&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;06&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;01.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;ETOX&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;b36&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;000&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;R&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;C&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;060400&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;B&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;07..07&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   66.183207399,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB#0000&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;07&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;02.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;PCID&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;af4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1001&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;R&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;C&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;010000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="n"&gt;scsi&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   66.183969138,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB#0000&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;07&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;03.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;PCID&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;af4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1001&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;R&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;C&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;010000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="n"&gt;scsi&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;(a lot more of this)&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   67.055196945,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB#0002&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="mf"&gt;.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;PCID&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;af4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1001&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;R&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;C&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;010000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="n"&gt;scsi&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   67.055926264,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PHB#0002&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="mf"&gt;.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;PCID&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;af4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1001&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;R&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;C&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;010000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="n"&gt;scsi&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   67.094591773,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;INIT&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Waiting&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="p"&gt;...&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   67.095105901,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;INIT&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;bit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;LE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;discovered&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   68.095749915,5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;INIT&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Starting&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;at&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x20010000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fdt&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;at&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x3075d270&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;168365&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bytes&lt;/span&gt;

&lt;span class="n"&gt;zImage&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;starting&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loaded&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;at&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x0000000020010000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nl"&gt;sp&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x0000000020d30ee8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;Allocating&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x1dc5098&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="p"&gt;...&lt;/span&gt;
&lt;span class="n"&gt;Decompressing&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mh"&gt;0x0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x000000002001f000&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mh"&gt;0x0000000020d2e578&lt;/span&gt;&lt;span class="p"&gt;)...&lt;/span&gt;
&lt;span class="n"&gt;Done&lt;/span&gt;&lt;span class="err"&gt;!&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Decompressed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x1c22900&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bytes&lt;/span&gt;

&lt;span class="n"&gt;Linux&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;PowerPC&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;load&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="n"&gt;Finalizing&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;device&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tree&lt;/span&gt;&lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tree&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;at&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x20d320a0&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.120562&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;watchdog&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;CPU&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;detected&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hard&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;LOCKUP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pnv_pci_cfg_write&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x88&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0xa4&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.120746&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;watchdog&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;CPU&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;TB&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;50402010473&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;last&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;heartbeat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;TB&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;45261673150&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10039&lt;/span&gt;&lt;span class="n"&gt;ms&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ago&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.120808&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Modules&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;linked&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.120906&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;CPU&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;PID&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;Comm&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;swapper&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;Not&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tainted&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;5.0.5&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;openpower1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;#2&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.120956&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;NIP&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;c000000000058544&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;LR&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;c00000000004d458&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;CTR&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000030052768&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.121006&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;REGS&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;c0000000fff5bd70&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;TRAP&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0900&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="ow"&gt;Not&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tainted&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;5.0.5&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;openpower1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.121030&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;MSR&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="mi"&gt;9000000002009033&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;SF&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;HV&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;VEC&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;EE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;ME&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;IR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;DR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;RI&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;LE&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nl"&gt;CR&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;48002482&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nl"&gt;XER&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;20000000&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.121215&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;CFAR&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;c00000000004d454&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;IRQMASK&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.121260&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;GPR00&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;00000000300051&lt;/span&gt;&lt;span class="n"&gt;ec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3130&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;c000000001bcaf00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.121368&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;GPR04&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000048002482&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;c000000000058544&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9000000002009033&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000031&lt;/span&gt;&lt;span class="n"&gt;c40060&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.121476&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;GPR08&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000031&lt;/span&gt;&lt;span class="n"&gt;c40060&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;c00000000004d46c&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9000000002001003&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.121584&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;GPR12&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000031&lt;/span&gt;&lt;span class="n"&gt;c40000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;c000000001dd0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;c00000000000f560&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.121692&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;GPR16&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000001&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.121800&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;GPR20&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.121908&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;GPR24&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000005&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000104&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122016&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;GPR28&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000002&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000004&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0000000000000086&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;c0000000fd9fba00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122150&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;NIP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000058544&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pnv_pci_cfg_write&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x88&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0xa4&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122187&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;LR&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c00000000004d458&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;opal_return&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x14&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x48&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122204&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;Call&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;Trace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122251&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3130&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000058544&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pnv_pci_cfg_write&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x88&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0xa4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;unreliable&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122332&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3150&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000000585d0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pnv_pci_write_config&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x70&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x9c&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122398&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c31a0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000234fec&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_bus_write_config_word&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x74&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x98&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122458&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c31f0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c00000000023764c&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__pci_read_base&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x88&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x3a4&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122518&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c32c0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000237a18&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_read_bases&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0xb0&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0xc8&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122605&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3300&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000002384bc&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_setup_device&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x4f8&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x5b0&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122670&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c33a0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000238d9c&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_single_device&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x9c&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0xd4&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122729&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c33f0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000238e2c&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_slot&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x58&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0xf4&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122796&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3430&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000239eb8&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_child_bus_extend&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x40&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x2a8&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122861&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c34a0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000239e34&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_bridge_extend&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x4d4&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x504&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122928&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3580&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c00000000023a0f8&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_child_bus_extend&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x280&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x2a8&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.122993&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c35f0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000239e34&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_bridge_extend&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x4d4&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x504&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123059&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c36d0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c00000000023a0f8&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_child_bus_extend&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x280&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x2a8&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123124&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3740&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000239e34&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_bridge_extend&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x4d4&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x504&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123191&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3820&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c00000000023a0f8&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_child_bus_extend&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x280&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x2a8&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123256&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3890&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000239b5c&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_bridge_extend&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x1fc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x504&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123322&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3970&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c00000000023a064&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_child_bus_extend&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x1ec&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x2a8&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123388&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c39e0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000239b5c&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_bridge_extend&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x1fc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x504&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123454&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3ac0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c00000000023a064&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pci_scan_child_bus_extend&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x1ec&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x2a8&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123516&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3b30&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000000030dcc&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pcibios_scan_phb&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x134&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x1f4&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123574&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3bd0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c00000000100a800&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pcibios_init&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x9c&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0xbc&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123635&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3c50&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c00000000000f398&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;do_one_initcall&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x80&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x15c&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123698&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3d10&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000001000e94&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel_init_freeable&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x248&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x24c&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123756&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3db0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c00000000000f574&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel_init&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x1c&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x150&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123820&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c0000000fd7c3e20&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c00000000000b72c&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ret_from_kernel_thread&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x5c&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x70&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123854&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Instruction&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;dump&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.123885&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="n"&gt;d054378&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;bff56f5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;60000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;38600000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;38210020&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;e8010010&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="n"&gt;c0803a6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;4e800020&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.124022&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;e86a0018&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;54&lt;/span&gt;&lt;span class="n"&gt;c6043e&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="n"&gt;d054378&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;bff5731&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="mi"&gt;60000000&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;bffffd8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;e86a0018&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="n"&gt;d054378&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.124180&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Kernel&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;panic&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;not&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;syncing&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Hard&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;LOCKUP&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.124232&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;CPU&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;PID&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;Comm&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;swapper&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;Not&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tainted&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;5.0.5&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;openpower1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;#2&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;   10.124251&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;Call&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;Trace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I wonder if I can submit that bug without someone throwing something at my desk.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Tue, 18 Jun 2019 16:47:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2019-06-18:/blog/2019/06/18/ten-thousand-disks/</guid><category>Development</category><category>linux</category><category>firmware</category><category>goodposts</category><category>realcontent</category><category>madposting</category><category>openpower</category><category>testing</category><category>op-test</category><category>qemu</category><category>pci</category></item><item><title>dm-crypt: Password Prompts Proliferate</title><link>https://sthbrx.github.io/blog/2019/05/29/dm-crypt-password-prompts-proliferate/</link><description>&lt;p&gt;&lt;a href="https://sthbrx.github.io/blog/2019/02/25/what-do-you-mean-no/"&gt;Just recently&lt;/a&gt; Petitboot added a method to ask the user for a password before allowing certain actions to proceed. Underneath the covers this is checking against the root password, but the UI "pop-up" asking for the password is relatively generic. Something else which has been on the to-do list for a while is support for mounting encrpyted partitions, but there wasn't a good way to retrieve the password for them - until now!&lt;/p&gt;
&lt;p&gt;With the password problem solved, there isn't too much else to do. If Petitboot sees an encrypted partition it makes a note of it and informs the UI via the &lt;code&gt;device_add&lt;/code&gt; interface. Seeing this the UI shows this device in the UI even though there aren't any boot options associated with it yet:&lt;/p&gt;
&lt;p&gt;&lt;img alt="encrypted_hdr" src="/images/sammj/encrypted_hdr.png"&gt;&lt;/p&gt;
&lt;p&gt;Unlike normal devices in the menu these are selectable; once that happens the user is prompted for the password:&lt;/p&gt;
&lt;p&gt;&lt;img alt="encrypted_password" src="/images/sammj/encrypted_password.png"&gt;&lt;/p&gt;
&lt;p&gt;With password in hand pb-discover will then try to open the device with &lt;code&gt;cryptsetup&lt;/code&gt;. If that succeeds the encrypted device is removed from the UI and replaced with the new un-encrypted device:&lt;/p&gt;
&lt;p&gt;&lt;img alt="unencrypted_hdr" src="/images/sammj/unencrypted_hdr.png"&gt;&lt;/p&gt;
&lt;p&gt;That's it! These devices can't be auto-booted from at the moment since the password needs to be manually entered. The UI also doesn't have a way yet to select specific options for &lt;code&gt;cryptsetup&lt;/code&gt;, but if you find yourself needing to do so you can run &lt;code&gt;cryptsetup&lt;/code&gt; manually from the shell and pb-discover will recognise the new unencrypted device automatically.&lt;/p&gt;
&lt;p&gt;This is in Petitboot as of v1.10.3 so go check it out! Just make sure your image includes the kernel and cryptsetup &lt;a href="https://github.com/open-power/op-build/pull/2840"&gt;dependencies&lt;/a&gt;.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Wed, 29 May 2019 16:47:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2019-05-29:/blog/2019/05/29/dm-crypt-password-prompts-proliferate/</guid><category>Petitboot</category><category>linux</category><category>firmware</category><category>goodposts</category><category>realcontent</category><category>sparseposting</category><category>openpower</category><category>petitboot</category><category>security</category></item><item><title>Visual Studio Code for Linux kernel development</title><link>https://sthbrx.github.io/blog/2019/05/07/visual-studio-code-for-linux-kernel-development/</link><description>&lt;p&gt;Here we are again - back in 2016 &lt;a href="https://sthbrx.github.io/blog/2016/06/07/using-the-atom-editor-for-linux-kernel-development/"&gt;I wrote an article on using Atom for kernel development&lt;/a&gt;, but I didn't stay using it for too long, instead moving back to Emacs.  Atom had too many shortcomings - it had that distinctive &lt;em&gt;Electron feel&lt;/em&gt;, which is a problem for a text editor - you need it to be snappy.  On top of that, vim support was mediocre at best, and even as a vim scrub I would find myself trying to do things that weren't implemented.&lt;/p&gt;
&lt;p&gt;So in the meantime I switched to &lt;a href="http://spacemacs.org/"&gt;spacemacs&lt;/a&gt;, which is a very well integrated "vim in Emacs" experience, with a lot of opinionated (but good) defaults.  spacemacs was pretty good to me but had some issues - disturbingly long startup times, mediocre completions and go-to-definitions, and integrating any module into spacemacs that wasn't already integrated was a big pain.&lt;/p&gt;
&lt;p&gt;After that I switched to &lt;a href="https://github.com/hlissner/doom-emacs"&gt;Doom Emacs&lt;/a&gt;, which is like spacemacs but faster and closer to Emacs itself.  It's very user configurable but much less user friendly, and I didn't really change much as my elisp-fu is practically non-existent.  I was decently happy with this, but there were still some issues, some of which are just inherent to Emacs itself - like no actually usable inbuilt terminal, despite having (at least) four of them.&lt;/p&gt;
&lt;p&gt;Anyway, since 2016 when I used Atom, Visual Studio Code (henceforth referred to as Code) came along and ate its lunch, using the framework (Electron) that was created for Atom.  I did try it years ago, but I was very turned off by its Microsoft-ness, it seeming lack of distinguishing features from Atom, and it didn't feel like a native editor at all.  Since it's massively grown in popularity since then, I decided I'd give it a try.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Visual Studio Code" src="/images/ruscur/vscode.png"&gt;&lt;/p&gt;
&lt;h3&gt;Vim emulation&lt;/h3&gt;
&lt;p&gt;First things first for me is getting a vim mode going, and Code has a pretty good one of those.  The key feature for me is that there's &lt;a href="https://neovim.io"&gt;Neovim&lt;/a&gt; integration for Ex-commands, filling a lot of shortcomings that come with most attempts at vim emulation.  In any case, everything I've tried to do that I'd do in vim (or Emacs) has worked, and there are a ton of options and things to tinker with.  Obviously it's not going to do as much as you could do with Vimscript, but it's definitely not bad.&lt;/p&gt;
&lt;h3&gt;Theming and UI customisation&lt;/h3&gt;
&lt;p&gt;As far as the editor goes - it's good.  A ton of different themes, you can change the colour of pretty much everything in the config file or in the UI, including icons for the sidebar.  There's a huge sore point though, you can't customise the interface outside the editor pretty much at all.  There's an extension for loading custom CSS, but it's out of the way, finnicky, and if I wanted to write CSS I wouldn't have become a kernel developer.&lt;/p&gt;
&lt;h3&gt;Extensibility&lt;/h3&gt;
&lt;p&gt;Extensibility is definitely a strong point, the ecosystem of extensions is good.  All the language extensions I've tried have been very fully featured with a ton of different options, integration into language-specific linters and build tools.  This is probably Code's strongest feature - the breadth of the extension ecosystem and the level of quality found within.&lt;/p&gt;
&lt;h3&gt;Kernel development&lt;/h3&gt;
&lt;p&gt;Okay, let's get into the main thing that matters - how well does the thing actually edit code.  The kernel is tricky.  It's huge, it has its own build system, and in my case I build it with cross compilers for another architecture.  Also, y'know, it's all in C and built with make, not exactly great for any kind of IDE integration.&lt;/p&gt;
&lt;p&gt;The first thing I did was check out the &lt;a href="https://github.com/amezin/vscode-linux-kernel"&gt;vscode-linux-kernel&lt;/a&gt; project by GitHub user "amezin", which is a great starting point.  All you have to do is clone the repo, build your kernel (with a cross compiler works fine too), and run the Python script to generate the &lt;code&gt;compile_commands.json&lt;/code&gt; file.  Once you've done this, go-to-definition (&lt;code&gt;gd&lt;/code&gt; in vim mode) works pretty well.  It's not flawless, but it does go cross-file, and will pop up a nice UI if it can't figure out which file you're after.&lt;/p&gt;
&lt;p&gt;Code has good built-in git support, so actions like staging files for a commit can be done from within the editor.  Ctrl-P lets you quickly navigate to any file with fuzzy-matching (which is impressively fast for a project of this size), and Ctrl-Shift-P will let you search commands, which I've been using for some git stuff.&lt;/p&gt;
&lt;p&gt;&lt;img alt="git command completion in Code" src="/images/ruscur/vscode-git.png"&gt;&lt;/p&gt;
&lt;p&gt;There are some rough edges, though.  Code is set on what so many modern editors are set on, which is the "one window per project" concept - so to get things working the way you want, you would open your kernel source as the current project.  This makes it a pain to just open something else to edit, like some script, or checking the value of something in firmware, or chucking something in your bashrc.&lt;/p&gt;
&lt;p&gt;Auto-triggering builds on change isn't something that makes a ton of sense for the kernel, and it's not present here.  The kernel support in the repo above is decent, but it's not going to get you close to what more modern languages can get you in an editor like this.&lt;/p&gt;
&lt;p&gt;Oh, and it has a powerpc assembly extension, but I didn't find it anywhere near as good as the one I "wrote" for Atom (I just took the x86 one and switched the instructions), so I'd rather use the C mode.&lt;/p&gt;
&lt;h3&gt;Terminal&lt;/h3&gt;
&lt;p&gt;Code has an actually good inbuilt terminal that uses your login shell.  You can bring it up with Ctrl-`.  The biggest gripe I have always had with Emacs is that you can never have a shell that you can actually do anything in, whether it's &lt;code&gt;eshell&lt;/code&gt; or &lt;code&gt;shell&lt;/code&gt; or &lt;code&gt;term&lt;/code&gt; or &lt;code&gt;ansi-term&lt;/code&gt;, you try to do something in it and it doesn't work or clashes with some Emacs command, and then when you try to do something Emacs-y in there it doesn't work.  No such issue is present here, and it's a pleasure to use for things like triggering a remote build or doing some git operation you don't want to do with commands in the editor itself.&lt;/p&gt;
&lt;p&gt;Not the most important feature, but I do like not having to alt-tab out and lose focus.&lt;/p&gt;
&lt;h3&gt;Well...is it good?&lt;/h3&gt;
&lt;p&gt;Yeah, it is.  It has shortcomings, but installing Code and using the repo above to get started is probably the simplest way to get a competent kernel development environment going, with more features than most kernel developers (probably) have in their editors.  Code is open source and so are its extensions, and it'd be the first thing I recommend to new developers who aren't already super invested into vim or Emacs, and it's worth a try if you have gripes with your current environment.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Russell Currey</dc:creator><pubDate>Tue, 07 May 2019 00:00:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2019-05-07:/blog/2019/05/07/visual-studio-code-for-linux-kernel-development/</guid><category>Development</category><category>vscode</category><category>code</category><category>linux</category></item><item><title>Article Review: Curing the Vulnerable Parser</title><link>https://sthbrx.github.io/blog/2019/04/02/article-review-curing-the-vulnerable-parser/</link><description>&lt;p&gt;Every once in a while I read papers or articles. Previously, I've just read them myself, but I was wondering if there were more useful things I could do beyond that. So I've written up a summary and my thoughts on an article I read - let me know if it's useful!&lt;/p&gt;
&lt;p&gt;I recently read &lt;a href="https://www.usenix.org/publications/login/spring2017/bratus"&gt;Curing the Vulnerable Parser: Design Patterns for Secure Input Handling&lt;/a&gt; (Bratus, et al; USENIX ;login: Spring 2017). It's not a formal academic paper but an article in the Usenix magazine, so it doesn't have a formal abstract I can quote, but in short it takes the long history of parser and parsing vulnerabilities and uses that as a springboard to talk about how you could design better ones. It introduces a toolkit based on that design for more safely parsing some binary formats.&lt;/p&gt;
&lt;h2&gt;Background&lt;/h2&gt;
&lt;p&gt;It's worth noting early on that this comes out of the &lt;a href="http://langsec.org/"&gt;LangSec crowd&lt;/a&gt;. They have a pretty strong underpinning philosophy:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;The Language-theoretic approach (LANGSEC) regards the Internet insecurity epidemic as a consequence of &lt;em&gt;ad hoc&lt;/em&gt; programming of input handling at all layers of network stacks, and in other kinds of software stacks. LANGSEC posits that the only path to trustworthy software that takes untrusted inputs is treating all valid or expected inputs as a formal language, and the respective input-handling routines as a &lt;em&gt;recognizer&lt;/em&gt; for that language. The recognition must be feasible, and the recognizer must match the language in required computation power.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;A big theme in this article is predictability:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Trustworthy input is input with predictable effects. The goal of input-checking is being able to predict the input’s effects on the rest of your program.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;This seems sensible enough at first, but leads to some questionable assertions, such as:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Safety is predictability. When it's impossible to predict what the effects of the input will be (however valid), there is no safety.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;They follow this with an example of Ethereum contracts stealing money from the DAO. The example is compelling enough, but again comes with a very strong assertion about the impossibility of securing a language virtual machine:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;From the viewpoint of language-theoretic security, a catastrophic exploit in Ethereum was only a matter of time: one can only find out what such programs do by running them. By then it is too late.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I'm not sure that (a) I buy the assertions, or that (b) they provide a useful way to deal with the world as we find it.&lt;/p&gt;
&lt;h3&gt;Is this even correct?&lt;/h3&gt;
&lt;p&gt;You can tease out 2 contentions in the first part of the article:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;there should be a formal language that describes the data, and&lt;/li&gt;
&lt;li&gt;this language should be as simple as possible, ideally being regular and context-free.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Neither of these are bad ideas - in fact they're both good ideas - but I don't know that I draw the same links between them and security.&lt;/p&gt;
&lt;p&gt;Consider PostScript as a possible counter-example. It's a Turing-complete language, so it absolutely cannot have predictable results. It has a well documented specification and executes in a restricted virtual machine. So let's say that it satisfies only the first plank of their argument.&lt;/p&gt;
&lt;p&gt;I'd say that PostScript has a good security record, despite being Turing complete. PostScript has been around since 1985 and apart from the recent bugs in GhostScript, it doesn't have a long history of bugs and exploits. Maybe this just because no-one has really looked, or maybe it is possible to have reasonably safe complex languages by restricting the execution environment, as PostScript consciously and deliberately does.&lt;/p&gt;
&lt;p&gt;Indeed, if you consider the recent spate &lt;a href="https://bugs.chromium.org/p/project-zero/issues/detail?id=1675"&gt;of&lt;/a&gt; &lt;a href="https://bugs.chromium.org/p/project-zero/issues/detail?id=1691"&gt;GhostScript&lt;/a&gt; &lt;a href="https://bugs.chromium.org/p/project-zero/issues/detail?id=1682"&gt;bugs&lt;/a&gt;, perhaps &lt;a href="https://bugs.chromium.org/p/project-zero/issues/detail?id=1640"&gt;some&lt;/a&gt; may be avoided by stricter compliance with a formal language specification. However, most seem to me to arise from the desirability of implementing some of the PostScript functionality in PostScript itself, and some of the GhostScript-specific, stupendously powerful operators exposed to the language to enable this. The bugs involve tricks to allow a user to get access to these operators. A non-Turing-complete language may be &lt;em&gt;sufficient&lt;/em&gt; to prevent these attacks, but it is not &lt;em&gt;necessary&lt;/em&gt;: just not doing this sort of meta-programming with such dangerous operators would also have worked. Storing the true values of the security state outside of a language-accessible object would also be good.&lt;/p&gt;
&lt;h3&gt;Is this a useful way to deal with the world as we find it?&lt;/h3&gt;
&lt;p&gt;My main problem with the general LangSec approach that this article takes is this: to get to their desired world, we need to rewrite a bunch of things with entirely different language foundations. The article talks about HTML and PDFs as examples of unsafe formats, but I cannot imagine the sudden wholesale replacement of either of these - although I would love to be proven wrong.&lt;/p&gt;
&lt;p&gt;Can we get even part of the way with existing standards? Kinda-sorta, but mostly no, and to the authors' credit, they are open about this. They argue that formal definition parsing the language should be the "most restrictive input definition" - they specifically require you to "give up attempting to accept arbitrarily complex data", and call for "subsetting of many protocols, formats, encodings and command languages, including eliminating unneeded variability and introducing determinism and static values".&lt;/p&gt;
&lt;p&gt;No doubt we would be in a better place if people took up these ideas for future programs.
However, for our current set of programs and use cases, this is probably not tractable in any meaningful way.&lt;/p&gt;
&lt;h2&gt;The rest of the paper&lt;/h2&gt;
&lt;p&gt;The rest of the paper is reasonably interesting. Their general theory is that you should build your parsers based on a formal definition of a language, and that the parser should convert the input data to a set of objects, and then your business logic should deal with those objects. This is the 'recognizer pattern', and is illustrated below:&lt;/p&gt;
&lt;p&gt;&lt;img alt="The recognizer pattern: separate code parses input according to a formal grammar, creating valid objects that are passed to the business logic" src="/images/dja/recogniser.png"&gt;&lt;/p&gt;
&lt;p&gt;In short, the article is full of great ideas if you happen to be parsing a simple language, or are designing not just a parser but a full language ecosystem. They do also provide a binary parser toolkit that might be helpful if you are parsing a binary format that can be expressed with a parser combinator.&lt;/p&gt;
&lt;p&gt;Overall, however, I think the burden of maintaining old systems is such that a security paradigm that relies on new code is pretty unlikely, and one that relies on new languages is fatally doomed from the outset. New systems should take up these ideas, yes. But I'd really like to see people grappling with how to deal with the complex and irregular languages that we're stuck with (HTML, PDF, etc) in secure ways.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Tue, 02 Apr 2019 00:00:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2019-04-02:/blog/2019/04/02/article-review-curing-the-vulnerable-parser/</guid><category>Education</category><category>security</category></item><item><title>What Do You Mean "No"?</title><link>https://sthbrx.github.io/blog/2019/02/25/what-do-you-mean-no/</link><description>&lt;p&gt;Quite often when building small Linux images having separate user accounts isn't always at the top of the list of things to include. Petitboot is no different; the most common operations like mounting disks, configuring interfaces, and calling &lt;code&gt;kexec&lt;/code&gt; all require root and Petitboot generally only exists long enough to boot into the next thing, so why not run it all as root?&lt;/p&gt;
&lt;p&gt;The picture is less clear when we start to think about what is possible to do in Petitboot by default. If someone comes across an open Petitboot console they're only a few keystrokes away from wiping disks, changing files, or even flashing firmware. Depending on how your system is used that may or may not be something you care about, but over time there have been a few requests to "add a password screen to Petitboot" to at least make it so that the system isn't open season for whoever sees it.&lt;/p&gt;
&lt;h2&gt;Enter Password:&lt;/h2&gt;
&lt;p&gt;The most direct way to avoid this would be to slap a password prompt onto Petitboot before any changes can be made. There are two immediate drawbacks to this:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The Petitboot UI still runs as root, and&lt;/li&gt;
&lt;li&gt;Exiting to the shell gives the user root permissions as well.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There is already a &lt;a href="https://github.com/open-power/petitboot/commit/f5dab0206a3baca73895a587583ddfa402f8f569"&gt;mechanism&lt;/a&gt; to prevent the user exiting to the shell, but this puts all of our eggs in the basket of petitboot-nc being a secure program. If a user can accidentally or otherwise find a way to exit or crash the UI then they're immediately in a root shell, and while petitboot-nc is a good UI it was never designed to be a hardened program protecting the system.&lt;/p&gt;
&lt;h2&gt;You Have No Power Here&lt;/h2&gt;
&lt;p&gt;The idea instead as of Petitboot v1.10.0 is not to care if the user drops to the shell; because now it's completely unprivileged.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Normal shell" src="/images/sammj/users-reboot.png"&gt;&lt;/p&gt;
&lt;p&gt;The only process now that runs as root is pb-discover itself; the console, UI, and helper scripts run as a new 'petituser'. For the server and clients to still communicate the "petitiboot.ui" socket permissions are modified to allow processes that are part of the 'petitgroup' to connect. However now if pb-discover notices that a client in the petitgroup is connecting  (or more accurately the client isn't running as root) by default it ignores any commands from it that would configure or boot the system.&lt;/p&gt;
&lt;p&gt;A new command, &lt;code&gt;PB_PROTOCOL_ACTION_AUTHENTICATE&lt;/code&gt;, lets a client send a password to the server to then be allowed to send all the usual commands like updating the config or booting a specific option. This keeps all the authentication on the server side, avoiding writing any "secure" ncurses code. In the UI the biggest difference is that when trying to change something the user will hit a password field:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Denied" src="/images/sammj/users-denied.png"&gt;&lt;/p&gt;
&lt;p&gt;Then the password is sent to the server, checked, and if correct the action goes ahead.&lt;/p&gt;
&lt;h2&gt;Whose Passwords?&lt;/h2&gt;
&lt;p&gt;But where does this password come from? Technically it's just the root password. The server computes a hash of the supplied password and compares it against the system's root password. Similarly in the shell the user can run &lt;code&gt;sudo&lt;/code&gt; with the root password to enter a full shell if needed:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Oops" src="/images/sammj/users-dd-random.png"&gt;&lt;/p&gt;
&lt;p&gt;Petitboot of course runs in memory, and writing a root password into the image itself would mean recompiling to change the password, so instead Petitboot pulls the root password from NVRAM. On startup Petitboot reads the &lt;code&gt;petitboot,password&lt;/code&gt; parameter which is the &lt;em&gt;hash&lt;/em&gt; of the root password and updates &lt;code&gt;/etc/shadow&lt;/code&gt; with it. This happens before any clients are up or can connect to the server.&lt;/p&gt;
&lt;h2&gt;Don't Panic&lt;/h2&gt;
&lt;p&gt;By default no password is set. After all we don't want people upgrading and then being somehow locked out of their system. For ease of use, and for testing-purposes, if no password is configured and the user drops to the shell it is automatically upgraded to a root shell:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Elevated" src="/images/sammj/users-elevated.png"&gt;&lt;/p&gt;
&lt;p&gt;To set a password there is a new subscreen in System Configuration:&lt;/p&gt;
&lt;p&gt;&lt;img alt="New Password" src="/images/sammj/users-new-password.png"&gt;&lt;/p&gt;
&lt;p&gt;This sends an authentication command to the server, and assuming the client is authenticated with the current password as well pb-discover updates the shadow file, and writes the hash back to NVRAM.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;User support exists in Petitboot v1.10.0 onwards. You will also need some support in your build system to set up the users, see how &lt;a href="https://github.com/open-power/op-build/commit/7b5f1efbee6ace9d4ee80640875aa3ad57e95c69"&gt;op-build did it for an example&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;There are a few items on the TODO list still which would be good to have. For example storing the password hash in an attached TPM if available, as well as splitting out more of what runs as root; for example the bootloader parsers in pb-discover preferably wouldn't run with root privileges, but they are all part of the one binary.&lt;/p&gt;
&lt;p&gt;As always, comments, suggestions, and patches welcome on the &lt;a href="https://lists.ozlabs.org/listinfo/petitboot"&gt;list&lt;/a&gt;!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Mon, 25 Feb 2019 16:47:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2019-02-25:/blog/2019/02/25/what-do-you-mean-no/</guid><category>Petitboot</category><category>linux</category><category>firmware</category><category>goodposts</category><category>realcontent</category><category>sparseposting</category><category>openpower</category><category>petitboot</category><category>security</category></item><item><title>IPMI: Initiating Better Overrides</title><link>https://sthbrx.github.io/blog/2018/12/19/ipmi-initiating-better-overrides/</link><description>&lt;p&gt;On platforms that support it Petitboot can interact with the inband IPMI interface to pull information from the BMC. One particularly useful example of this is the "Get System Boot Options" command which we use to implement boot "overrides". By setting parameter 5 of the command a user can remotely force Petitboot to boot from only one class of device or disable autoboot completely. This is great for automation or debug purposes, but since it can only specify device types like "disk" or "network" it can't be used to boot from specific devices.&lt;/p&gt;
&lt;p&gt;Introducing..&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Boot Initiator Mailbox&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Alexander Amelkin &lt;a href="https://github.com/open-power/petitboot/issues/45"&gt;pointed out&lt;/a&gt; that the "Get System Boot Options" command also specifies parameter 7, "Boot Initiator Mailbox". This parameter just defines a region of vendor-defined data that can be used to influence the booting behaviour of the system. The parameter description specifies that a BMC must support at least 80 bytes of data in that mailbox so as Alex pointed out we could easily use it to set a partition UUID. But why stop there? Let's go further and use the mailbox to provide an alterate "petitboot,bootdevs=.." parameter and let a user set a full substitute boot order!&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Mailbox Format&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Parameter 7 has two fields, 1 byte for the "set selector", and up to 16 bytes of "block data". The spec sets the minimum amount of data to support at 80 bytes, which means a BMC must support at least 5 of these 16-byte "blocks" which can be individually accessed via the set selector. Aside from the first 3 bytes which must be an IANA ID number, the rest of the data is defined by us.&lt;/p&gt;
&lt;p&gt;So if we want to set an alternate Petitboot boot order such as "network, usb, disk", the format of the mailbox would be:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="gh"&gt;Block # |       Block Data              |&lt;/span&gt;
&lt;span class="gh"&gt;-----------------------------------------&lt;/span&gt;
0       |2|0|0|p|e|t|i|t|b|o|o|t|,|b|o|o|
1       |t|d|e|v|s|=|n|e|t|w|o|r|k| |u|s|
2       |b| |d|i|s|k| | | | | | | | | | |
3       | | | | | | | | | | | | | | | | |
4       | | | | | | | | | | | | | | | | |
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Where the string is null-terminated, &lt;code&gt;2,0,0&lt;/code&gt; is the IBM IANA ID, and the contents of any remaining data is not important. The &lt;a href="https://github.com/open-power/petitboot/blob/master/utils/ipmi-mailbox-config.py"&gt;ipmi-mailbox-config.py&lt;/a&gt; script constructs and sends the required IPMI commands from a given parameter string to make this easier, eg:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;./utils/ipmi-mailbox-config.py -b bmc-ip -u user -p pass -m 5 \
        -c &amp;quot;petitboot,bootdevs=uuid:c6e4c4f9-a9a2-4c30-b0db-6fa00f433b3b&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;img alt="Active Mailbox Override" src="/images/sammj/ipmi-mailbox.png"&gt;&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;That is basically all there is to it. Setting a boot order this way overrides the existing order from NVRAM if there is one. Parameter 7 doesn't have a 'persistent' flag so the contents need to either be manually cleared from the BMC or cleared via the "Clear" button in the System Configuration screen.&lt;/p&gt;
&lt;p&gt;From the machines I've been able to test on at least AMI BMCs support the mailbox, and hopefully &lt;a href="https://lists.ozlabs.org/pipermail/openbmc/2018-December/014224.html"&gt;OpenBMC&lt;/a&gt; will be able to add it to their IPMI implementation. This is supported in Petitboot as of v1.10.0 so go ahead and try it out!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Wed, 19 Dec 2018 10:08:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2018-12-19:/blog/2018/12/19/ipmi-initiating-better-overrides/</guid><category>Petitboot</category><category>linux</category><category>firmware</category><category>goodposts</category><category>realcontent</category><category>sparseposting</category><category>openpower</category><category>openbmc</category><category>ipmi</category><category>petitboot</category></item><item><title>OpenPOWER Summit Europe 2018: A Software Developer's Introduction to OpenCAPI</title><link>https://sthbrx.github.io/blog/2018/11/21/openpower-summit-europe-2018-a-software-developers-introduction-to-opencapi/</link><description>&lt;p&gt;Last month, I was in Amsterdam at &lt;a href="https://openpowerfoundation.org/summit-2018-10-eu/"&gt;OpenPOWER Summit Europe&lt;/a&gt;. It was great to see so much interest in OpenPOWER, with a particularly strong contingent of researchers sharing how they're exploiting the unique advantages of OpenPOWER platforms, and a number of OpenPOWER hardware partners announcing products.&lt;/p&gt;
&lt;p&gt;(It was also my first time visiting Europe, so I had a lot of fun exploring Amsterdam, taking a few days off in Vienna, then meeting some of my IBM Linux Technology Centre colleagues in Toulouse. I also now appreciate just what ~50 hours on planes does to you!)&lt;/p&gt;
&lt;p&gt;One particular area which got a lot of attention at the Summit was &lt;a href="https://opencapi.org"&gt;OpenCAPI&lt;/a&gt;, an open coherent high-performance bus interface designed for accelerators, which is supported on POWER9. We had plenty of talks about OpenCAPI and the interesting work that is already happening with OpenCAPI accelerators.&lt;/p&gt;
&lt;p&gt;I was invited to present on the Linux Technology Centre's work on enabling OpenCAPI from the software side. In this talk, I outline the OpenCAPI software stack and how you can interface with an OpenCAPI device through the ocxl kernel driver and the &lt;a href="https://github.com/opencapi/libocxl"&gt;libocxl&lt;/a&gt; userspace library.&lt;/p&gt;
&lt;iframe width="560" height="315" src="https://www.youtube.com/embed/zCIMHbZDRS0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen&gt;&lt;/iframe&gt;

&lt;p&gt;My &lt;a href="https://sthbrx.github.io/misc/OPFEU2018_OpenCAPI.pdf"&gt;slides&lt;/a&gt; are available, though you'll want to watch the presentation for context.&lt;/p&gt;
&lt;p&gt;Apart from myself, the OzLabs team were well represented at the Summit:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/grooverdan"&gt;Daniel Black&lt;/a&gt; spoke on Power performance optimisation&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.flamingspork.com/"&gt;Stewart Smith&lt;/a&gt; gave an overview of the Power boot process&lt;/li&gt;
&lt;li&gt;&lt;a href="https://shenki.github.io/"&gt;Joel Stanley&lt;/a&gt; presented on recent developments in OpenBMC&lt;/li&gt;
&lt;li&gt;&lt;a href="http://jk.ozlabs.org/"&gt;Jeremy Kerr&lt;/a&gt; talked about the various OpenPOWER firmware projects&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Unfortunately none of their videos are up yet, but they'll be there over the next few weeks. Keep an eye on the &lt;a href="https://openpowerfoundation.org/summit-2018-10-eu/"&gt;Summit website&lt;/a&gt; and the &lt;a href="https://www.youtube.com/playlist?list=PLEqfbaomKgQo5CimgYbVxdtiVtyloWc9e"&gt;Summit YouTube playlist&lt;/a&gt;, where you'll find all the rest of the Summit content.&lt;/p&gt;
&lt;p&gt;If you've got any questions about OpenCAPI feel free to leave a comment!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Donnellan</dc:creator><pubDate>Wed, 21 Nov 2018 18:20:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2018-11-21:/blog/2018/11/21/openpower-summit-europe-2018-a-software-developers-introduction-to-opencapi/</guid><category>OpenPOWER</category><category>linux</category><category>firmware</category><category>openpower</category><category>opencapi</category><category>openpower summit</category></item><item><title>Open Source Firmware Conference 2018</title><link>https://sthbrx.github.io/blog/2018/10/09/open-source-firmware-conference-2018/</link><description>&lt;p&gt;I recently had the pleasure of attending the &lt;a href="https://osfc.io"&gt;2018 Open Source Firmware Conference&lt;/a&gt; in Erlangen, Germany. Compared to other more general conferences I've attended in the past, the laser focus of OSFC on firmware and especially firmware security was fascinating. Seeing developers from across the world coming together to discuss how they are improving their corner of the stack was great, and I've walked away with plenty of new knowledge and ideas (and several kilos of German food and drink..).&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;What was especially exciting though is that I had the chance to talk about my work on &lt;a href="https://github.com/open-power/petitboot"&gt;Petitboot&lt;/a&gt; and what's happened from the POWER8 launch until now. If you're interested in that, or seeing how I talk after 36 hours of travel, check it out here:&lt;/p&gt;
&lt;iframe width="560" height="315" src="https://www.youtube.com/embed/8Z1vEinNU7I" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen&gt;&lt;/iframe&gt;

&lt;hr&gt;
&lt;p&gt;OSFC have made all the talks from the first two days available in a &lt;a href="https://www.youtube.com/playlist?list=PLJ4u8GLmFVmoRCX_gFXV6fhWmsOQ5cmuj"&gt;playlist on Youtube&lt;/a&gt;&lt;br&gt;
If you're after a few suggestions there was, in no particular order:&lt;/p&gt;
&lt;p&gt;&lt;a href="https://www.youtube.com/watch?v=ZjAu0VYRTno"&gt;Ryan O'Leary giving an update on Linuxboot&lt;/a&gt; - also known as NERF, Google's approach to a Linux bootloader all written in Go.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://www.youtube.com/watch?v=qUgo-AVsSC"&gt;Subrate Banik talking about porting Coreboot on top of Intel's FSP&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href="https://www.youtube.com/watch?v=ukSh1n7wjSA"&gt;Ron Minnich describing his work with "rompayloads" on Coreboot&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href="https://www.youtube.com/watch?v=gC-lbMNmIsg"&gt;Vadmin Bendebury describing Google's "Secure Microcontroller" Chip&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href="https://www.youtube.com/watch?v=eKVSBESoKUc"&gt;Facebook presenting their use of Linuxboot and "systemboot"&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;And heaps more, check out the full playlist!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Tue, 09 Oct 2018 10:08:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2018-10-09:/blog/2018/10/09/open-source-firmware-conference-2018/</guid><category>Petitboot</category><category>linux</category><category>firmware</category><category>security</category><category>goodposts</category><category>realcontent</category><category>sparseposting</category><category>openpower</category><category>openbmc</category><category>easyposts</category><category>linuxboot</category><category>google</category><category>intel</category><category>osfc</category><category>shortposts</category><category>facebook</category></item><item><title>Improving performance of Phoronix benchmarks on POWER9</title><link>https://sthbrx.github.io/blog/2018/08/15/improving-performance-of-phoronix-benchmarks-on-power9/</link><description>&lt;p&gt;Recently Phoronix ran a range of
&lt;a href="https://www.phoronix.com/scan.php?page=article&amp;amp;item=power9-talos-2&amp;amp;num=1"&gt;benchmarks&lt;/a&gt;
comparing the performance of our POWER9 processor against the Intel Xeon and AMD
EPYC processors. &lt;/p&gt;
&lt;p&gt;We did well in the Stockfish, LLVM Compilation, Zstd compression, and the
Tinymembench benchmarks. A few of my colleagues did a bit of investigating into
some the benchmarks where we didn't perform quite so well.&lt;/p&gt;
&lt;h3&gt;LBM / Parboil&lt;/h3&gt;
&lt;p&gt;The &lt;a href="http://impact.crhc.illinois.edu/parboil/parboil.aspx"&gt;Parboil benchmarks&lt;/a&gt; are a
collection of programs from various scientific and commercial fields that are
useful for examining the performance and development of different architectures
and tools.  In this round of benchmarks Phoronix used the lbm
&lt;a href="https://www.spec.org/cpu2006/Docs/470.lbm.html"&gt;benchmark&lt;/a&gt;: a fluid dynamics
simulation using the Lattice-Boltzmann Method.&lt;/p&gt;
&lt;p&gt;lbm is an iterative algorithm - the problem is broken down into discrete
time steps, and at each time step a bunch of calculations are done to
simulate the change in the system. Each time step relies on the results
of the previous one.&lt;/p&gt;
&lt;p&gt;The benchmark uses OpenMP to parallelise the workload, spreading the
calculations done in each time step across many CPUs. The number of
calculations scales with the resolution of the simulation.&lt;/p&gt;
&lt;p&gt;Unfortunately, the resolution (and therefore the work done in each time
step) is too small for modern CPUs with large numbers of SMT (simultaneous multi-threading) threads. OpenMP 
doesn't have enough work to parallelise and the system stays relatively idle. This
means the benchmark scales relatively poorly, and is definitely
not making use of the large POWER9 system&lt;/p&gt;
&lt;p&gt;Also this benchmark is compiled without any optimisation. Recompiling with -O3 improves the
   results 3.2x on POWER9.&lt;/p&gt;
&lt;h3&gt;x264 Video Encoding&lt;/h3&gt;
&lt;p&gt;x264 is a library that encodes videos into the H.264/MPEG-4 format. x264 encoding
requires a lot of integer kernels doing operations on image elements. The math
and vectorisation optimisations are quite complex, so Nick only had a quick look at
the basics. The systems and environments (e.g. gcc version 8.1 for Skylake, 8.0
for POWER9) are not completely apples to apples so for now patterns are more
important than the absolute results. Interestingly the output video files between
architectures are not the same, particularly with different asm routines and 
compiler options used, which makes it difficult to verify the correctness of any changes.&lt;/p&gt;
&lt;p&gt;All tests were run single threaded to avoid any SMT effects.&lt;/p&gt;
&lt;p&gt;With the default upstream build of x264, Skylake is significantly faster than POWER9 on this benchmark
(Skylake: 9.20 fps, POWER9: 3.39 fps). POWER9 contains some vectorised routines, so an
initial suspicion is that Skylake's larger vector size may be responsible for its higher throughput.&lt;/p&gt;
&lt;p&gt;Let's test our vector size suspicion by restricting
Skylake to SSE4.2 code (with 128 bit vectors, the same width as POWER9). This hardly
slows down the x86 CPU at all (Skylake: 8.37 fps, POWER9: 3.39 fps), which indicates it's
not taking much advantage of the larger vectors.&lt;/p&gt;
&lt;p&gt;So the next guess would be that x86 just has more and better optimized versions of costly
functions (in the version of x264 that Phoronix used there are only six powerpc specific
files compared with 21 x86 specific files). Without the time or expertise to dig into the
complex task of writing vector code, we'll see if the compiler can help, and turn
on autovectorisation (x264 compiles with -fno-tree-vectorize by default, which disables 
auto vectorization). Looking at a perf profile of the benchmark we can see
that one costly function, quant_4x4x4, is not autovectorised. With a small change to the
code, gcc does vectorise it, giving a slight speedup with the output file checksum unchanged
(Skylake: 9.20 fps, POWER9: 3.83 fps).&lt;/p&gt;
&lt;p&gt;We got a small improvement with the compiler, but it looks like we may have gains left on the
table with our vector code. If you're interested in looking into this, we do have some
&lt;a href="https://www.bountysource.com/teams/ibm/bounties"&gt;active bounties&lt;/a&gt; for x264 (lu-zero/x264).&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;Skylake&lt;/th&gt;
&lt;th&gt;POWER9&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Original - AVX256&lt;/td&gt;
&lt;td&gt;9.20 fps&lt;/td&gt;
&lt;td&gt;3.39 fps&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Original - SSE4.2&lt;/td&gt;
&lt;td&gt;8.37 fps&lt;/td&gt;
&lt;td&gt;3.39 fps&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Autovectorisation enabled, quant_4x4x4 vectorised&lt;/td&gt;
&lt;td&gt;9.20 fps&lt;/td&gt;
&lt;td&gt;3.83 fps&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Nick also investigated running this benchmark with SMT enabled and across multiple cores, and it looks like the code is
not scalable enough to feed 176 threads on a 44 core system. Disabling SMT in parallel runs
actually helped, but there was still idle time. That may be another thing to look at,
although it may not be such a problem for smaller systems.&lt;/p&gt;
&lt;h3&gt;Primesieve&lt;/h3&gt;
&lt;p&gt;&lt;a href="https://primesieve.org/"&gt;Primesieve&lt;/a&gt; is a program and C/C++
library that generates all the prime numbers below a given number. It uses an
optimised &lt;a href="https://upload.wikimedia.org/wikipedia/commons/b/b9/Sieve_of_Eratosthenes_animation.gif"&gt;Sieve of Eratosthenes&lt;/a&gt;
implementation.&lt;/p&gt;
&lt;p&gt;The algorithm uses the L1 cache size as the sieve size for the core loop.  This
is an issue when we are running in SMT mode (aka more than one thread per core)
as all threads on a core share the same L1 cache and so will constantly be 
invalidating each others cache-lines. As you can see
in the table below, running the benchmark in single threaded mode is 30% faster
than in SMT4 mode!&lt;/p&gt;
&lt;p&gt;This means in SMT-4 mode the workload is about 4x too large for the L1 cache.  A
better sieve size to use would be the L1 cache size / number of
threads per core. Anton posted a &lt;a href="https://github.com/kimwalisch/primesieve/pull/54"&gt;pull request&lt;/a&gt; 
to update the sieve size.&lt;/p&gt;
&lt;p&gt;It is interesting that the best overall performance on POWER9 is with the patch applied and in
SMT2 mode:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;SMT level&lt;/th&gt;
&lt;th&gt;baseline&lt;/th&gt;
&lt;th&gt;patched&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;14.728s&lt;/td&gt;
&lt;td&gt;14.899s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;15.362s&lt;/td&gt;
&lt;td&gt;14.040s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;19.489s&lt;/td&gt;
&lt;td&gt;17.458s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3&gt;LAME&lt;/h3&gt;
&lt;p&gt;Despite its name, a recursive acronym for "LAME Ain't an MP3 Encoder",
&lt;a href="http://lame.sourceforge.net/"&gt;LAME&lt;/a&gt; is indeed an MP3 encoder.&lt;/p&gt;
&lt;p&gt;Due to configure options &lt;a href="https://sourceforge.net/p/lame/mailman/message/36371506/"&gt;not being parsed correctly&lt;/a&gt; this
benchmark is built without any optimisation regardless of architecture. We see a
massive speedup by turning optimisations on, and a further 6-8% speedup by
enabling
&lt;a href="https://sourceforge.net/p/lame/mailman/message/36372005/"&gt;USE_FAST_LOG&lt;/a&gt; (which
is already enabled for Intel).&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;LAME&lt;/th&gt;
&lt;th&gt;Duration&lt;/th&gt;
&lt;th&gt;Speedup&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Default&lt;/td&gt;
&lt;td&gt;82.1s&lt;/td&gt;
&lt;td&gt;n/a&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;With optimisation flags&lt;/td&gt;
&lt;td&gt;16.3s&lt;/td&gt;
&lt;td&gt;5.0x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;With optimisation flags and USE_FAST_LOG set&lt;/td&gt;
&lt;td&gt;15.6s&lt;/td&gt;
&lt;td&gt;5.3x&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;For more detail see Joel's
&lt;a href="https://shenki.github.io/LameMP3-on-Power9/"&gt;writeup&lt;/a&gt;.&lt;/p&gt;
&lt;h3&gt;FLAC&lt;/h3&gt;
&lt;p&gt;&lt;a href="https://xiph.org/flac/"&gt;FLAC&lt;/a&gt; is an alternative encoding format to
MP3. But unlike MP3 encoding it is lossless!  The benchmark here was encoding
audio files into the FLAC format. &lt;/p&gt;
&lt;p&gt;The key part of this workload is missing
vector support for POWER8 and POWER9. Anton and Amitay submitted this
&lt;a href="http://lists.xiph.org/pipermail/flac-dev/2018-July/006351.html"&gt;patch series&lt;/a&gt; that
adds in POWER specific vector instructions. It also fixes the configuration options
to correctly detect the POWER8 and POWER9 platforms. With this patch series we get see about a 3x
improvement in this benchmark.&lt;/p&gt;
&lt;h3&gt;OpenSSL&lt;/h3&gt;
&lt;p&gt;&lt;a href="https://www.openssl.org/"&gt;OpenSSL&lt;/a&gt; is among other things a cryptographic library. The Phoronix benchmark
measures the number of RSA 4096 signs per second:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;openssl&lt;span class="w"&gt; &lt;/span&gt;speed&lt;span class="w"&gt; &lt;/span&gt;-multi&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;nproc&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;rsa4096
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Phoronix used OpenSSL-1.1.0f, which is almost half as slow for this benchmark (on POWER9) than mainline OpenSSL.
Mainline OpenSSL has some powerpc multiplication and squaring assembly code which seems
to be responsible for most of this speedup.&lt;/p&gt;
&lt;p&gt;To see this for yourself, add these four powerpc specific commits on top of OpenSSL-1.1.0f:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="https://github.com/openssl/openssl/commit/b17ff188b17499e83ca3b9df0be47a2f513ac3c5"&gt;perlasm/ppc-xlate.pl: recognize .type directive&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/openssl/openssl/commit/0310becc82d240288a4ab5c6656c10c18cab4454"&gt;bn/asm/ppc-mont.pl: prepare for extension&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/openssl/openssl/commit/68f6d2a02c8cc30c5c737fc948b7cf023a234b47"&gt;bn/asm/ppc-mont.pl: add optimized multiplication and squaring subroutines&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/openssl/openssl/commit/80d27cdb84985c697f8fabb7649abf1f54714d13"&gt;ppccap.c: engage new multipplication and squaring subroutines&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The following results were from a dual 16-core POWER9:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Version of OpenSSL&lt;/th&gt;
&lt;th&gt;Signs/s&lt;/th&gt;
&lt;th&gt;Speedup&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1.1.0f&lt;/td&gt;
&lt;td&gt;1921&lt;/td&gt;
&lt;td&gt;n/a&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1.1.0f with 4 patches&lt;/td&gt;
&lt;td&gt;3353&lt;/td&gt;
&lt;td&gt;1.74x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1.1.1-pre1&lt;/td&gt;
&lt;td&gt;3383&lt;/td&gt;
&lt;td&gt;1.76x&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3&gt;SciKit-Learn&lt;/h3&gt;
&lt;p&gt;&lt;a href="http://scikit-learn.org/"&gt;SciKit-Learn&lt;/a&gt; is a bunch of python tools for data mining and
analysis (aka machine learning).&lt;/p&gt;
&lt;p&gt;Joel noticed that the benchmark spent 92% of the time in libblas. Libblas is a
very basic BLAS (basic linear algebra subprograms) library that python-numpy
uses to do vector and matrix operations.  The default libblas on Ubuntu is only
compiled with -O2. Compiling with -Ofast and using alternative BLAS's that have
powerpc optimisations (such as libatlas or libopenblas) we see big improvements
in this benchmark:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;BLAS used&lt;/th&gt;
&lt;th&gt;Duration&lt;/th&gt;
&lt;th&gt;Speedup&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;libblas -O2&lt;/td&gt;
&lt;td&gt;64.2s&lt;/td&gt;
&lt;td&gt;n/a&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;libblas -Ofast&lt;/td&gt;
&lt;td&gt;36.1s&lt;/td&gt;
&lt;td&gt;1.8x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;libatlas&lt;/td&gt;
&lt;td&gt;8.3s&lt;/td&gt;
&lt;td&gt;7.7x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;libopenblas&lt;/td&gt;
&lt;td&gt;4.2s&lt;/td&gt;
&lt;td&gt;15.3x&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;You can read more details about this
&lt;a href="https://shenki.github.io/Scikit-Learn-on-Power9/"&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;h3&gt;Blender&lt;/h3&gt;
&lt;p&gt;&lt;a href="https://www.blender.org/"&gt;Blender&lt;/a&gt; is a 3D graphics suite that supports image rendering,
animation, simulation and game creation. On the surface it appears that Blender
2.79b (the distro package version that Phoronix used by system/blender-1.0.2)
failed to use more than 15 threads, even when "-t 128" was added to the Blender
command line.&lt;/p&gt;
&lt;p&gt;It turns out that even though this benchmark was supposed to be run on CPUs only
(you can choose to render on CPUs or GPUs), the GPU file was always being used.
The GPU file is configured with a very large tile size (256x256) -
which is &lt;a href="https://docs.blender.org/manual/en/dev/render/cycles/settings/scene/render/performance.html#tiles"&gt;fine for
GPUs&lt;/a&gt;
but not great for CPUs. The image size (1280x720) to tile size ratio limits the
number of jobs created and therefore the number threads used.&lt;/p&gt;
&lt;p&gt;To obtain a realistic CPU measurement with more that 15 threads you can force
the use of the CPU file by overwriting the GPU file with the CPU one:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;cp
~/.phoronix-test-suite/installed-tests/system/blender-1.0.2/benchmark/pabellon_barcelona/pavillon_barcelone_cpu.blend
~/.phoronix-test-suite/installed-tests/system/blender-1.0.2/benchmark/pabellon_barcelona/pavillon_barcelone_gpu.blend
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;As you can see in the image below, now all of the cores are being utilised!
&lt;img alt="Blender with CPU Blend file" src="/images/phoronix/blender-88threads.png" title="Blender with CPU Blend file"&gt;&lt;/p&gt;
&lt;p&gt;Fortunately this has already been fixed in 
&lt;a href="https://openbenchmarking.org/test/pts/blender"&gt;pts/blender-1.1.1&lt;/a&gt;.
Thanks to the &lt;a href="https://github.com/phoronix-test-suite/test-profiles/issues/24"&gt;report&lt;/a&gt; by Daniel it
has also been fixed in &lt;a href="http://openbenchmarking.org/test/system/blender-1.1.0"&gt;system/blender-1.1.0&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Pinning the pts/bender-1.0.2, Pabellon Barcelona, CPU-Only test to a single
22-core POWER9 chip (&lt;code&gt;sudo ppc64_cpu --cores-on=22&lt;/code&gt;) and two POWER9 chips
(&lt;code&gt;sudo ppc64_cpu --cores-on=44&lt;/code&gt;) show a huge speedup:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Benchmark&lt;/th&gt;
&lt;th&gt;Duration (deviation over 3 runs)&lt;/th&gt;
&lt;th&gt;Speedup&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Baseline (GPU blend file)&lt;/td&gt;
&lt;td&gt;1509.97s (0.30%)&lt;/td&gt;
&lt;td&gt;n/a&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Single 22-core POWER9 chip (CPU blend file)&lt;/td&gt;
&lt;td&gt;458.64s (0.19%)&lt;/td&gt;
&lt;td&gt;3.29x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Two 22-core POWER9 chips (CPU blend file)&lt;/td&gt;
&lt;td&gt;241.33s (0.25%)&lt;/td&gt;
&lt;td&gt;6.25x&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3&gt;tl;dr&lt;/h3&gt;
&lt;p&gt;Some of the benchmarks where we don't perform as well as Intel are where the
benchmark has inline assembly for x86 but uses generic C compiler generated
assembly for POWER9. We could probably benefit with some more powerpc optimsed functions.&lt;/p&gt;
&lt;p&gt;We also found a couple of things that should result in better performance for all three architectures,
not just POWER.&lt;/p&gt;
&lt;p&gt;A summary of the performance improvements we found:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Benchmark&lt;/th&gt;
&lt;th&gt;Approximate Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Parboil&lt;/td&gt;
&lt;td&gt;3x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;x264&lt;/td&gt;
&lt;td&gt;1.1x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Primesieve&lt;/td&gt;
&lt;td&gt;1.1x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LAME&lt;/td&gt;
&lt;td&gt;5x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;FLAC&lt;/td&gt;
&lt;td&gt;3x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenSSL&lt;/td&gt;
&lt;td&gt;2x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SciKit-Learn&lt;/td&gt;
&lt;td&gt;7-15x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Blender&lt;/td&gt;
&lt;td&gt;3x&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;There is obviously room for more improvements, especially with the Primesieve and x264 benchmarks,
but it would be interesting to see a re-run of the Phoronix benchmarks with these changes. &lt;/p&gt;
&lt;p&gt;Thanks to Anton, Daniel, Joel and Nick for the analysis of the above benchmarks.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Rashmica Gupta</dc:creator><pubDate>Wed, 15 Aug 2018 14:22:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2018-08-15:/blog/2018/08/15/improving-performance-of-phoronix-benchmarks-on-power9/</guid><category>Performance</category><category>performance</category><category>phoronix</category><category>benchmarks</category></item><item><title>Stupid Solutions to Stupid Problems: Hardcoding Your SSH Key in the Kernel</title><link>https://sthbrx.github.io/blog/2017/09/23/stupid-solutions-to-stupid-problems-hardcoding-your-ssh-key-in-the-kernel/</link><description>&lt;h2&gt;The "problem"&lt;/h2&gt;
&lt;p&gt;I'm currently working on firmware and kernel support for &lt;a href="http://opencapi.org/"&gt;OpenCAPI&lt;/a&gt; on POWER9.&lt;/p&gt;
&lt;p&gt;I've recently been allocated a machine in the lab for development purposes. We use an internal IBM tool running on a secondary machine that triggers hardware initialisation procedures, then loads a specified &lt;a href="https://github.com/open-power/skiboot"&gt;skiboot&lt;/a&gt; firmware image, a kernel image, and a root file system directly into RAM. This allows us to get skiboot and Linux running without requiring the usual &lt;a href="https://github.com/open-power/hostboot"&gt;hostboot&lt;/a&gt; initialisation and gives us a lot of options for easier tinkering, so it's super-useful for our developers working on bringup.&lt;/p&gt;
&lt;p&gt;When I got access to my machine, I figured out the necessary scripts, developed a workflow, and started fixing my code... so far, so good.&lt;/p&gt;
&lt;p&gt;One day, I was trying to debug something and get logs off the machine using &lt;code&gt;ssh&lt;/code&gt; and &lt;code&gt;scp&lt;/code&gt;, when I got frustrated with having to repeatedly type in our ultra-secret, ultra-secure root password, &lt;code&gt;abc123&lt;/code&gt;. So, I ran &lt;code&gt;ssh-copy-id&lt;/code&gt; to copy over my public key, and all was good.&lt;/p&gt;
&lt;p&gt;Until I rebooted the machine, when strangely, my key stopped working. It took me longer than it should have to realise that this is an obvious consequence of running entirely from an initrd that's reloaded every boot...&lt;/p&gt;
&lt;h2&gt;The "solution"&lt;/h2&gt;
&lt;p&gt;I mentioned something about this to Jono, my housemate/partner-in-stupid-ideas, one evening a few weeks ago. We decided that clearly, the best way to solve this problem was to hardcode my SSH public key in the kernel.&lt;/p&gt;
&lt;p&gt;This would definitely be the easiest and most sensible way to solve the problem, as opposed to, say, just keeping my own copy of the root filesystem image. Or asking &lt;a href="https://twitter.com/mikeyneuling"&gt;Mikey&lt;/a&gt;, whose desk is three metres away from mine, whether he could use his write access to add my key to the image. Or just writing a wrapper around &lt;a href="https://linux.die.net/man/1/sshpass"&gt;sshpass&lt;/a&gt;...&lt;/p&gt;
&lt;p&gt;One Tuesday afternoon, I was feeling bored...&lt;/p&gt;
&lt;h2&gt;The approach&lt;/h2&gt;
&lt;p&gt;The SSH daemon looks for authorised public keys in &lt;code&gt;~/.ssh/authorized_keys&lt;/code&gt;, so we need to have a read of &lt;code&gt;/root/.ssh/authorized_keys&lt;/code&gt; return a specified hard-coded string.&lt;/p&gt;
&lt;p&gt;I did a bit of investigation. My first thought was to put some kind of hook inside whatever filesystem driver was being used for the root. After some digging, I found out that the filesystem type &lt;code&gt;rootfs&lt;/code&gt;, as seen in &lt;code&gt;mount&lt;/code&gt;, is actually backed by the &lt;code&gt;tmpfs&lt;/code&gt; filesystem. I took a look around the &lt;code&gt;tmpfs&lt;/code&gt; code for a while, but didn't see any way to hook in a fake file without a lot of effort - the &lt;code&gt;tmpfs&lt;/code&gt; code wasn't exactly designed with this in mind.&lt;/p&gt;
&lt;p&gt;I thought about it some more - what would be the easiest way to create a file such that it just returns a string?&lt;/p&gt;
&lt;p&gt;Then I remembered sysfs, the filesystem normally mounted at &lt;code&gt;/sys&lt;/code&gt;, which is used by various kernel subsystems to expose configuration and debugging information to userspace in the form of files. The sysfs API allows you to define a file and specify callbacks to handle reads and writes to the file.&lt;/p&gt;
&lt;p&gt;That got me thinking - could I create a file in &lt;code&gt;/sys&lt;/code&gt;, and then use a &lt;a href="https://unix.stackexchange.com/questions/198590/what-is-a-bind-mount"&gt;bind mount&lt;/a&gt; to have that file appear where I need it in &lt;code&gt;/root/.ssh/authorized_keys&lt;/code&gt;? This approach seemed fairly straightforward, so I decided to give it a try.&lt;/p&gt;
&lt;p&gt;First up, creating a pseudo-file. It had been a while since the last time I'd used the sysfs API...&lt;/p&gt;
&lt;h2&gt;sysfs&lt;/h2&gt;
&lt;p&gt;The sysfs pseudo file system was first introduced in Linux 2.6, and is generally used for exposing system and device information.&lt;/p&gt;
&lt;p&gt;Per the &lt;a href="https://www.kernel.org/doc/Documentation/filesystems/sysfs.txt"&gt;sysfs documentation&lt;/a&gt;, sysfs is tied in very closely with the &lt;a href="https://www.kernel.org/doc/Documentation/kobject.txt"&gt;kobject&lt;/a&gt; infrastructure. sysfs exposes kobjects as directories, containing "attributes" represented as files. The kobject infrastructure provides a way to define kobjects representing entities (e.g. devices) and ksets which define collections of kobjects (e.g. devices of a particular type).&lt;/p&gt;
&lt;p&gt;Using kobjects you can do lots of fancy things such as sending events to userspace when devices are hotplugged - but that's all out of the scope of this post. It turns out there's some fairly straightforward wrapper functions if all you want to do is create a kobject just to have a simple directory in sysfs.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;linux/kobject.h&amp;gt;&lt;/span&gt;

&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__init&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;ssh_key_init&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;kobject&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;ssh_kobj&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;ssh_kobj&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kobject_create_and_add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;ssh&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;ssh_kobj&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="n"&gt;pr_err&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;SSH: kobject creation failed!&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;ENOMEM&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;late_initcall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ssh_key_init&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This creates and adds a kobject called &lt;code&gt;ssh&lt;/code&gt;. And just like that, we've got a directory in &lt;code&gt;/sys/ssh/&lt;/code&gt;!&lt;/p&gt;
&lt;p&gt;The next thing we have to do is define a sysfs attribute for our &lt;code&gt;authorized_keys&lt;/code&gt; file. sysfs provides a framework for subsystems to define their own custom types of attributes with their own metadata - but for our purposes, we'll use the generic &lt;code&gt;bin_attribute&lt;/code&gt; attribute type.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;linux/sysfs.h&amp;gt;&lt;/span&gt;

&lt;span class="k"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;PUBLIC KEY HERE...&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;ssize_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;show_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;file&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;kobject&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;kobj&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;bin_attribute&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;bin_attr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="n"&gt;loff_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pos&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;size_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;memory_read_from_buffer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;pos&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bin_attr&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;bin_attribute&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;authorized_keys_attr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;authorized_keys&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mo"&gt;0444&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;show_key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;sizeof&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;We provide a simple callback, &lt;code&gt;show_key()&lt;/code&gt;, that copies the key string into the file's buffer, and we put it in a &lt;code&gt;bin_attribute&lt;/code&gt; with the appropriate name, size and permissions.&lt;/p&gt;
&lt;p&gt;To actually add the attribute, we put the following in &lt;code&gt;ssh_key_init()&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sysfs_create_bin_file&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ssh_kobj&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;authorized_keys_attr&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;pr_err&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;SSH: sysfs creation failed, rc %d&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Woo, we've now got &lt;code&gt;/sys/ssh/authorized_keys&lt;/code&gt;! Time to move on to the bind mount.&lt;/p&gt;
&lt;h2&gt;Mounting&lt;/h2&gt;
&lt;p&gt;Now that we've got a directory with the key file in it, it's time to figure out the bind mount.&lt;/p&gt;
&lt;p&gt;Because I had no idea how any of the file system code works, I started off by running &lt;code&gt;strace&lt;/code&gt; on &lt;code&gt;mount --bind ~/tmp1 ~/tmp2&lt;/code&gt; just to see how the userspace &lt;code&gt;mount&lt;/code&gt; tool uses the &lt;code&gt;mount&lt;/code&gt; syscall to request the bind mount.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;execve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/bin/mount&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;mount&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;--bind&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/home/ajd/tmp1&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/home/ajd/tmp2&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="cm"&gt;/* 18 vars */&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;

&lt;span class="p"&gt;...&lt;/span&gt;

&lt;span class="n"&gt;mount&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/home/ajd/tmp1&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/home/ajd/tmp2&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x18b78bf00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MS_MGC_VAL&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;MS_BIND&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The first and second arguments are the source and target paths respectively. The third argument, looking at the signature of the &lt;code&gt;mount&lt;/code&gt; syscall, is a pointer to a string with the file system type. Because this is a bind mount, the type is irrelevant (upon further digging, it turns out that this particular pointer is to the string "none").&lt;/p&gt;
&lt;p&gt;The fourth argument is where we specify the flags bitfield. &lt;code&gt;MS_MGC_VAL&lt;/code&gt; is a magic value that was required before Linux 2.4 and can now be safely ignored. &lt;code&gt;MS_BIND&lt;/code&gt;, as you can probably guess, signals that we want a bind mount.&lt;/p&gt;
&lt;p&gt;(The final argument is used to pass file system specific data - as you can see it's ignored here.)&lt;/p&gt;
&lt;p&gt;Now, how is the syscall actually handled on the kernel side? The answer is found in &lt;a href="http://elixir.free-electrons.com/linux/latest/source/fs/namespace.c#L2969"&gt;fs/namespace.c&lt;/a&gt;.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;SYSCALL_DEFINE5&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__user&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dev_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__user&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dir_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__user&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__user&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ret&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="cm"&gt;/* ... copy parameters from userspace memory ... */&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;ret&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;do_mount&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kernel_dev&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dir_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;options&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="cm"&gt;/* ... cleanup ... */&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;So in order to achieve the same thing from within the kernel, we just call &lt;code&gt;do_mount()&lt;/code&gt; with exactly the same parameters as the syscall uses:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;do_mount&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/sys/ssh&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/root/.ssh&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;sysfs&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MS_BIND&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;pr_err&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;SSH: bind mount failed, rc %d&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;...and we're done, right? Not so fast:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;SSH&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bind&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mount&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;failed&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;-2 is &lt;code&gt;ENOENT&lt;/code&gt; - no such file or directory. For some reason, we can't find &lt;code&gt;/sys/ssh&lt;/code&gt;... of course, that would be because even though we've created the sysfs entry, we haven't actually mounted sysfs on &lt;code&gt;/sys&lt;/code&gt;.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;do_mount&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;sysfs&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/sys&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;sysfs&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="n"&gt;MS_NOSUID&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MS_NOEXEC&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MS_NODEV&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;At this point, my key worked!&lt;/p&gt;
&lt;p&gt;Note that this requires that your root file system has an empty directory created at &lt;code&gt;/sys&lt;/code&gt; to be the mount point. Additionally, in a typical Linux distribution environment (as opposed to my hardware bringup environment), your initial root file system will contain an init script that mounts your real root file system somewhere and calls &lt;code&gt;pivot_root()&lt;/code&gt; to switch to the new root file system. At that point, the bind mount won't be visible from children processes using the new root - I think this could be worked around but would require some effort.&lt;/p&gt;
&lt;h2&gt;Kconfig&lt;/h2&gt;
&lt;p&gt;The final piece of the puzzle is building our new code into the kernel image.&lt;/p&gt;
&lt;p&gt;To allow us to switch this important functionality on and off, I added a config option to &lt;code&gt;fs/Kconfig&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;config&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;SSH_KEY&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;bool&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Andrew&amp;#39;s dumb SSH key hack&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;default&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;y&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;help&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nv"&gt;Hardcode&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;an&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;SSH&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;key&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;root&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;.&lt;span class="nv"&gt;ssh&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;authorized_keys&lt;/span&gt;.

&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nv"&gt;This&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;stupid&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;idea&lt;/span&gt;.&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;If&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;unsure&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;say&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;N&lt;/span&gt;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This will show up in &lt;code&gt;make menuconfig&lt;/code&gt; under the &lt;code&gt;File systems&lt;/code&gt; menu.&lt;/p&gt;
&lt;p&gt;And in &lt;code&gt;fs/Makefile&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;obj-$(CONFIG_SSH_KEY)&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ssh_key.o
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If &lt;code&gt;CONFIG_SSH_KEY&lt;/code&gt; is set to &lt;code&gt;y&lt;/code&gt;, &lt;code&gt;obj-$(CONFIG_SSH_KEY)&lt;/code&gt; evaluates to &lt;code&gt;obj-y&lt;/code&gt; and thus &lt;code&gt;ssh-key.o&lt;/code&gt; gets compiled. Conversely, &lt;code&gt;obj-n&lt;/code&gt; is completely ignored by the build system.&lt;/p&gt;
&lt;p&gt;I thought I was all done... then &lt;a href="https://twitter.com/mramboar"&gt;Andrew&lt;/a&gt; suggested I make the contents of the key configurable, and I had to oblige. Conveniently, Kconfig options can also be strings:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;config&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;SSH_KEY_VALUE&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;string&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Value for SSH key&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;depends&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;SSH_KEY&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;help&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nv"&gt;Enter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;content&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;root&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;.&lt;span class="nv"&gt;ssh&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;authorized_keys&lt;/span&gt;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Including the string in the C file is as simple as:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;CONFIG_SSH_KEY_VALUE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And there we have it, a nicely configurable albeit highly limited kernel SSH backdoor!&lt;/p&gt;
&lt;h2&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;I've put the &lt;a href="https://github.com/ajdlinux/linux/commit/052c0cb7296f7510fd482fecbe572b641c29239f"&gt;full code&lt;/a&gt; up on GitHub for perusal. Please don't use it, I will be extremely disappointed in you if you do.&lt;/p&gt;
&lt;p&gt;Thanks to Jono for giving me stupid ideas, and the rest of OzLabs for being very angry when they saw the disgusting things I was doing.&lt;/p&gt;
&lt;p&gt;Comments and further stupid suggestions welcome!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Donnellan</dc:creator><pubDate>Sat, 23 Sep 2017 03:00:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-09-23:/blog/2017/09/23/stupid-solutions-to-stupid-problems-hardcoding-your-ssh-key-in-the-kernel/</guid><category>Development</category><category>kernel</category><category>stupid ideas</category></item><item><title>NCSI - Nice Network You've Got There</title><link>https://sthbrx.github.io/blog/2017/09/22/ncsi-nice-network-youve-got-there/</link><description>&lt;p&gt;A neat piece of kernel code dropped into my lap recently, and as a way of
processing having to inject an entire network stack into by brain in
less-than-ideal time I thought we'd have a look at it here: NCSI!&lt;/p&gt;
&lt;h2&gt;NCSI - Not the TV Show&lt;/h2&gt;
&lt;p&gt;NCSI stands for Network Controller Sideband Interface, and put most simply it
is a way for a management controller (eg. a BMC like those found on our OpenPOWER
machines) to share a single physical network interface with a host machine.
Instead of two distinct network interfaces you plug in a single cable and both
the host and the BMC have network connectivity.&lt;/p&gt;
&lt;p&gt;NCSI-capable network controllers achieve this by filtering network traffic as
it arrives and determining if it is host- or BMC-bound. To know how to do this
the BMC needs to tell the network controller what to look out for, and from a
Linux driver perspective this the focus of the NCSI protocol.&lt;/p&gt;
&lt;p&gt;&lt;img alt="NCSI Overview" src="/images/sammj/ncsi_overview.png"&gt;&lt;/p&gt;
&lt;h2&gt;Hi My Name Is 70:e2:84:14:24:a1&lt;/h2&gt;
&lt;p&gt;The major components of what NCSI helps facilitate are:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Network Controllers, known as 'Packages' in this context. There may be multiple
  separate packages which contain one or more Channels.&lt;/li&gt;
&lt;li&gt;Channels, most easily thought of as the individual physical network interfaces.
  If a package is the network card, channels are the individual network jacks. (Somewhere a pedant's head is spinning in circles).&lt;/li&gt;
&lt;li&gt;Management Controllers, or our BMC, with their own network interfaces. Hypothetically there can be multiple
  management controllers in a single NCSI system, but I've not come across such
  a setup yet.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;NCSI is the medium and protocol via which these components communicate.&lt;/p&gt;
&lt;p&gt;&lt;img alt="NCSI Packages" src="/images/sammj/ncsi_packages.png"&gt;&lt;/p&gt;
&lt;p&gt;The interface between Management Controller and one or more
Packages carries both general network traffic to/from the Management
Controller as well as NCSI traffic between the Management Controller
and the Packages &amp;amp; Channels. Management traffic is differentiated from
regular traffic via the inclusion of a special NCSI tag inserted
in the Ethernet frame header.
These management commands are used to discover and configure the state of the
NCSI packages and channels.&lt;/p&gt;
&lt;p&gt;If a BMC's network interface is configured to use NCSI, as soon as the interface
is brought up NCSI gets to work finding and configuring a usable channel.
The NCSI driver at first glance is an intimidating combination of state machines
and packet handlers, but with enough coffee it can be represented like this:&lt;/p&gt;
&lt;p&gt;&lt;img alt="NCSI State Diagram" src="/images/sammj/ncsi_states.png"&gt;&lt;/p&gt;
&lt;p&gt;Without getting into the nitty gritty details the overall process for configuring
a channel enough to get packets flowing is fairly straightforward:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Find available packages.&lt;/li&gt;
&lt;li&gt;Find each package's available channels.&lt;/li&gt;
&lt;li&gt;(At least in the Linux driver) select a channel with link.&lt;/li&gt;
&lt;li&gt;Put this channel into the Initial Config State.
The Initial Config State is where all the useful configuration occurs. Here we
find out what the selected channel is capable of and its current configuration,
and set it up to recognise the traffic we're interested in. The first and most
basic way of doing this is configuring the channel to filter traffic based on
our MAC address.&lt;/li&gt;
&lt;li&gt;Enable the channel and let the packets flow.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;At this point NCSI takes a back seat to normal network traffic, transmitting
a "Get Link Status" packet at regular intervals to monitor the channel.&lt;/p&gt;
&lt;h2&gt;AEN Packets&lt;/h2&gt;
&lt;p&gt;Changes can occur from the package side too; the NCSI package communicates these
back to the BMC with Asynchronous Event Notification (AEN) packets. As the name
suggests these can occur at any time and the driver needs to catch and handle these.
There are different types but they essentially boil down to changes in link state,
telling the BMC the channel needs to be reconfigured, or to select a different
channel.
These are only transmitted once and no effort is made to recover lost AEN packets -
another good reason for the NCSI driver to periodically monitor the channel.&lt;/p&gt;
&lt;h2&gt;Filtering&lt;/h2&gt;
&lt;p&gt;Each channel can be configured to filter traffic based on MAC address,
broadcast traffic, multicast traffic, and VLAN tagging. Associated with each of
these filters is a &lt;em&gt;filter table&lt;/em&gt; which can hold a finite number of entries.
In the case of the VLAN filter each channel could match against 15 different
VLAN IDs for example, but in practice the physical device will likely
support less. Indeed the popular BCM5718 controller supports only two!&lt;/p&gt;
&lt;p&gt;This is where I dived into NCSI. The driver had a lot of the pieces for
configuring VLAN filters but none of it was actually hooked up in the configure
state, and didn't have a way of actually knowing which VLAN IDs were meant to be
configured on the interface. The bulk of that work appears in &lt;a href="https://github.com/torvalds/linux/commit/21acf63013ed3d6fce3176cc34b74064052a31b4#diff-f391518f4e552724349be3589e00dfa7"&gt;this commit&lt;/a&gt; where we take advantage of some useful network stack callbacks to get the VLAN configuration and set them during the configuration state. Getting &lt;em&gt;to&lt;/em&gt; the configuration state at some arbitrary time and then managing to assign multiple IDs was the trickiest bit, and is something I'll be looking at simplifying in the future.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;NCSI! A neat way to give physically separate users access to a single network controller, and if it works right you won't notice it at all. I'll surely be spending more time here (fleshing out the driver's features, better error handling, and making the state machine a touch more readable to start, and I haven't even &lt;em&gt;mentioned&lt;/em&gt; HWA), so watch this space!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Fri, 22 Sep 2017 10:08:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-09-22:/blog/2017/09/22/ncsi-nice-network-youve-got-there/</guid><category>Development</category><category>linux</category><category>kernel</category><category>goodposts</category><category>realcontent</category><category>network</category><category>sparseposting</category><category>openpower</category><category>openbmc</category></item><item><title>memcmp() for POWER8 - part II</title><link>https://sthbrx.github.io/blog/2017/09/01/memcmp-for-power8-part-ii/</link><description>&lt;p&gt;This entry is a followup to part I which you should absolutely read
&lt;a href="https://sthbrx.github.io/blog/2017/08/07/memcmp-for-power8/"&gt;here&lt;/a&gt; before continuing
on.&lt;/p&gt;
&lt;h2&gt;Where we left off&lt;/h2&gt;
&lt;p&gt;We concluded that while a vectorised &lt;code&gt;memcmp()&lt;/code&gt; is a win, there are
some cases where it won't quite perform.&lt;/p&gt;
&lt;h2&gt;The overhead of enabling ALTIVEC&lt;/h2&gt;
&lt;p&gt;In the kernel we explicitly don't touch ALTIVEC unless we need to,
this means that in the general case we can leave the userspace
registers in place and not have do anything to service a syscall for a
process.&lt;/p&gt;
&lt;p&gt;This means that if we do want to use ALTIVEC in the kernel, there is
some setup that must be done. Notably, we must enable the facility (a
potentially time consuming move to MSR), save off the registers (if
userspace we using them) and an inevitable restore later on.&lt;/p&gt;
&lt;p&gt;If all this needs to be done for a &lt;code&gt;memcmp()&lt;/code&gt; in the order of tens of
bytes then it really wasn't worth it.&lt;/p&gt;
&lt;p&gt;There are two reasons that &lt;code&gt;memcmp()&lt;/code&gt; might go for a small number of
bytes, firstly and trivially detectable is simply that parameter n is
small. The other is harder to detect, if the memcmp() is going to fail
(return non zero) early then it also wasn't worth enabling ALTIVEC.&lt;/p&gt;
&lt;h2&gt;Detecting early failures&lt;/h2&gt;
&lt;p&gt;Right at the start of &lt;code&gt;memcmp()&lt;/code&gt;, before enabling ALTIVEC, the first
64 bytes are checked using general purpose registers. Why the first 64
bytes, well why not? In a strange twist of fate 64 bytes happens to be
the amount of bytes in four ALTIVEC registers (128 bits per register,
so 16 bytes multiplied by 4) and by utter coincidence that happens to
be the stride of the ALTIVEC compare loop.&lt;/p&gt;
&lt;h2&gt;What does this all look like&lt;/h2&gt;
&lt;p&gt;Well unlike part I the results appear slightly less consistent across
three runs of measurement but there are some very key differences with
part I. The trends do appear to be the same across all three runs,
just less pronounced - why this is is unclear.&lt;/p&gt;
&lt;p&gt;The difference between run two and run three clipped at deltas of
1000ns is interesting:
&lt;img alt="Sample 2: Deltas below 1000ns" src="/images/power8_memcmp/v2deltas2-1000.png" title="Sample 2: Deltas below 1000ns"&gt;&lt;/p&gt;
&lt;p&gt;vs&lt;/p&gt;
&lt;p&gt;&lt;img alt="Sample 3: Deltas below 1000ns" src="/images/power8_memcmp/v2deltas3-1000.png" title="Sample 3: Deltas below 1000ns"&gt;&lt;/p&gt;
&lt;p&gt;The results are similar except for a spike in the amount of deltas in
the unpatched kernel at around 600ns. This is not present in the first
sample (deltas1) of data. There are a number of reasons why this spike
could have appeared here, it is possible that the kernel or hardware
did something under the hood, prefetch could have brought deltas for a
&lt;code&gt;memcmp()&lt;/code&gt; that would otherwise have yielded a greater delta into the
600ns range.&lt;/p&gt;
&lt;p&gt;What these two graphs do both demonstrate quite clearly is that
optimisations down at the sub 100ns end have resulted in more sub
100ns deltas for the patched kernel, a significant win over the
original data. Zooming out and looking at a graph which includes
deltas up to 5000ns shows that the sub 100ns delta optimisations
haven't noticeably slowed the performance of long duration &lt;code&gt;memcmp()&lt;/code&gt;,
&lt;img alt="Samply 2: Deltas below 5000ns" src="/images/power8_memcmp/v2deltas2-5000.png" title="Sample 2: Deltas below 5000ns"&gt;.&lt;/p&gt;
&lt;h2&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;The small amount of extra development effort has yielded tangible
results in reducing the low end &lt;code&gt;memcmp()&lt;/code&gt; times. This second round of
data collection and performance analysis only confirms the that for
any significant amount of comparison, a vectorised loop is
significantly quicker.&lt;/p&gt;
&lt;p&gt;The results obtained here show no downside to adopting this approach
for all power8 and onwards chips as this new version of the patch
solves the performance regression for small compares.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Cyril Bur</dc:creator><pubDate>Fri, 01 Sep 2017 12:00:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-09-01:/blog/2017/09/01/memcmp-for-power8-part-ii/</guid><category>OpenPOWER</category><category>performance</category><category>power</category></item><item><title>memcmp() for POWER8</title><link>https://sthbrx.github.io/blog/2017/08/07/memcmp-for-power8/</link><description>&lt;h2&gt;Userspace&lt;/h2&gt;
&lt;p&gt;When writing C programs in userspace there is libc which does so much
of the heavy lifting. One important thing libc provides is portability
in performing syscalls, that is, you don't need to know the
architectural details of performing a syscall on each architecture
your program might be compiled for. Another important feature that
libc provides for the average userspace programmer is highly optimised
routines to do things that are usually performance critical. It would
be extremely inefficient for each userspace programmer if they had to
implement even the naive version of these functions let alone
optimised versions. Let us take &lt;code&gt;memcmp()&lt;/code&gt; for example, I could
trivially implement this in C like:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;memcmp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;uint8_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;p1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;uint8_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;p2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;p2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;-1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p1&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;p2&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;However, while it is incredibly portable it is simply not going to
perform, which is why the nice people who write libc have highly
optimised ones in assembly for each architecture.&lt;/p&gt;
&lt;h2&gt;Kernel&lt;/h2&gt;
&lt;p&gt;When writing code for the Linux kernel, there isn't the luxury of a
fully featured libc since it expects (and needs) to be in userspace,
therefore we need to implement the features we need ourselves. Linux
doesn't need all the features but something like &lt;code&gt;memcmp()&lt;/code&gt; is
definitely a requirement.&lt;/p&gt;
&lt;p&gt;There have been some recent optimisations in &lt;a href="https://sourceware.org/git/?p=glibc.git;a=blob_plain;f=sysdeps/powerpc/powerpc64/power8/memcmp.S;h=46b9c0067ad7cd74a36c4800ebfe03eb1be0311e;hb=dec4a7105edcdbabdcac5f358f5bc5dca4f4ed1b" title="power8 optimised memcmp"&gt;glibc&lt;/a&gt; from which the
kernel could benefit too! The question to be asked is, does the glibc
optimised &lt;code&gt;power8_memcmp()&lt;/code&gt; actually go faster or is it all smoke and
mirrors?&lt;/p&gt;
&lt;h2&gt;Benchmarking &lt;code&gt;memcmp()&lt;/code&gt;&lt;/h2&gt;
&lt;p&gt;With things like &lt;code&gt;memcmp()&lt;/code&gt; it is actually quite easy to choose
datasets which can make any implementation look good. For example; the
new &lt;code&gt;power8_memcmp()&lt;/code&gt; makes use of the vector unit of the power8
processor, in order to do so in the kernel there must be a small
amount of setup code so that the rest of the kernel knows that the
vector unit has been used and it correctly saves and restores the
userspace vector registers. This means that &lt;code&gt;power8_memcmp()&lt;/code&gt; has a
slightly larger overhead than the current one, so for small compares
or compares which are different early on then the newer 'faster'
&lt;code&gt;power8_memcmp()&lt;/code&gt; might actually not perform as well. For any kind of
large compare however, using the vector unit should outperform a CPU
register load and compare loop. It is for this reason that I wanted to
avoid using micro benchmarks and use a 'real world' test as much as
possible.&lt;/p&gt;
&lt;p&gt;The biggest user of &lt;code&gt;memcmp()&lt;/code&gt; in the kernel, at least on POWER is Kernel
Samepage Merging (KSM). KSM provides code to inspect all the pages of
a running system to determine if they're identical and deduplicate
them if possible. This kind of feature allows for memory overcommit
when used in a KVM host environment as guest kernels are likely to
have a lot of similar, readonly pages which can be merged with no
overhead afterwards. In order to determine if the pages are the same
KSM must do a lot of page sized &lt;code&gt;memcmp()&lt;/code&gt;.&lt;/p&gt;
&lt;h2&gt;Performance&lt;/h2&gt;
&lt;p&gt;Performing a lot of page sized &lt;code&gt;memcmp()&lt;/code&gt; is the one flaw with this
test, the sizes of the &lt;code&gt;memcmp()&lt;/code&gt; don't vary, hopefully the data will be
'random' enough that we can still observe differences in the two
approaches.&lt;/p&gt;
&lt;p&gt;My approach for testing involved getting the delta of &lt;code&gt;ktime_get()&lt;/code&gt;
across calls to &lt;code&gt;memcmp()&lt;/code&gt; in &lt;code&gt;memcmp_pages()&lt;/code&gt; (mm/ksm.c). This actually
generated massive amounts of data, so, for consistency the following
analysis is performed on the first 400MB of deltas collected.&lt;/p&gt;
&lt;p&gt;The host was compiled with &lt;code&gt;powernv_defconfig&lt;/code&gt; and run out of a
ramdisk. For consistency the host was rebooted between each run so as
to not have any previous tests affect the next. The host was rebooted
a total of six times, the first three with my 'patched'
&lt;code&gt;power8_memcmp()&lt;/code&gt; kernel was booted the second three times with just
my data collection patch applied, the 'vanilla' kernel. Both
kernels are based off &lt;code&gt;4.13-rc3&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Each boot the following script was run and the resulting deltas file
saved somewhere before reboot. The command line argument was always
15.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;

ppc64_cpu&lt;span class="w"&gt; &lt;/span&gt;--smt&lt;span class="o"&gt;=&lt;/span&gt;off

&lt;span class="c1"&gt;#Host actually boots with ksm off but be sure&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/sys/kernel/mm/ksm/run

&lt;span class="c1"&gt;#Scan a lot of pages&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;999999&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/sys/kernel/mm/ksm/pages_to_scan

&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Starting QEMUs&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;while&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$i&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-lt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;qemu-system-ppc64&lt;span class="w"&gt; &lt;/span&gt;-smp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;1G&lt;span class="w"&gt; &lt;/span&gt;-nographic&lt;span class="w"&gt; &lt;/span&gt;-vga&lt;span class="w"&gt; &lt;/span&gt;none&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;-machine&lt;span class="w"&gt; &lt;/span&gt;pseries,accel&lt;span class="o"&gt;=&lt;/span&gt;kvm,kvm-type&lt;span class="o"&gt;=&lt;/span&gt;HV&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;-kernel&lt;span class="w"&gt; &lt;/span&gt;guest.kernel&lt;span class="w"&gt;  &lt;/span&gt;-initrd&lt;span class="w"&gt; &lt;/span&gt;guest.initrd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;-monitor&lt;span class="w"&gt; &lt;/span&gt;pty&lt;span class="w"&gt; &lt;/span&gt;-serial&lt;span class="w"&gt; &lt;/span&gt;pty&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;expr&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$i&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;+&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;

&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Letting all the VMs boot&amp;quot;&lt;/span&gt;
sleep&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;30&lt;/span&gt;

&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Turning KSM om&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/sys/kernel/mm/ksm/run

&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Letting KSM do its thing&amp;quot;&lt;/span&gt;
sleep&lt;span class="w"&gt; &lt;/span&gt;2m

&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/sys/kernel/mm/ksm/run

dd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/sys/kernel/debug/ksm/memcmp_deltas&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;of&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;deltas&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;bs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;4096&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;count&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;100&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The guest kernel was a &lt;code&gt;pseries_le_defconfig&lt;/code&gt; &lt;code&gt;4.13-rc3&lt;/code&gt; with the same
ramdisk the host used. It booted to the login prompt and was left to
idle.&lt;/p&gt;
&lt;h2&gt;Analysis&lt;/h2&gt;
&lt;p&gt;A variety of histograms were then generated in an attempt to see how
the behaviour of &lt;code&gt;memcmp()&lt;/code&gt; changed between the two implementations.
It should be noted here that the y axis in the following graphs is a
log scale as there were a lot of small deltas. The first observation
is that the vanilla kernel had more smaller deltas, this is made
particularly evident by the 'tally' points which are a running total
of all deltas with less than the tally value.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Sample 1 - Deltas below 200ns" src="/images/power8_memcmp/deltas1-200.png" title="Sample 1: Deltas below 200ns"&gt;
Graph 1 depicting the vanilla kernel having a greater amount of small
(sub 20ns) deltas than the patched kernel. The green points rise
faster (left to right) and higher than the yellow points.&lt;/p&gt;
&lt;p&gt;Still looking at the tallies, &lt;a href="/images/power8_memcmp/deltas1-200.png" title="Sample 1: Deltas below 200ns"&gt;graph 1&lt;/a&gt; also shows that the tally
of deltas is very close by the 100ns mark, which means that the
overhead of &lt;code&gt;power8_memcmp()&lt;/code&gt; is not too great.&lt;/p&gt;
&lt;p&gt;The problem with looking at only deltas under 200ns is that the
performance results we want, that is, the difference between the
algorithms is being masked by things like cache effects. To avoid this
problem is may be wise to look at longer running (larger delta)
&lt;code&gt;memcmp()&lt;/code&gt; calls.&lt;/p&gt;
&lt;p&gt;The following graph plots all deltas below 5000ns - still relatively
short calls to &lt;code&gt;memcmp()&lt;/code&gt; but an interesting trend emerges:
&lt;img alt="Sample 1 - Deltas below 5000ns" src="/images/power8_memcmp/deltas1-5000.png" title="Sample 1: Deltas below 5000ns"&gt;
Graph 2 shows that above 500ns the blue (patched kernel) points appear
to have all shifted left with respect to the purple (vanilla kernel)
points. This shows that for any &lt;code&gt;memcmp()&lt;/code&gt; which will take more than
500ns to get a result it is favourable to use &lt;code&gt;power8_memcmp()&lt;/code&gt; and it
is only detrimental to use  &lt;code&gt;power8_memcmp()&lt;/code&gt; if the time will be
under 50ns (a conservative estimate).&lt;/p&gt;
&lt;p&gt;It is worth noting that &lt;a href="/images/power8_memcmp/deltas1-200.png" title="Sample 1: Deltas below 200ns"&gt;graph 1&lt;/a&gt; and &lt;a href="/images/power8_memcmp/deltas1-5000.png" title="Sample 1: Deltas below 5000ns"&gt;graph 2&lt;/a&gt; are generated by
combining the first run of data collected from the vanilla and patched
kernels. All the deltas for both runs are can be viewed separately
&lt;a href="/images/power8_memcmp/vanilla_deltas1.png" title="All vanilla deltas"&gt;here for vanilla&lt;/a&gt; and &lt;a href="/images/power8_memcmp/patched_deltas1.png" title="All patched deltas"&gt;here for patched&lt;/a&gt;. Finally, the results
from the other four runs look very much identical and provide me with
a fair amount of confidence that these results make sense.&lt;/p&gt;
&lt;h2&gt;Conclusions&lt;/h2&gt;
&lt;p&gt;It is important to separate possible KSM optimisations with generic
&lt;code&gt;memcmp()&lt;/code&gt; optimisations, for example, perhaps KSM shouldn't be
calling &lt;code&gt;memcmp()&lt;/code&gt; if it suspects the first byte will differ. On the
other hand, things that &lt;code&gt;power8_memcmp()&lt;/code&gt; could do (which it currently
doesn't) is check the length parameter and perhaps avoid the overhead
of enabling kernel vector if the compare is less than some small
amount of bytes.&lt;/p&gt;
&lt;p&gt;It does seem like at least for the 'average case' glibcs
&lt;code&gt;power8_memcmp()&lt;/code&gt; is an improvement over what we have now.&lt;/p&gt;
&lt;h2&gt;Future work&lt;/h2&gt;
&lt;p&gt;A second round of data collection and plotting of delta vs position of
first byte to differ should confirm these results, this would mean a
more invasive patch to KSM.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Cyril Bur</dc:creator><pubDate>Mon, 07 Aug 2017 12:00:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-08-07:/blog/2017/08/07/memcmp-for-power8/</guid><category>OpenPOWER</category><category>performance</category><category>power</category></item><item><title>XDP on Power</title><link>https://sthbrx.github.io/blog/2017/07/17/xdp-on-power/</link><description>&lt;p&gt;This post is a bit of a break from the standard IBM fare of this blog,
as I now work for Canonical. But I have a soft spot for Power from my
time at IBM - and Canonical officially supports 64-bit, little-endian
Power - so when I get a spare moment I try to make sure that cool,
officially-supported technologies work on Power &lt;em&gt;before&lt;/em&gt; we end up
with a customer emergency! So, without further ado, this is the story
of XDP on Power.&lt;/p&gt;
&lt;h2&gt;XDP&lt;/h2&gt;
&lt;p&gt;eXpress Data Path (XDP) is a cool Linux technology to allow really
fast processing of network packets.&lt;/p&gt;
&lt;p&gt;Normally in Linux, a packet is received by the network card, an SKB
(&lt;a href="http://vger.kernel.org/~davem/skb.html"&gt;socket buffer&lt;/a&gt;) is
allocated, and the packet is passed up through the networking stack.&lt;/p&gt;
&lt;p&gt;This introduces an inescapable latency penalty: we have to allocate
some memory and copy stuff around. XDP allows some network cards and
drivers to process packets early - even before the allocation of the
SKB. This is much faster, and so has applications in DDOS mitigation
and other high-speed networking use-cases. The IOVisor project has
&lt;a href="https://www.iovisor.org/technology/xdp"&gt;much more information&lt;/a&gt; if you
want to learn more.&lt;/p&gt;
&lt;h2&gt;eBPF&lt;/h2&gt;
&lt;p&gt;XDP processing is done by an eBPF program. eBPF - the extended
Berkeley Packet Filter - is an in-kernel virtual machine with a
limited set of instructions. The kernel can statically validate eBPF
programs to ensure that they terminate and are memory safe. From this
it follows that the programs cannot be Turing-complete: they do not
have backward branches, so they cannot do fancy things like
loops. Nonetheless, they're surprisingly powerful for packet
processing and tracing. eBPF programs are translated into efficient
machine code using in-kernel JIT compilers on many platforms, and
interpreted on platforms that do not have a JIT. (Yes, there are
multiple JIT implementations in the kernel. I find this a terrifying
thought.)&lt;/p&gt;
&lt;p&gt;Rather than requiring people to write raw eBPF programs, you can write
them in a somewhat-restricted subset of C, and use Clang's eBPF target
to translate them. This is super handy, as it gives you access to the
kernel headers - which define a number of useful data structures like
headers for various network protocols.&lt;/p&gt;
&lt;h2&gt;Trying it&lt;/h2&gt;
&lt;p&gt;There are a few really interesting project that are already up and
running that allow you to explore XDP without learning the innards of
both eBPF and the kernel networking stack. I explored the samples in
the &lt;a href="https://github.com/iovisor/bcc"&gt;bcc compiler collection&lt;/a&gt; and also
the samples from the &lt;a href="https://github.com/netoptimizer/prototype-kernel/"&gt;netoptimizer/prototype-kernel repository&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The easiest way to get started with these is with a virtual machine,
as recent virtio network drivers support XDP. If you are using Ubuntu,
you can use the &lt;a href="https://help.ubuntu.com/lts/serverguide/cloud-images-and-uvtool.html"&gt;uvt-kvm
tooling&lt;/a&gt;
to trivially set up a VM running Ubuntu Zesty on your local machine.&lt;/p&gt;
&lt;p&gt;Once your VM is installed, you need to shut it down and edit the virsh XML. &lt;/p&gt;
&lt;p&gt;You need 2 vCPUs (or more) and a virtio+vhost network card. You also
need to edit the 'interface' section and add the following snippet
(with thanks to the &lt;a href="https://www.spinics.net/lists/xdp-newbies/msg00029.html"&gt;xdp-newbies
list&lt;/a&gt;):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;driver&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;name=&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;vhost&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;queues=&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;4&amp;#39;&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;&amp;lt;host&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;tso4=&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;off&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;tso6=&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;off&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;ecn=&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;off&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;ufo=&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;off&amp;#39;&lt;/span&gt;&lt;span class="nt"&gt;/&amp;gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;&amp;lt;guest&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;tso4=&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;off&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;tso6=&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;off&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;ecn=&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;off&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;ufo=&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;off&amp;#39;&lt;/span&gt;&lt;span class="nt"&gt;/&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/driver&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;(If you have more than 2 vCPUs, set the queues parameter to 2x the
number of vCPUs.)&lt;/p&gt;
&lt;p&gt;Then, install a modern clang (we've had issues with 3.8 - I recommend
v4+), and the usual build tools.&lt;/p&gt;
&lt;p&gt;I recommend testing with the prototype-kernel tools - the DDOS
prevention tool is a good demo. Then - on x86 - you just follow their
instructions. I'm not going to repeat that here.&lt;/p&gt;
&lt;h2&gt;POWERful XDP&lt;/h2&gt;
&lt;p&gt;What happens when you try this on Power? Regular readers of my posts
will know to expect some
&lt;a href="https://sthbrx.github.io/blog/2017/02/13/high-power-lustre/"&gt;minor&lt;/a&gt;
&lt;a href="https://sthbrx.github.io/blog/2017/02/01/namd-on-nvlink/"&gt;hitches&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;XDP does not disappoint.&lt;/p&gt;
&lt;p&gt;Firstly, the prototype-kernel repository &lt;a href="https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/Makefile#L92"&gt;hard codes x86&lt;/a&gt;
as the architecture for kernel headers. You need to change it for
powerpc.&lt;/p&gt;
&lt;p&gt;Then, once you get the stuff compiled, and try to run it on a
current-at-time-of-writing Zesty kernel, you'll hit a massive debug
splat ending in:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;61&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;r1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)(&lt;/span&gt;&lt;span class="n"&gt;r8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;misaligned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;packet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;access&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;off&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;
&lt;span class="n"&gt;load_bpf_file&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Permission&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;denied&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It turns out this is because in Ubuntu's Zesty kernel,
CONFIG_HAS_EFFICIENT_UNALIGNED_ACCESS is not set on ppc64el. Because
of that, the eBPF verifier will check that all loads are aligned - and
this load (part of checking some packet header) is not, and so the
verifier rejects the program. Unaligned access is not enabled because
the Zesty kernel is being compiled for CPU_POWER7 instead of
CPU_POWER8, and we don't have efficient unaligned access on POWER7.&lt;/p&gt;
&lt;p&gt;As it turns out, IBM never released any officially supported Power7 LE
systems - LE was only ever supported on Power8. So, I &lt;a href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699627"&gt;filed a bug&lt;/a&gt; and
&lt;a href="https://lists.ubuntu.com/archives/kernel-team/2017-June/085074.html"&gt;sent a patch&lt;/a&gt;
to build Zesty kernels for POWER8 instead, and that has been accepted
and will be part of the next stable update due real soon now.&lt;/p&gt;
&lt;p&gt;Sure enough, if you install a kernel with that config change, you can
verify the XDP program and load it into the kernel!&lt;/p&gt;
&lt;p&gt;If you have real powerpc hardware, that's enough to use XDP on Power!
Thanks to &lt;a href="http://michael.ellerman.id.au/"&gt;Michael Ellerman&lt;/a&gt;,
maintainer extraordinaire, for verifying this for me.&lt;/p&gt;
&lt;p&gt;If - like me - you don't have ready access to Power hardware, you're
stuffed. You can't use qemu in TCG mode: to use XDP with a VM, you
need multi-queue support, which only exists in the vhost driver, which
is only available for KVM guests. Maybe IBM should release a developer
workstation. (Hint, hint!)&lt;/p&gt;
&lt;p&gt;Overall, I was pleasantly surprised by how easy things were for people
with real ppc hardware - it's encouraging to see something not require
kernel changes!&lt;/p&gt;
&lt;p&gt;eBPF and XDP are definitely growing technologies - as &lt;a href="https://twitter.com/brendangregg/status/866078955530444800"&gt;Brendan Gregg notes&lt;/a&gt;,
now is a good time to learn them! (And those on Power have no excuse
either!)&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Mon, 17 Jul 2017 10:08:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-07-17:/blog/2017/07/17/xdp-on-power/</guid><category>Development</category><category>xdp</category><category>power</category><category>networking</category><category>remoteposts</category></item><item><title>Evaluating CephFS on Power</title><link>https://sthbrx.github.io/blog/2017/03/29/evaluating-cephfs-on-power/</link><description>&lt;h2&gt;Methodology&lt;/h2&gt;
&lt;p&gt;To evaluate CephFS, we will create a ppc64le virtual machine, with sufficient
space to compile the software, as well as 3 sparse 1TB disks to create the
object store.&lt;/p&gt;
&lt;p&gt;We will then build &amp;amp; install the Ceph packages, after adding the PowerPC
optimisiations to the code. This is done, as ceph-deploy will fetch prebuilt
packages that do not have the performance patches if the packages are not
installed.&lt;/p&gt;
&lt;p&gt;Finally, we will use the ceph-deploy to deploy the instance. We will ceph-deploy
via pip, to avoid file conflicts with the packages that we built.&lt;/p&gt;
&lt;p&gt;For more information on what each command does, visit the following tutorial,
upon which which this is based:
&lt;a href="http://palmerville.github.io/2016/04/30/single-node-ceph-install.html"&gt;http://palmerville.github.io/2016/04/30/single-node-ceph-install.html&lt;/a&gt;&lt;/p&gt;
&lt;h3&gt;Virtual Machine Config&lt;/h3&gt;
&lt;p&gt;Create a virtual machine with at least the following:
 - 16GB of memory
 - 16 CPUs
 - 64GB disk for the root filesystem
 - 3 x 1TB for the Ceph object store
 - Ubuntu 16.04 default install (only use the 64GB disk, leave the others unpartitioned)&lt;/p&gt;
&lt;h3&gt;Initial config&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Enable ssh&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;    sudo apt install openssh-server
    sudo apt update
    sudo apt upgrade
    sudo reboot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;Install build tools&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;    sudo apt install git debhelper
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3&gt;Build Ceph&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Clone the Ceph repo by following the instructions here: &lt;a href="http://docs.ceph.com/docs/master/install/clone-source/"&gt;http://docs.ceph.com/docs/master/install/clone-source/&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;mkdir&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;$&lt;span class="nv"&gt;HOME&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;src&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;$&lt;span class="nv"&gt;HOME&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;src&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;git&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;clone&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="nv"&gt;recursive&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;https&lt;/span&gt;:&lt;span class="o"&gt;//&lt;/span&gt;&lt;span class="nv"&gt;github&lt;/span&gt;.&lt;span class="nv"&gt;com&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;ceph&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;ceph&lt;/span&gt;.&lt;span class="nv"&gt;git&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;#&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;This&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;may&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;take&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;while&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;ceph&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;git&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;checkout&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;master&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;git&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;submodule&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;update&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="nv"&gt;force&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="nv"&gt;init&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="nv"&gt;recursive&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;Cherry-pick the Power performance patches:&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;    git remote add kestrels https://github.com/kestrels/ceph.git
    git fetch --all
    git cherry-pick 59bed55a676ebbe3ad97d8ec005c2088553e4e11
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;Install prerequisites&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;    ./install-deps.sh
    sudo apt install python-requests python-flask resource-agents curl python-cherrypy python3-pip python-django python-dateutil python-djangorestframework
    sudo pip3 install ceph-deploy
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;Build the packages as per the instructions: &lt;a href="http://docs.ceph.com/docs/master/install/build-ceph/"&gt;http://docs.ceph.com/docs/master/install/build-ceph/&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;HOME&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;src&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;ceph&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;sudo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dpkg&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;buildpackage&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;J&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;nproc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;This&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;will&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;take&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;couple&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;hours&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;cpus&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;Install the packages (note that python3-ceph-argparse will fail, but is safe to ignore)&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;    cd $HOME/src
    sudo dpkg -i *.deb
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3&gt;Create the ceph-deploy user&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;    sudo adduser ceph-deploy
    echo &amp;quot;ceph-deploy ALL = (root) NOPASSWD:ALL&amp;quot; | sudo tee /etc/sudoers.d/ceph-deploy
    sudo chmod 0440 /etc/sudoers.d/ceph-deploy
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3&gt;Configure the ceph-deploy user environment&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;su&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ceph&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;deploy&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;ssh&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;keygen&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n n-Quoted"&gt;`hostname`&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;ssh&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ceph&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;deploy&lt;/span&gt;&lt;span class="nv"&gt;@$node&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;mkdir&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;$HOME&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;ceph&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;cluster&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;$HOME&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;ceph&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;cluster&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;ceph&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;deploy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;$node&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# If this fails, remove the bogus 127.0.1.1 entry from /etc/hosts&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;osd pool default size = 2&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ceph&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;conf&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;osd crush chooseleaf type = 0&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ceph&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;conf&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3&gt;Complete the Ceph deployment&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="w"&gt;    &lt;/span&gt;ceph-deploy&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$node&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;ceph-deploy&lt;span class="w"&gt; &lt;/span&gt;mon&lt;span class="w"&gt; &lt;/span&gt;create-initial
&lt;span class="w"&gt;    &lt;/span&gt;drives=&amp;quot;vda&lt;span class="w"&gt; &lt;/span&gt;vdb&lt;span class="w"&gt; &lt;/span&gt;vdc&amp;quot;&lt;span class="w"&gt;  &lt;/span&gt;#&lt;span class="w"&gt; &lt;/span&gt;the&lt;span class="w"&gt; &lt;/span&gt;1TB&lt;span class="w"&gt; &lt;/span&gt;drives&lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;check&lt;span class="w"&gt; &lt;/span&gt;that&lt;span class="w"&gt; &lt;/span&gt;these&lt;span class="w"&gt; &lt;/span&gt;are&lt;span class="w"&gt; &lt;/span&gt;correct&lt;span class="w"&gt; &lt;/span&gt;for&lt;span class="w"&gt; &lt;/span&gt;your&lt;span class="w"&gt; &lt;/span&gt;system
&lt;span class="w"&gt;    &lt;/span&gt;for&lt;span class="w"&gt; &lt;/span&gt;drive&lt;span class="w"&gt; &lt;/span&gt;in&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$drives&lt;/span&gt;;&lt;span class="w"&gt; &lt;/span&gt;do&lt;span class="w"&gt; &lt;/span&gt;ceph-deploy&lt;span class="w"&gt; &lt;/span&gt;disk&lt;span class="w"&gt; &lt;/span&gt;zap&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$node&lt;/span&gt;:&lt;span class="nv"&gt;$drive&lt;/span&gt;;&lt;span class="w"&gt; &lt;/span&gt;ceph-deploy&lt;span class="w"&gt; &lt;/span&gt;osd&lt;span class="w"&gt; &lt;/span&gt;prepare&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$node&lt;/span&gt;:&lt;span class="nv"&gt;$drive&lt;/span&gt;;&lt;span class="w"&gt; &lt;/span&gt;done
&lt;span class="w"&gt;    &lt;/span&gt;for&lt;span class="w"&gt; &lt;/span&gt;drive&lt;span class="w"&gt; &lt;/span&gt;in&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$drives&lt;/span&gt;;&lt;span class="w"&gt; &lt;/span&gt;do&lt;span class="w"&gt; &lt;/span&gt;ceph-deploy&lt;span class="w"&gt; &lt;/span&gt;osd&lt;span class="w"&gt; &lt;/span&gt;activate&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$node&lt;/span&gt;:/dev/&lt;span class="cp"&gt;${&lt;/span&gt;&lt;span class="n"&gt;drive&lt;/span&gt;&lt;span class="cp"&gt;}&lt;/span&gt;1;&lt;span class="w"&gt; &lt;/span&gt;done
&lt;span class="w"&gt;    &lt;/span&gt;ceph-deploy&lt;span class="w"&gt; &lt;/span&gt;admin&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$node&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;chmod&lt;span class="w"&gt; &lt;/span&gt;+r&lt;span class="w"&gt; &lt;/span&gt;/etc/ceph/ceph.client.admin.keyring
&lt;span class="w"&gt;    &lt;/span&gt;ceph&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;#&lt;span class="w"&gt; &lt;/span&gt;Check&lt;span class="w"&gt; &lt;/span&gt;the&lt;span class="w"&gt; &lt;/span&gt;state&lt;span class="w"&gt; &lt;/span&gt;of&lt;span class="w"&gt; &lt;/span&gt;the&lt;span class="w"&gt; &lt;/span&gt;cluster
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3&gt;Configure CephFS&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;    ceph-deploy mds create $node
    ceph osd pool create cephfs_data 128
    ceph osd pool create cephfs_metadata 128
    ceph fs new cephfs cephfs_metadata cephfs_data
    sudo systemctl status ceph\*.service ceph\*.target # Ensure the ceph-osd, ceph-mon &amp;amp; ceph-mds daemons are running
    sudo mkdir /mnt/cephfs
    key=`grep key ~/ceph-cluster/ceph.client.admin.keyring | cut -d &amp;#39; &amp;#39; -f 3`
    sudo mount -t ceph $node:6789:/ /mnt/cephfs -o name=admin,secret=$key
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2&gt;References&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="http://docs.ceph.com/docs/master/install/clone-source/"&gt;http://docs.ceph.com/docs/master/install/clone-source/&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://docs.ceph.com/docs/master/install/build-ceph/"&gt;http://docs.ceph.com/docs/master/install/build-ceph/&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://palmerville.github.io/2016/04/30/single-node-ceph-install.html"&gt;http://palmerville.github.io/2016/04/30/single-node-ceph-install.html&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Alastair D'Silva</dc:creator><pubDate>Wed, 29 Mar 2017 00:00:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-03-29:/blog/2017/03/29/evaluating-cephfs-on-power/</guid><category>Development</category><category>ceph</category><category>raid</category><category>storage</category></item><item><title>Erasure Coding for Programmers, Part 2</title><link>https://sthbrx.github.io/blog/2017/03/24/erasure-coding-for-programmers-part-2/</link><description>&lt;p&gt;We left &lt;a href="/blog/2017/03/20/erasure-coding-for-programmers-part-1/"&gt;part 1&lt;/a&gt; having explored GF(2^8) and RAID 6, and asking the question "what does all this have to do with Erasure Codes?"&lt;/p&gt;
&lt;p&gt;Basically, the thinking goes "RAID 6 is cool, but what if, instead of two parity disks, we had an arbitrary number of parity disks?"&lt;/p&gt;
&lt;p&gt;How would we do that? Well, let's introduce our new best friend: Coding Theory!&lt;/p&gt;
&lt;p&gt;Say we want to transmit some data across an error-prone medium. We don't know where the errors might occur, so we add some extra information to allow us to detect and possibly correct for errors. This is a code. Codes are a largish field of engineering, but rather than show off my knowledge about systematic linear block codes, let's press on.&lt;/p&gt;
&lt;p&gt;Today, our error-prone medium is an array of inexpensive disks. Now we make this really nice assumption about disks, namely that they are either perfectly reliable or completely missing. In other words, we consider that a disk will either be present or 'erased'. We come up with 'erasure codes' that are able to reconstruct data when it is known to be missing. (This is a slightly different problem to being able to verify and correct data that might or might not be subtly corrupted. Disks also have to deal with this problem, but it is &lt;em&gt;not&lt;/em&gt; something erasure codes address!)&lt;/p&gt;
&lt;p&gt;The particular code we use is a Reed-Solomon code. The specific details are unimportant, but there's a really good graphical outline of the broad concepts in sections 1 and 3 of &lt;a href="http://jerasure.org/jerasure-2.0/"&gt;the Jerasure paper/manual&lt;/a&gt;. (Don't go on to section 4.)&lt;/p&gt;
&lt;p&gt;That should give you some background on how this works at a pretty basic mathematical level. Implementation is a matter of mapping that maths (matrix multiplication) onto hardware primitives, and making it go fast.&lt;/p&gt;
&lt;h2&gt;Scope&lt;/h2&gt;
&lt;p&gt;I'm deliberately &lt;em&gt;not&lt;/em&gt; covering some pretty vast areas of what would be required to write your own erasure coding library from scratch. I'm not going to talk about how to compose the matricies, how to invert them, or anything like that. I'm not sure how that would be a helpful exercise - ISA-L and jerasure already exist and do that for you.&lt;/p&gt;
&lt;p&gt;What I want to cover is an efficient implementation of the some algorithms, once you have the matricies nailed down.&lt;/p&gt;
&lt;p&gt;I'm also going to assume your library already provides a generic multiplication function in GF(2^8). That's required to construct the matrices, so it's a pretty safe assumption.&lt;/p&gt;
&lt;h2&gt;The beginnings of an API&lt;/h2&gt;
&lt;p&gt;Let's make this a bit more concrete.&lt;/p&gt;
&lt;p&gt;This will be heavily based on the &lt;a href="https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version/documentation/isa-l-open-source-api"&gt;ISA-L API&lt;/a&gt; but you probably want to plug into ISA-L anyway, so that shouldn't be a problem.&lt;/p&gt;
&lt;p&gt;What I want to do is build up from very basic algorithmic components into something useful.&lt;/p&gt;
&lt;p&gt;The first thing we want to do is to be able to is Galois Field multiplication of an entire region of bytes by an arbitrary constant.&lt;/p&gt;
&lt;p&gt;We basically want &lt;code&gt;gf_vect_mul(size_t len, &amp;lt;something representing the constant&amp;gt;, unsigned char * src, unsigned char * dest)&lt;/code&gt;&lt;/p&gt;
&lt;h3&gt;Simple and slow approach&lt;/h3&gt;
&lt;p&gt;The simplest way is to do something like this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;gf_vect_mul_simple&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;size_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;size_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;gf_mul&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="err"&gt;}&lt;/span&gt;
&lt;span class="err"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That does multiplication element by element using the library's supplied &lt;code&gt;gf_mul&lt;/code&gt; function, which - as the name suggests - does GF(2^8) multiplication of a scalar by a scalar.&lt;/p&gt;
&lt;p&gt;This works. The problem is that it is very, painfully, slow - in the order of a few hundred megabytes per second.&lt;/p&gt;
&lt;h3&gt;Going faster&lt;/h3&gt;
&lt;p&gt;How can we make this faster?&lt;/p&gt;
&lt;p&gt;There are a few things we can try: if you want to explore a whole range of different ways to do this, check out the &lt;a href="http://jerasure.org/gf-complete-1.02/"&gt;gf-complete&lt;/a&gt; project. I'm going to assume we want to skip right to the end and know what is the fastest we've found.&lt;/p&gt;
&lt;p&gt;Cast your mind back to the &lt;a href="https://www.kernel.org/pub/linux/kernel/people/hpa/raid6.pdf"&gt;RAID 6 paper&lt;/a&gt; (PDF). I talked about in &lt;a href="/blog/2017/03/20/erasure-coding-for-programmers-part-1/"&gt;part 1&lt;/a&gt;. That had a way of doing an efficient multiplication in GF(2^8) using vector instructions.&lt;/p&gt;
&lt;p&gt;To refresh your memory, we split the multiplication into two parts - low bits and high bits, looked them up separately in a lookup table, and joined them with XOR. We then discovered that on modern Power chips, we could do that in one instruction with &lt;code&gt;vpermxor&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;So, a very simple way to do this would be:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;generate the table for &lt;code&gt;a&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;for each 16-byte chunk of our input:&lt;ul&gt;
&lt;li&gt;load the input&lt;/li&gt;
&lt;li&gt;do the &lt;code&gt;vpermxor&lt;/code&gt; with the table&lt;/li&gt;
&lt;li&gt;save it out&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Generating the tables is reasonably straight-forward, in theory. Recall that the tables are &lt;code&gt;a&lt;/code&gt; * {{00},{01},...,{0f}} and &lt;code&gt;a&lt;/code&gt; * {{00},{10},..,{f0}} - a couple of loops in C will generate them without difficulty. ISA-L has a function to do this, as does gf-complete in split-table mode, so I won't repeat them here.&lt;/p&gt;
&lt;p&gt;So, let's recast our function to take the tables as an input rather than the constant &lt;code&gt;a&lt;/code&gt;. Assume we're provided the two tables concatenated into one 32-byte chunk. That would give us:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;void gf_vect_mul_v2(size_t len, unsigned char * table, unsigned char * src, unsigned char * dest)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Here's how you would do it in C:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;gf_vect_mul_v2&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;size_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;len&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;table&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;src&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;dest&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;{
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;vector&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;tbl1&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;tbl2&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;in&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;out&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;size_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="cm"&gt;/* Assume table, src, dest are aligned and len is a multiple of 16 */&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;tbl1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vec_ld&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;table&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;tbl2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vec_ld&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;table&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="c1"&gt;; i&amp;lt;len; i+=16) {&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vec_ld&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="nv"&gt;src&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;__asm__&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vpermxor %0, %1, %2, %3&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;=v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;out&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;tbl1&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;tbl2&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;in&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;vec_st&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;out&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="nv"&gt;dest&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;}
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There's a few quirks to iron out - making sure the table is laid out in the vector register in the way you expect, etc, but that generally works and is quite fast - my Power 8 VM does about 17-18 GB/s with non-cache-contained data with this implementation.&lt;/p&gt;
&lt;p&gt;We can go a bit faster by doing larger chunks at a time:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="c1"&gt;; i&amp;lt;vlen; i+=64) {&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;in1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vec_ld&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="nv"&gt;src&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;in2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vec_ld&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="nv"&gt;src&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;in3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vec_ld&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;32&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="nv"&gt;src&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;in4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vec_ld&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;48&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="nv"&gt;src&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;__asm__&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vpermxor %0, %1, %2, %3&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;=v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;out1&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;tbl1&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;tbl2&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;in1&lt;/span&gt;&lt;span class="ss"&gt;))&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;__asm__&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vpermxor %0, %1, %2, %3&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;=v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;out2&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;tbl1&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;tbl2&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;in2&lt;/span&gt;&lt;span class="ss"&gt;))&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;__asm__&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vpermxor %0, %1, %2, %3&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;=v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;out3&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;tbl1&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;tbl2&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;in3&lt;/span&gt;&lt;span class="ss"&gt;))&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;__asm__&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vpermxor %0, %1, %2, %3&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;=v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;out4&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;tbl1&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;tbl2&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;v&amp;quot;&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;in4&lt;/span&gt;&lt;span class="ss"&gt;))&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;vec_st&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;out1&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="nv"&gt;dest&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;vec_st&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;out2&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="nv"&gt;dest&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;vec_st&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;out3&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;32&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="nv"&gt;dest&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;vec_st&lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;out4&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;48&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="nv"&gt;dest&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This goes at about 23.5 GB/s.&lt;/p&gt;
&lt;p&gt;We can go one step further and do the core loop in assembler - that means we control the instruction layout and so on. I tried this: it turns out that for the basic vector multiply loop, if we turn off ASLR and pin to a particular CPU, we can see a improvement of a few percent (and a decrease in variability) over C code.&lt;/p&gt;
&lt;h2&gt;Building from vector multiplication&lt;/h2&gt;
&lt;p&gt;Once you're comfortable with the core vector multiplication, you can start to build more interesting routines.&lt;/p&gt;
&lt;p&gt;A particularly useful one on Power turned out to be the multiply and add routine: like gf_vect_mul, except that rather than overwriting the output, it loads the output and xors the product in. This is a simple extension of the gf_vect_mul function so is left as an exercise to the reader.&lt;/p&gt;
&lt;p&gt;The next step would be to start building erasure coding proper. Recall that to get an element of our output, we take a dot product: we take the corresponding input element of each disk, multiply it with the corresponding GF(2^8) coding matrix element and sum all those products. So all we need now is a dot product algorithm.&lt;/p&gt;
&lt;p&gt;One approach is the conventional dot product:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;for each element&lt;ul&gt;
&lt;li&gt;zero accumulator&lt;/li&gt;
&lt;li&gt;for each source&lt;ul&gt;
&lt;li&gt;load &lt;code&gt;input[source][element]&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;do GF(2^8) multiplication&lt;/li&gt;
&lt;li&gt;xor into accumulator&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;save accumulator to &lt;code&gt;output[element]&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The other approach is multiply and add:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;for each source&lt;ul&gt;
&lt;li&gt;for each element&lt;ul&gt;
&lt;li&gt;load &lt;code&gt;input[source][element]&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;do GF(2^8) multiplication&lt;/li&gt;
&lt;li&gt;load &lt;code&gt;output[element]&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;xor in product&lt;/li&gt;
&lt;li&gt;save &lt;code&gt;output[element]&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The dot product approach has the advantage of fewer writes. The multiply and add approach has the advantage of better cache/prefetch performance. The approach you ultimately go with will probably depend on the characteristics of your machine and the length of data you are dealing with.&lt;/p&gt;
&lt;p&gt;For what it's worth, ISA-L ships with only the first approach in x86 assembler, and Jerasure leans heavily towards the second approach.&lt;/p&gt;
&lt;p&gt;Once you have a vector dot product sorted, you can build a full erasure coding setup: build your tables with your library, then do a dot product to generate each of your outputs!&lt;/p&gt;
&lt;p&gt;In ISA-L, this is implemented something like this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cm"&gt;/*&lt;/span&gt;
&lt;span class="cm"&gt; * ec_encode_data_simple(length of each data input, number of inputs,&lt;/span&gt;
&lt;span class="cm"&gt; *                       number of outputs, pre-generated GF(2^8) tables,&lt;/span&gt;
&lt;span class="cm"&gt; *                       input data pointers, output code pointers)&lt;/span&gt;
&lt;span class="cm"&gt; */&lt;/span&gt;
&lt;span class="nx"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ec_encode_data_simple&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;len&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;g_tbls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;                           &lt;/span&gt;&lt;span class="nx"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="nx"&gt;coding&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;while&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nx"&gt;gf_vect_dot_prod&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;len&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;g_tbls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nx"&gt;coding&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nx"&gt;g_tbls&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;k&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nx"&gt;coding&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nx"&gt;rows&lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2&gt;Going faster still&lt;/h2&gt;
&lt;p&gt;Eagle eyed readers will notice that however we generate an output, we have to read all the input elements. This means that if we're doing a code with 10 data disks and 4 coding disks, we have to read each of the 10 inputs 4 times.&lt;/p&gt;
&lt;p&gt;We could do better if we could calculate multiple outputs for each pass through the inputs. This is a little fiddly to implement, but does lead to a speed improvement.&lt;/p&gt;
&lt;p&gt;ISA-L is an excellent example here. Intel goes up to 6 outputs at once: the number of outputs you can do is only limited by how many vector registers you have to put the various operands and results in.&lt;/p&gt;
&lt;h2&gt;Tips and tricks&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;Benchmarking is tricky. I do the following on a bare-metal, idle machine, with ASLR off and pinned to an arbitrary hardware thread. (Code is for the &lt;a href="https://fishshell.com/"&gt;fish shell&lt;/a&gt;)&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="n"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seq&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;setarch&lt;/span&gt; &lt;span class="n"&gt;ppc64le&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;R&lt;/span&gt; &lt;span class="n"&gt;taskset&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="mi"&gt;24&lt;/span&gt; &lt;span class="n"&gt;erasure_code&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;gf_vect_mul_perf&lt;/span&gt;
&lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt; &lt;span class="n"&gt;awk&lt;/span&gt; &lt;span class="s"&gt;&amp;#39;/MB/ {sum+=$13} END {print sum/50, &amp;quot;MB/s&amp;quot;}&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Debugging is tricky; the more you can do in C and the less you do in assembly, the easier your life will be.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Vector code is notoriously alignment-sensitive - if you can't figure out why something is wrong, check alignment. (Pro-tip: ISA-L does &lt;em&gt;not&lt;/em&gt; guarantee the alignment of the &lt;code&gt;gftbls&lt;/code&gt; parameter, and many of the tests supply an unaligned table from the stack. For testing &lt;code&gt;__attribute__((aligned(16)))&lt;/code&gt; is your friend!)&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Related: GCC is moving towards assignment over vector intrinsics, at least on Power:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;vector unsigned char a;
unsigned char * data;
// good, also handles word-aligned data with VSX
a = *(vector unsigned char *)data;
// bad, requires special handling of non-16-byte aligned data
a = vec_ld(0, (unsigned char *) data);
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Hopefully by this point you're equipped to figure out how your erasure coding library of choice works, and write your own optimised implementation (or maintain an implementation written by someone else).&lt;/p&gt;
&lt;p&gt;I've referred to a number of resources throughout this series:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;ISA-L &lt;a href="https://github.com/01org/isa-l"&gt;code&lt;/a&gt;, &lt;a href=""&gt;API description&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Jerasure &lt;a href="http://jerasure.org/"&gt;code&lt;/a&gt;, &lt;a href="http://jerasure.org/jerasure-2.0/"&gt;docs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;gf-complete &lt;a href="http://jerasure.org/"&gt;code&lt;/a&gt;, &lt;a href="http://jerasure.org/gf-complete-1.02/"&gt;docs&lt;/a&gt; &lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.kernel.org/pub/linux/kernel/people/hpa/raid6.pdf"&gt;The mathematics of RAID-6&lt;/a&gt; (PDF), H. Peter Anvin&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If you want to go deeper, I also read the following and found them quite helpful in understanding Galois Fields and Reed-Solomon coding:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19900019023.pdf"&gt;Tutorial on Reed-Solomon Error Correction Coding&lt;/a&gt; (PDF), William A. Geisel, NASA&lt;/li&gt;
&lt;li&gt;&lt;a href="https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19900019023.pdf"&gt;Reed-Solomon error correction&lt;/a&gt; (PDF), BBC R&amp;amp;D White Paper WHP 031, C. K. P. Clarke.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For a more rigorous mathematical approach to rings and fields, a university mathematics course may be of interest. For more on coding theory, a university course in electronics engineering may be helpful.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Fri, 24 Mar 2017 10:08:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-03-24:/blog/2017/03/24/erasure-coding-for-programmers-part-2/</guid><category>Development</category><category>erasure</category><category>raid</category><category>storage</category></item><item><title>Erasure Coding for Programmers, Part 1</title><link>https://sthbrx.github.io/blog/2017/03/20/erasure-coding-for-programmers-part-1/</link><description>&lt;p&gt;Erasure coding is an increasingly popular storage technology - allowing the same level of fault tolerance as replication with a significantly reduced storage footprint.&lt;/p&gt;
&lt;p&gt;Increasingly, erasure coding is available 'out of the box' on storage solutions such as Ceph and OpenStack Swift. Normally, you'd just pull in a library like &lt;a href="https://github.com/01org/isa-l"&gt;ISA-L&lt;/a&gt; or &lt;a href="http://jerasure.org"&gt;jerasure&lt;/a&gt;, and set some config options, and you'd be done.&lt;/p&gt;
&lt;p&gt;This post is not about that. This post is about how I went from knowing nothing about erasure coding to writing POWER optimised routines to make it go fast. (These are in the process of being polished for upstream at the moment.) If you want to understand how erasure coding works under the hood - and in particular if you're interested in writing optimised routines to make it run quickly in your platform - this is for you.&lt;/p&gt;
&lt;h2&gt;What are erasure codes anyway?&lt;/h2&gt;
&lt;p&gt;I think the easiest way to begin thinking about erasure codes is "RAID 6 on steroids". RAID 6 allows you to have up to 255 data disks and 2 parity disks (called P and Q), thus allowing you to tolerate the failure of up to 2 arbitrary disks without data loss.&lt;/p&gt;
&lt;p&gt;Erasure codes allow you to have k data disks and m 'parity' or coding disks. You then have a total of m + k disks, and you can tolerate the failure of up to m without losing data.&lt;/p&gt;
&lt;p&gt;The downside of erasure coding is that computing what to put on those parity disks is CPU intensive. Lets look at what we put on them.&lt;/p&gt;
&lt;h2&gt;RAID 6&lt;/h2&gt;
&lt;p&gt;RAID 6 is the easiest way to get started on understanding erasure codes for a number of reasons. H Peter Anvin's paper on RAID 6 in the Linux kernel is an excellent start, but does dive in a bit quickly to the underlying mathematics. So before reading that, read on!&lt;/p&gt;
&lt;h2&gt;Rings and Fields&lt;/h2&gt;
&lt;p&gt;As programmers we're pretty comfortable with modular arithmetic - the idea that if you have:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;unsigned char a = 255;
a++;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;the new value of &lt;code&gt;a&lt;/code&gt; will be 0, not 256.&lt;/p&gt;
&lt;p&gt;This is an example of an algebraic structure called a &lt;em&gt;ring&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Rings obey certain laws. For our purposes, we'll consider the following incomplete and somewhat simplified list:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;There is an addition operation.&lt;/li&gt;
&lt;li&gt;There is an additive identity (normally called 0), such that 'a + 0 = a'.&lt;/li&gt;
&lt;li&gt;Every element has an additive inverse, that is, for every element 'a', there is an element -a such that 'a + (-a) = 0'&lt;/li&gt;
&lt;li&gt;There is a multiplication operation.&lt;/li&gt;
&lt;li&gt;There is a multiplicative identity (normally called 1), such that 'a * 1 = a'.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;These operations aren't necessarily addition or multiplication as we might expect from the integers or real numbers. For example, in our modular arithmetic example, we have 'wrap around'. (There are also certain rules the addition and multiplication rules must satisfy - we are glossing over them here.)&lt;/p&gt;
&lt;p&gt;One thing a ring doesn't have a 'multiplicative inverse'. The multiplicative inverse of some non-zero element of the ring (call it a), is the value b such that a * b = 1. (Often instead of b we write 'a^-1', but that looks bad in plain text, so we shall stick to b for now.)&lt;/p&gt;
&lt;p&gt;We do have some inverses in 'mod 256': the inverse of 3 is 171 as 3 * 171 = 513, and 513 = 1 mod 256, but there is no b such that 2 * b = 1 mod 256.&lt;/p&gt;
&lt;p&gt;If every non-zero element of our ring had a multiplicative inverse, we would have what is called a &lt;em&gt;field&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Now, let's look at a the integers modulo 2, that is, 0 and 1.&lt;/p&gt;
&lt;p&gt;We have this for addition:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;+&lt;/th&gt;
&lt;th&gt;0&lt;/th&gt;
&lt;th&gt;1&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;0&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;1&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Eagle-eyed readers will notice that this is the same as XOR.&lt;/p&gt;
&lt;p&gt;For multiplication: &lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;*&lt;/th&gt;
&lt;th&gt;0&lt;/th&gt;
&lt;th&gt;1&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;0&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;1&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;As we said, a field is a ring where every non-zero element has a multiplicative inverse. As we can see, the integers modulo 2 shown above is a field: it's a ring, and 1 is its own multiplicative inverse.&lt;/p&gt;
&lt;p&gt;So this is all well and good, but you can't really do very much in a field with 2 elements. This is sad, so we make bigger fields. For this application, we consider the Galois Field with 256 elements - GF(2^8). This field has some surprising and useful properties.&lt;/p&gt;
&lt;p&gt;Remember how we said that integers modulo 256 weren't a field because they didn't have multiplicative inverses? I also just said that GF(2^8) also has 256 elements, but is a field - i.e., it does have inverses! How does that work?&lt;/p&gt;
&lt;p&gt;Consider an element in GF(2^8). There are 2 ways to look at an element in GF(2^8). The first is to consider it as an 8-bit number. So, for example, let's take 100. We can express that as as an 8 bit binary number: 0b01100100.&lt;/p&gt;
&lt;p&gt;We can write that more explicitly as a sum of powers of 2:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;7&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;
&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now the other way we can look at elements in GF(2^8) is to replace the '2's with 'x's, and consider them as polynomials. Each of our bits then represents the coefficient of a term of a polynomial, that is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;7&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;or more simply&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;x^6 + x^5 + x^2
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now, and this is &lt;strong&gt;important&lt;/strong&gt;: each of the coefficients are elements of the integers modulo 2: x + x = 2x = 0 as 2 mod 2 = 0. There is no concept of 'carrying' in this addition.&lt;/p&gt;
&lt;p&gt;Let's try: what's 100 + 79 in GF(2^8)?&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;b01100100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;79&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;b01001111&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;

&lt;span class="mf"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;79&lt;/span&gt;&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;
&lt;span class="w"&gt;                 &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;b00101011&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;43&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;So, 100 + 79 = 43 in GF(2^8)&lt;/p&gt;
&lt;p&gt;You may notice we could have done that much more efficiently: we can add numbers in GF(2^8) by just XORing their binary representations together. Subtraction, amusingly, is the same as addition: 0 + x = x =  0 - x, as -1 is congruent to 1 modulo 2.&lt;/p&gt;
&lt;p&gt;So at this point you might be wanting to explore a few additions yourself. Fortuantely there's a lovely tool that will allow you to do that:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;sudo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;apt&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;install&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;gf&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;complete&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;tools&lt;/span&gt;
&lt;span class="n"&gt;gf_add&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;A&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;B&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This will give you A + B in GF(2^8).&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&amp;gt; gf_add 100 79 8
43
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Excellent!&lt;/p&gt;
&lt;p&gt;So, hold on to your hats, as this is where things get really weird. In modular arithmetic example, we considered the elements of our ring to be numbers, and we performed our addition and multiplication modulo 256. In GF(2^8), we consider our elements as polynomials and we perform our addition and multiplication modulo a polynomial. There is one conventional polynomial used in applications:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x11d&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;b1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0001&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1101&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It is possible to use other polynomials if they satisfy particular requirements, but for our applications we don't need to worry as we will always use 0x11d. I am not going to attempt to explain anything about this polynomial - take it as an article of faith.&lt;/p&gt;
&lt;p&gt;So when we multiply two numbers, we multiply their polynomial representations. Then, to find out what that is modulo 0x11d, we do polynomial long division by 0x11d, and take the remainder.&lt;/p&gt;
&lt;p&gt;Some examples will help.&lt;/p&gt;
&lt;p&gt;Let's multiply 100 by 3.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;b01100100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="mf"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;b00000011&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;

&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;7&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;                         &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;7&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Notice that some of the terms have disappeared: x^6 + x^6 = 0.&lt;/p&gt;
&lt;p&gt;The degree (the largest power of a term) is 7. 7 is less than the degree of 0x11d, which is 8, so we don't need to do anything: the remainder modulo 0x11d is simply x^7 + x^5 + x^3 + x^2.&lt;/p&gt;
&lt;p&gt;In binary form, that is 0b10101100 = 172, so 100 * 3 = 172 in GF(2^8).&lt;/p&gt;
&lt;p&gt;Fortunately &lt;code&gt;gf-complete-tools&lt;/code&gt; also allows us to check multiplications:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&amp;gt; gf_mult 100 3 8
172
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Excellent!&lt;/p&gt;
&lt;p&gt;Now let's see what happens if we multiply by a larger number. Let's multiply 100 by 5.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;b01100100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;b00000101&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;

&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;7&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;                           &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;7&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Here we have an x^8 term, so we have a degree of 8. This means will get a different remainder when we divide by our polynomial. We do this with polynomial long division, which you will hopefully remember if you did some solid algebra in high school.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c"&gt;                              1&lt;/span&gt;
&lt;span class="c"&gt;                           &lt;/span&gt;&lt;span class="nb"&gt;---------------------------------------------&lt;/span&gt;
&lt;span class="c"&gt;x^8 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^4 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^3 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^2 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; 1 | x^8 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^7 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^6 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^5 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^4       &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^2&lt;/span&gt;
&lt;span class="c"&gt;                          &lt;/span&gt;&lt;span class="nb"&gt;-&lt;/span&gt;&lt;span class="c"&gt; x^8                   &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^4 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^3 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^2 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; 1&lt;/span&gt;
&lt;span class="c"&gt;                            &lt;/span&gt;&lt;span class="nb"&gt;-------------------------------------------&lt;/span&gt;
&lt;span class="c"&gt;                          =       x^7 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^6 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^5       &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^3       &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; 1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;So we have that our original polynomial (x^8 + x^4 + x^3 + x^2 + 1) is congruent to (x^7 + x^6 + x^5 + x^3 + 1) modulo the polynomial 0x11d.
Looking at the binary representation of that new polynomial, we have 0b11101001 = 233.&lt;/p&gt;
&lt;p&gt;Sure enough:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&amp;gt; gf_mult 100 5 8
233
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Just to solidify the polynomial long division a bit, let's try a slightly larger example, 100 * 9:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;b01100100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="mf"&gt;9&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;b00001001&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;

&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;9&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;                           &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;9&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Doing long division to reduce our result:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c"&gt;                              x&lt;/span&gt;
&lt;span class="c"&gt;                           &lt;/span&gt;&lt;span class="nb"&gt;-----------------------------------&lt;/span&gt;
&lt;span class="c"&gt;x^8 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^4 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^3 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^2 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; 1 | x^9 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^8       &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^6                   &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^2&lt;/span&gt;
&lt;span class="c"&gt;                          &lt;/span&gt;&lt;span class="nb"&gt;-&lt;/span&gt;&lt;span class="c"&gt; x^9                   &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^5 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^4 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^3       &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x&lt;/span&gt;
&lt;span class="c"&gt;                            &lt;/span&gt;&lt;span class="nb"&gt;-------------------------------------------------&lt;/span&gt;
&lt;span class="c"&gt;                          =       x^8       &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^6 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^5 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^4 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^3 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x^2 &lt;/span&gt;&lt;span class="nb"&gt;+&lt;/span&gt;&lt;span class="c"&gt; x&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;We still have a polynomial of degree 8, so we can do another step:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                              x +   1
                           -----------------------------------
x^8 + x^4 + x^3 + x^2 + 1 | x^9 + x^8       + x^6                   + x^2
                          - x^9                   + x^5 + x^4 + x^3       + x
                            -------------------------------------------------
                          =       x^8       + x^6 + x^5 + x^4 + x^3 + x^2 + x
                          -       x^8                   + x^4 + x^3 + x^2     + 1
                                  -----------------------------------------------
                          =                   x^6 + x^5                   + x + 1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;We now have a polynomial of degree less than 8 that is congruent to our original polynomial modulo 0x11d, and the binary form is 0x01100011 = 99.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&amp;gt; gf_mult 100 9 8
99
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This process can be done more efficiently, of course - but understanding what is going on will make you &lt;em&gt;much&lt;/em&gt; more comfortable with what is going on!&lt;/p&gt;
&lt;p&gt;I will not try to convince you that all multiplicative inverses exist in this magic shadow land of GF(2^8), but it's important for the rest of the algorithms to work that they do exist. Trust me on this.&lt;/p&gt;
&lt;h2&gt;Back to RAID 6&lt;/h2&gt;
&lt;p&gt;Equipped with this knowledge, you are ready to take on &lt;a href="https://www.kernel.org/pub/linux/kernel/people/hpa/raid6.pdf"&gt;RAID6 in the kernel&lt;/a&gt; (PDF) sections 1 - 2.&lt;/p&gt;
&lt;p&gt;Pause when you get to section 3 - this snippet is a bit magic and benefits from some explanation:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Multiplication by {02} for a single byte can be implemeted using the C code:&lt;/p&gt;
&lt;/blockquote&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;uint8_t c, cc;
cc = (c &amp;lt;&amp;lt; 1) ^ ((c &amp;amp; 0x80) ? 0x1d : 0);
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;How does this work? Well:&lt;/p&gt;
&lt;p&gt;Say you have a binary number 0bNMMM MMMM. Mutiplication by 2 gives you 0bNMMMMMMM0, which is 9 bits. Now, there are two cases to consider.&lt;/p&gt;
&lt;p&gt;If your leading bit (N) is 0, your product doesn't have an x^8 term, so we don't need to reduce it modulo the irreducible polynomial.&lt;/p&gt;
&lt;p&gt;If your leading bit is 1 however, your product is x^8 + something, which does need to be reduced. Fortunately, because we took an 8 bit number and multiplied it by 2, the largest term is x^8, so we only need to reduce it once. So we xor our number with our polynomial to subtract it.&lt;/p&gt;
&lt;p&gt;We implement this by letting the top bit overflow out and then xoring the lower 8 bits with the low 8 bits of the polynomial (0x1d)&lt;/p&gt;
&lt;p&gt;So, back to the original statement:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ss"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;c&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;((&lt;/span&gt;&lt;span class="nv"&gt;c&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="nv"&gt;x80&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;?&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="nv"&gt;x1d&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="ss"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;multiply&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;by&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;high&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;bit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;will&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;product&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;have&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;an&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;x&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;term&lt;/span&gt;?
&lt;span class="w"&gt;                          &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;                          &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;so&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;reduce&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;by&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;polynomial&lt;/span&gt;
&lt;span class="w"&gt;                                &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;                                &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;otherwise&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;leave&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;alone&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Hopefully that makes sense.&lt;/p&gt;
&lt;h3&gt;Key points&lt;/h3&gt;
&lt;p&gt;It's critical you understand the section on Altivec (the vperm stuff), so let's cover it in a bit more detail.&lt;/p&gt;
&lt;p&gt;Say you want to do A * V, where A is a constant and V is an 8-bit variable. We can express V as V_a + V_b, where V_a is the top 4 bits of V, and V_b is the bottom 4 bits. A * V = A * V_a + A * V_b&lt;/p&gt;
&lt;p&gt;We can then make lookup tables for multiplication by A.&lt;/p&gt;
&lt;p&gt;If we did this in the most obvious way, we would need a 256 entry lookup table. But by splitting things into the top and bottom halves, we can reduce that to two 16 entry tables. For example, say A = 02.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;V_a&lt;/th&gt;
&lt;th&gt;A * V_a&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;00&lt;/td&gt;
&lt;td&gt;00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;01&lt;/td&gt;
&lt;td&gt;02&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02&lt;/td&gt;
&lt;td&gt;04&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;...&lt;/td&gt;
&lt;td&gt;...&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0f&lt;/td&gt;
&lt;td&gt;1e&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;V_b&lt;/th&gt;
&lt;th&gt;A * V_b&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;00&lt;/td&gt;
&lt;td&gt;00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;40&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;...&lt;/td&gt;
&lt;td&gt;...&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;f0&lt;/td&gt;
&lt;td&gt;fd&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;We then use vperm to look up entries in these tables and vxor to combine our results.&lt;/p&gt;
&lt;p&gt;So - and this is a key point - for each A value we wish to multiply by, we need to generate a new lookup table.&lt;/p&gt;
&lt;p&gt;So if we wanted A = 03:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;V_a&lt;/th&gt;
&lt;th&gt;A * V_a&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;00&lt;/td&gt;
&lt;td&gt;00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;01&lt;/td&gt;
&lt;td&gt;03&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02&lt;/td&gt;
&lt;td&gt;06&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;...&lt;/td&gt;
&lt;td&gt;...&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0f&lt;/td&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;V_b&lt;/th&gt;
&lt;th&gt;A * V_b&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;00&lt;/td&gt;
&lt;td&gt;00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;30&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;60&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;...&lt;/td&gt;
&lt;td&gt;...&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;f0&lt;/td&gt;
&lt;td&gt;0d&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;One final thing is that Power8 adds a vpermxor instruction, so we can reduce the entire 4 instruction sequence in the paper:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;vsrb v1, v0, v14
vperm v2, v12, v12, v0
vperm v1, v13, v13, v1
vxor v1, v2, v1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;to 1 vpermxor:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;vpermxor v1, v12, v13, v0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Isn't POWER grand?&lt;/p&gt;
&lt;h2&gt;OK, but how does this relate to erasure codes?&lt;/h2&gt;
&lt;p&gt;I'm glad you asked.&lt;/p&gt;
&lt;p&gt;Galois Field arithmetic, and its application in RAID 6 is the basis for erasure coding. (It's also the basis for CRCs - two for the price of one!)&lt;/p&gt;
&lt;p&gt;But, that's all to come in part 2, which will definitely be published before 7 April!&lt;/p&gt;
&lt;p&gt;Many thanks to Sarah Axtens who reviewed the mathematical content of this post and suggested significant improvements. All errors and gross oversimplifications remain my own. Thanks also to the OzLabs crew for their feedback and comments.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Mon, 20 Mar 2017 10:43:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-03-20:/blog/2017/03/20/erasure-coding-for-programmers-part-1/</guid><category>Development</category><category>erasure</category><category>raid</category><category>storage</category></item><item><title>High Power Lustre</title><link>https://sthbrx.github.io/blog/2017/02/13/high-power-lustre/</link><description>&lt;p&gt;(Most of the hard work here was done by fellow blogger Rashmica - I just verified her instructions and wrote up this post.)&lt;/p&gt;
&lt;p&gt;&lt;a href="http://lustre.org/"&gt;Lustre&lt;/a&gt; is a high-performance clustered file system. Traditionally the Lustre client and server have run on x86, but both the server and client will also work on Power. Here's how to get them running.&lt;/p&gt;
&lt;h1&gt;Server&lt;/h1&gt;
&lt;p&gt;Lustre normally requires a patched 'enterprise' kernel - normally an old RHEL, CentOS or SUSE kernel. We tested with a CentOS 7.3 kernel. We tried to follow &lt;a href="https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=52104622"&gt;the Intel instructions&lt;/a&gt; for building the kernel as much as possible - any deviations we had to make are listed below.&lt;/p&gt;
&lt;h2&gt;Setup quirks&lt;/h2&gt;
&lt;p&gt;We are told to edit &lt;code&gt;~/kernel/rpmbuild/SPEC/kernel.spec&lt;/code&gt;. This doesn't exist because the directory is &lt;code&gt;SPECS&lt;/code&gt; not &lt;code&gt;SPEC&lt;/code&gt;: you need to edit &lt;code&gt;~/kernel/rpmbuild/SPECS/kernel.spec&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;I also found there was an extra quote mark in the supplied patch script after &lt;code&gt;-lustre.patch&lt;/code&gt;. I removed that and ran this instead:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;for&lt;span class="w"&gt; &lt;/span&gt;patch&lt;span class="w"&gt; &lt;/span&gt;in&lt;span class="w"&gt; &lt;/span&gt;$(&lt;span class="err"&gt;&amp;lt;&lt;/span&gt;&amp;quot;3.10-rhel7.series&amp;quot;);&lt;span class="w"&gt; &lt;/span&gt;do&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;      &lt;/span&gt;patch_file=&amp;quot;&lt;span class="nv"&gt;$HOME&lt;/span&gt;/lustre-release/lustre/kernel_patches/patches/&lt;span class="cp"&gt;${&lt;/span&gt;&lt;span class="n"&gt;patch&lt;/span&gt;&lt;span class="cp"&gt;}&lt;/span&gt;&amp;quot;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;      &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&amp;quot;&lt;span class="cp"&gt;${&lt;/span&gt;&lt;span class="n"&gt;patch_file&lt;/span&gt;&lt;span class="cp"&gt;}&lt;/span&gt;&amp;quot;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;/lustre-kernel-x86_64-lustre.patch&lt;span class="w"&gt; &lt;/span&gt;\
done
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The fact that there is 'x86_64' in the patch name doesn't matter as you're about to copy it under a different name to a place where it will be included by the spec file.&lt;/p&gt;
&lt;h2&gt;Building for ppc64le&lt;/h2&gt;
&lt;p&gt;Building for ppc64le was reasonably straight-forward. I had one small issue:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;build&lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="n"&gt;dja&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;centos&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;guest&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rpmbuild&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rpmbuild&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;bp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="err"&gt;`&lt;/span&gt;&lt;span class="n"&gt;uname&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="err"&gt;`&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;./&lt;/span&gt;&lt;span class="n"&gt;SPECS&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;spec&lt;/span&gt;
&lt;span class="n"&gt;Building&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;platforms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ppc64le&lt;/span&gt;
&lt;span class="n"&gt;Building&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ppc64le&lt;/span&gt;
&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Failed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;build&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dependencies&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="n"&gt;net&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;needed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;by&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;3.10&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;327.36&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;3.&lt;/span&gt;&lt;span class="n"&gt;el7&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ppc64le&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Fixing this was as simple as a &lt;code&gt;yum install net-tools&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;This was sufficient to build the kernel RPMs. I installed them and booted to my patched kernel - so far so good!&lt;/p&gt;
&lt;h1&gt;Building the client packages: CentOS&lt;/h1&gt;
&lt;p&gt;I then tried to build and install the RPMs from &lt;a href="https://git.hpdd.intel.com/?p=fs/lustre-release.git;a=summary"&gt;&lt;code&gt;lustre-release&lt;/code&gt;&lt;/a&gt;. This repository provides the sources required to build the client and utility binaries.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;./configure&lt;/code&gt; and &lt;code&gt;make&lt;/code&gt; succeeded, but when I went to install the packages with &lt;code&gt;rpm&lt;/code&gt;, I found I was missing some dependencies:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Failed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dependencies&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;ldiskfsprogs&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1.42&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;wc1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;needed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;by&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kmod&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;lustre&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;osd&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;ldiskfs&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;2.9&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;52&lt;/span&gt;&lt;span class="n"&gt;_60_g1d2fbad_dirty&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;el7&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;centos&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ppc64le&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;sg3_utils&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;needed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;by&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lustre&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;iokit&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;2.9&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;52&lt;/span&gt;&lt;span class="n"&gt;_60_g1d2fbad_dirty&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;el7&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;centos&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ppc64le&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;needed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;by&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lustre&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;tests&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;2.9&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;52&lt;/span&gt;&lt;span class="n"&gt;_60_g1d2fbad_dirty&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;el7&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;centos&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ppc64le&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;lsof&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;needed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;by&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lustre&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;tests&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;2.9&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;52&lt;/span&gt;&lt;span class="n"&gt;_60_g1d2fbad_dirty&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;el7&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;centos&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ppc64le&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I was able to install &lt;code&gt;sg3_utils&lt;/code&gt;, &lt;code&gt;attr&lt;/code&gt; and &lt;code&gt;lsof&lt;/code&gt;, but I was still missing &lt;code&gt;ldiskfsprogs&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;It seems we need the lustre-patched version of &lt;code&gt;e2fsprogs&lt;/code&gt; - I found a &lt;a href="https://groups.google.com/forum/#!topic/lustre-discuss-list/U93Ja6Xkxfk"&gt;mailing list post&lt;/a&gt; to that effect.&lt;/p&gt;
&lt;p&gt;So, following the instructions on the walkthrough, I grabbed &lt;a href="https://downloads.hpdd.intel.com/public/e2fsprogs/latest/el7/SRPMS/"&gt;the SRPM&lt;/a&gt; and installed the dependencies: &lt;code&gt;yum install -y texinfo libblkid-devel libuuid-devel&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;I then tried &lt;code&gt;rpmbuild -ba SPECS/e2fsprogs-RHEL-7.spec&lt;/code&gt;. This built but failed tests. Some failed because I ran out of disk space - they were using 10s of gigabytes. I found that there were some comments in the spec file about this with suggested tests to disable, so I did that. Even with that fix, I was still failing two tests:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;f_pgsize_gt_blksize&lt;/code&gt;: Intel added this to their fork, and no equivalent exists in the master e2fsprogs branches. This relates to Intel specific assumptions about page sizes which don't hold on Power.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;f_eofblocks&lt;/code&gt;: This may need fixing for large page sizes, see &lt;a href="https://jira.hpdd.intel.com/browse/LU-4677?focusedCommentId=78814&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-78814"&gt;this bug&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I disabled the tests by adding the following two lines to the spec file, just before &lt;code&gt;make %{?_smp_mflags} check&lt;/code&gt;.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;rm -rf tests/f_pgsize_gt_blksize
rm -rf tests/f_eofblocks
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With those tests disabled I was able to build the packages successfully. I installed them with &lt;code&gt;yum localinstall *1.42.13.wc5*&lt;/code&gt; (I needed that rather weird pattern to pick up important RPMs that didn't fit the &lt;code&gt;e2fs*&lt;/code&gt; pattern - things like &lt;code&gt;libcom_err&lt;/code&gt; and &lt;code&gt;libss&lt;/code&gt;)&lt;/p&gt;
&lt;p&gt;Following that I went back to the &lt;code&gt;lustre-release&lt;/code&gt; build products and was able to successfully run &lt;code&gt;yum localinstall *ppc64le.rpm&lt;/code&gt;!&lt;/p&gt;
&lt;h1&gt;Testing the server&lt;/h1&gt;
&lt;p&gt;After disabling SELinux and rebooting, I ran the test script:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo /usr/lib64/lustre/tests/llmount.sh
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This spat out one scary warning:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;mount&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lustre&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;FATAL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;unhandled&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;unloaded&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fs&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ext3&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The test did seem to succeed overall, and it would seem that is a &lt;a href="https://jira.hpdd.intel.com/browse/LU-9059"&gt;known problem&lt;/a&gt;, so I pressed on undeterred.&lt;/p&gt;
&lt;p&gt;I then attached a couple of virtual harddrives for the metadata and object store volumes, and having set them up, proceeded to try to mount my freshly minted lustre volume from some clients.&lt;/p&gt;
&lt;h1&gt;Testing with a ppc64le client&lt;/h1&gt;
&lt;p&gt;My first step was to test whether another ppc64le machine would work as a client.&lt;/p&gt;
&lt;p&gt;I tried with an existing Ubuntu 16.04 VM that I use for much of my day to day development.&lt;/p&gt;
&lt;p&gt;A quick google suggested that I could grab the &lt;code&gt;lustre-release&lt;/code&gt; repository and run &lt;code&gt;make debs&lt;/code&gt; to get Debian packages for my system.&lt;/p&gt;
&lt;p&gt;I needed the following dependencies:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo apt install module-assistant debhelper dpatch libsnmp-dev quilt
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With those the packages built successfully, and could be easily installed:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;dpkg -i lustre-client-modules-4.4.0-57-generic_2.9.52-60-g1d2fbad-dirty-1_ppc64el.deblustre-utils_2.9.52-60-g1d2fbad-dirty-1_ppc64el.deb
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I tried to connect to the server:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;sudo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mount&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lustre&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;SERVER_IP&lt;/span&gt;&lt;span class="nv"&gt;@tcp&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;lustre&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;lustre&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Initially I wasn't able to connect to the server at all. I remembered that (unlike Ubuntu), CentOS comes with quite an aggressive firewall by default. I ran the following on the server:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl stop firewalld
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And voila! I was able to connect, mount the lustre volume, and successfully read and write to it. This is very much an over-the-top hack - I should have poked holes in the firewall to allow just the ports lustre needed. This is left as an exercise for the reader.&lt;/p&gt;
&lt;h1&gt;Testing with an x86_64 client&lt;/h1&gt;
&lt;p&gt;I then tried to run &lt;code&gt;make debs&lt;/code&gt; on my Ubuntu 16.10 x86_64 laptop.&lt;/p&gt;
&lt;p&gt;This did not go well - I got the following error:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;liblustreapi.c: In function ‘llapi_get_poollist’:
liblustreapi.c:1201:3: error: ‘readdir_r’ is deprecated [-Werror=deprecated-declarations]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This looks like one of the new errors introduced in recent GCC versions, and is &lt;a href="https://jira.hpdd.intel.com/browse/LU-8724?focusedCommentId=175244&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-175244"&gt;a known bug&lt;/a&gt;. To work around it, I found the following stanza in a &lt;code&gt;lustre/autoconf/lustre-core.m4&lt;/code&gt;, and removed the &lt;code&gt;-Werror&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;AS_IF([test $target_cpu == &amp;quot;i686&amp;quot; -o $target_cpu == &amp;quot;x86_64&amp;quot;],
        [CFLAGS=&amp;quot;$CFLAGS -Wall -Werror&amp;quot;])
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Even this wasn't enough: I got the following errors:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;home&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;dja&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;dev&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;lustre&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;release&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;debian&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;tmp&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;modules&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;deb&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;usr_src&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;modules&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;lustre&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;lustre&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;llite&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;dcache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;387&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;initialization&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;incompatible&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;pointer&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;Werror&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;incompatible&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;pointer&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;types&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;d_compare&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ll_dcompare&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;                  &lt;/span&gt;&lt;span class="o"&gt;^~~~~~~~~~~&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;home&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;dja&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;dev&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;lustre&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;release&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;debian&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;tmp&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;modules&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;deb&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;usr_src&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;modules&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;lustre&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;lustre&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;llite&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;dcache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;387&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;note&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;near&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;initialization&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;‘&lt;/span&gt;&lt;span class="nx"&gt;ll_d_ops&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;d_compare&lt;/span&gt;&lt;span class="err"&gt;’&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I figured this was probably because Ubuntu 16.10 has a 4.8 kernel, and Ubuntu 16.04 has a 4.4 kernel. Work on supporting 4.8 &lt;a href="https://jira.hpdd.intel.com/browse/LU-9003"&gt;is ongoing&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Sure enough, when I fired up a 16.04 x86_64 VM with a 4.4 kernel, I was able to build and install fine.&lt;/p&gt;
&lt;p&gt;Connecting didn't work first time - the guest failed to mount, but I did get the following helpful error on the server:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;LNetError&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2595&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;:(&lt;/span&gt;&lt;span class="n"&gt;acceptor&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;c&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;406&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="n"&gt;lnet_acceptor&lt;/span&gt;&lt;span class="o"&gt;())&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Refusing&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;connection&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;10.61&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;2.227&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;insecure&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Refusing insecure port 1024 made me thing that perhaps the NATing that qemu was performing for me was interfering - perhaps the server expected to get a connection where the source port was privileged, and qemu wouldn't be able to do that with NAT.&lt;/p&gt;
&lt;p&gt;Sure enough, switching NAT to bridging was enough to get the x86 VM to talk to the ppc64le server. I verified that &lt;code&gt;ls&lt;/code&gt;, reading and writing all succeeded.&lt;/p&gt;
&lt;h1&gt;Next steps&lt;/h1&gt;
&lt;p&gt;The obvious next steps are following up the disabled tests in e2fsprogs, and doing a lot of internal performance and functionality testing.&lt;/p&gt;
&lt;p&gt;Happily, it looks like Lustre might be in the mainline kernel before too long - parts have already started to go in to staging. This will make our lives a lot easier: for example, the breakage between 4.4 and 4.8 would probably have already been picked up and fixed if it was the main kernel tree rather than an out-of-tree patch set.&lt;/p&gt;
&lt;p&gt;In the long run, we'd like to make Lustre on Power just as easy as Lustre on x86. (And, of course, more performant!) We'll keep you up to date!&lt;/p&gt;
&lt;p&gt;(Thanks to fellow bloggers Daniel Black and Andrew Donnellan for useful feedback on this post.)&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Mon, 13 Feb 2017 16:29:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-02-13:/blog/2017/02/13/high-power-lustre/</guid><category>OpenPOWER</category><category>lustre</category><category>hpc</category></item><item><title>NAMD on NVLink</title><link>https://sthbrx.github.io/blog/2017/02/01/namd-on-nvlink/</link><description>&lt;p&gt;NAMD is a molecular dynamics program that can use GPU acceleration to speed up its calculations. Recent OpenPOWER machines like the IBM Power Systems S822LC for High Performance Computing (Minsky) come with a new interconnect for GPUs called NVLink, which offers extremely high bandwidth to a number of very powerful Nvidia Pascal P100 GPUs. So they're ideal machines for this sort of workload.&lt;/p&gt;
&lt;p&gt;Here's how to set up NAMD 2.12 on your Minsky, and how to debug some common issues. We've targeted this script for CentOS, but we've successfully compiled NAMD on Ubuntu as well.&lt;/p&gt;
&lt;h2&gt;Prerequisites&lt;/h2&gt;
&lt;h3&gt;GPU Drivers and CUDA&lt;/h3&gt;
&lt;p&gt;Firstly, you'll need CUDA and the NVidia drivers.&lt;/p&gt;
&lt;p&gt;You can install CUDA by following the instructions on NVidia's &lt;a href="https://developer.nvidia.com/cuda-downloads"&gt;CUDA Downloads&lt;/a&gt; page.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;yum&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;install&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;epel&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;release&lt;/span&gt;
&lt;span class="n"&gt;yum&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;install&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dkms&lt;/span&gt;
&lt;span class="c1"&gt;# download the rpm from the NVidia website&lt;/span&gt;
&lt;span class="n"&gt;rpm&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cuda&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rhel7&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;ga2&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;8.0&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;54&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;1.&lt;/span&gt;&lt;span class="n"&gt;ppc64le&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rpm&lt;/span&gt;
&lt;span class="n"&gt;yum&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;clean&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;expire&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;cache&lt;/span&gt;
&lt;span class="n"&gt;yum&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;install&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cuda&lt;/span&gt;
&lt;span class="c1"&gt;# this will take a while...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then, we set up a profile file to automatically load CUDA into our path:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;profile&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;cuda_path&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sh&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;EOF&lt;/span&gt;
&lt;span class="c1"&gt;# From http://developer.download.nvidia.com/compute/cuda/8.0/secure/prod/docs/sidebar/CUDA_Quick_Start_Guide.pdf - 4.4.2.1&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;cuda&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;8.0&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;PATH&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;PATH&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;LD_LIBRARY_PATH&lt;/span&gt;&lt;span class="o"&gt;=/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;cuda&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;8.0&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;lib64&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;LD_LIBRARY_PATH&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;LD_LIBRARY_PATH&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt;
&lt;span class="n"&gt;EOF&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now, open a new terminal session and check to see if it works:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cuda-install-samples-8.0.sh ~
cd ~/NVIDIA_CUDA-8.0_Samples/1_Utilities/bandwidthTest
make &amp;amp;&amp;amp; ./bandwidthTest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you see a figure of ~32GB/s, that means NVLink is working as expected. A figure of ~7-8GB indicates that only PCI is working, and more debugging is required.&lt;/p&gt;
&lt;h3&gt;Compilers&lt;/h3&gt;
&lt;p&gt;You need a c++ compiler:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;yum install gcc-c++
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2&gt;Building NAMD&lt;/h2&gt;
&lt;p&gt;Once CUDA and the compilers are installed, building NAMD is reasonably straightforward. The one hitch is that because we're using CUDA 8.0, and the NAMD build scripts assume CUDA 7.5, we need to supply an updated &lt;a href="/images/namd/Linux-POWER.cuda"&gt;Linux-POWER.cuda file&lt;/a&gt;. (We also enable code generation for the Pascal in this file.)&lt;/p&gt;
&lt;p&gt;We've documented the entire process as a script which you can &lt;a href="/images/namd/install-namd.sh"&gt;download&lt;/a&gt;. We'd recommend executing the commands one by one, but if you're brave you can run the script directly.&lt;/p&gt;
&lt;p&gt;The script will fetch NAMD 2.12 and build it for you, but won't install it. It will look for the CUDA override file in the directory you are running the script from, and will automatically move it into the correct place so it is picked up by the build system..&lt;/p&gt;
&lt;p&gt;The script compiles for a single multicore machine setup, rather than for a cluster. However, it should be a good start for an Ethernet or Infiniband setup.&lt;/p&gt;
&lt;p&gt;If you're doing things by hand, you may see some errors during the compilation of charm - as long as you get &lt;code&gt;charm++ built successfully.&lt;/code&gt; at the end, you should be OK.&lt;/p&gt;
&lt;h2&gt;Testing NAMD&lt;/h2&gt;
&lt;p&gt;We have been testing NAMD using the STMV files available from the &lt;a href="http://www.ks.uiuc.edu/Research/namd/utilities/"&gt;NAMD website&lt;/a&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cd NAMD_2.12_Source/Linux-POWER-g++
wget http://www.ks.uiuc.edu/Research/namd/utilities/stmv.tar.gz
tar -xf stmv.tar.gz
sudo ./charmrun +p80 ./namd2 +pemap 0-159:2 +idlepoll +commthread stmv/stmv.namd
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This binds a namd worker thread to every second hardware thread. This is because hardware threads share resources, so using every hardware thread costs overhead and doesn't give us access to any more physical resources.&lt;/p&gt;
&lt;p&gt;You should see messages about finding and using GPUs:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Pe 0 physical rank 0 binding to CUDA device 0 on &amp;lt;hostname&amp;gt;: &amp;#39;Graphics Device&amp;#39;  Mem: 4042MB  Rev: 6.0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This should be &lt;em&gt;significantly&lt;/em&gt; faster than on non-NVLink machines - we saw a gain of about 2x in speed going from a machine with Nvidia K80s to a Minsky. If things aren't faster for you, let us know!&lt;/p&gt;
&lt;h2&gt;Downloads&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="/images/namd/install-namd.sh"&gt;Install script for CentOS&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="/images/namd/Linux-POWER.cuda"&gt;Linux-POWER.cuda file&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Other notes&lt;/h2&gt;
&lt;p&gt;Namd requires some libraries, some of which they supply as binary downloads on &lt;a href="http://www.ks.uiuc.edu/Research/namd/libraries/"&gt;their website&lt;/a&gt;.
Make sure you get the ppc64le versions, not the ppc64 versions, otherwise you'll get errors like:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nl"&gt;ld:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;failed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;merge&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;specific&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rootdir&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;tcl&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;lib&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;libtcl8&lt;/span&gt;&lt;span class="mf"&gt;.5&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;regfree&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nl"&gt;ld:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rootdir&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;tcl&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;lib&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;libtcl8&lt;/span&gt;&lt;span class="mf"&gt;.5&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;regerror&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;compiled&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;big&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;endian&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;system&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;and&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;little&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;endian&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nl"&gt;ld:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;failed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;merge&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;specific&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rootdir&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;tcl&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;lib&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;libtcl8&lt;/span&gt;&lt;span class="mf"&gt;.5&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;regerror&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nl"&gt;ld:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rootdir&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;tcl&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;lib&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;libtcl8&lt;/span&gt;&lt;span class="mf"&gt;.5&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tclAlloc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;compiled&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;big&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;endian&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;system&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;and&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;little&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;endian&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The script we supply should get these right automatically.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Wed, 01 Feb 2017 08:32:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-02-01:/blog/2017/02/01/namd-on-nvlink/</guid><category>OpenPOWER</category><category>nvlink</category><category>namd</category><category>cuda</category><category>gpu</category><category>hpc</category><category>minsky</category><category>S822LC for hpc</category></item><item><title>linux.conf.au 2017 review</title><link>https://sthbrx.github.io/blog/2017/01/31/linuxconfau-2017-review/</link><description>&lt;p&gt;I recently attended LCA 2017, where I gave a talk at the Linux Kernel miniconf (run by fellow sthbrx blogger Andrew Donnellan!) and a talk at the main conference.&lt;/p&gt;
&lt;p&gt;I received some really interesting feedback so I've taken the opportunity to write some of it down to complement the talk videos and slides that are online. (And to remind me to follow up on it!)&lt;/p&gt;
&lt;h2&gt;Miniconf talk: Sparse Warnings&lt;/h2&gt;
&lt;p&gt;My kernel miniconf talk was on sparse warnings (&lt;a href="https://github.com/daxtens/sparse-warnings-talk/blob/master/talk.pdf"&gt;pdf slides&lt;/a&gt;, &lt;a href="https://www.youtube.com/watch?v=hmCukzpevUc"&gt;23m video&lt;/a&gt;).&lt;/p&gt;
&lt;p&gt;The abstract read (in part):&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;sparse is a semantic parser for C, and is one of the static analysis tools available to kernel devs.&lt;/p&gt;
&lt;p&gt;Sparse is a powerful tool with good integration into the kernel build system. However, we suffer from warning overload - there are too many sparse warnings to spot the serious issues amongst the trivial. This makes it difficult to use, both for developers and maintainers.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Happily, I received some feedback that suggests it's not all doom and gloom like I had thought!&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;Dave Chinner told me that the xfs team uses sparse regularly to make sure that the file system is endian-safe. This is good news - we really would like that to be endian-safe!&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Paul McKenney let me know that the 0day bot does do some sparse checking - it would just seem that it's not done on PowerPC.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Main talk: 400,000 Ephemeral Containers&lt;/h2&gt;
&lt;p&gt;My main talk was entitled "400,000 Ephemeral Containers: testing entire ecosystems with Docker". You can read the &lt;a href="https://linux.conf.au/schedule/presentation/81/"&gt;abstract&lt;/a&gt; for full details, but it boils down to:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;What if you want to test how &lt;em&gt;all&lt;/em&gt; the packages in a given ecosystem work in a given situation?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;My main example was testing how many of the Ruby packages successfully install on Power, but I also talk about other languages and other cool tests you could run.&lt;/p&gt;
&lt;p&gt;The &lt;a href="https://www.youtube.com/watch?v=v7wSqOQeGhA"&gt;44m video&lt;/a&gt; is online. I haven't put the slides up yet but they should be available &lt;a href="https://github.com/daxtens/400000-ephemeral-containers"&gt;on GitHub&lt;/a&gt; soonish.&lt;/p&gt;
&lt;p&gt;Unlike with the kernel talk, I didn't catch the names of most of the people with feedback.&lt;/p&gt;
&lt;h3&gt;Docker memory issues&lt;/h3&gt;
&lt;p&gt;One of the questions I received during the talk was about running into memory issues in Docker. I attempted to answer that during the Q&amp;amp;A. The person who asked the question then had a chat with me afterwards, and it turns out I had completely misunderstood the question. I thought it was about memory usage of running containers in parallel. It was actually about memory usage in the docker daemon when running lots of containers in serial. Apparently the docker daemon doesn't free memory during the life of the process, and the question was whether or not I had observed that during my runs.&lt;/p&gt;
&lt;p&gt;I didn't have a good answer for this at the time other than "it worked for me", so I have gone back and looked at the docker daemon memory usage.&lt;/p&gt;
&lt;p&gt;After a full Ruby run, the daemon is using about 13.9G of virtual memory, and 1.975G of resident memory. If I restart it, the memory usage drops to 1.6G of virtual and 43M of resident memory. So it would appear that the person asking the question was right, and I'm just not seeing it have an effect.&lt;/p&gt;
&lt;h3&gt;Other interesting feedback&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;Someone was quite interested in testing on Sparc, once they got their Go runtime nailed down.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;A Rackspacer was quite interested in Python testing for OpenStack - this has some intricacies around Py2/Py3, but we had an interesting discussion around just testing to see if packages that claim Py3 support provide Py3 support.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;A large jobs site mentioned using this technique to help them migrate their dependencies between versions of Go.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;I was 'gently encouraged' to try to do better with how long the process takes to run - if for no other reason than to avoid burning more coal. This is a fair point. I did not explain very well what I meant with diminishing returns in the talk: there's &lt;em&gt;lots&lt;/em&gt; you could do to make the process faster, it's just comes at the cost of the simplicity that I really wanted when I first started the project. I am working (on and off) on better ways to deal with this by considering the dependency graph.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Tue, 31 Jan 2017 16:07:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-01-31:/blog/2017/01/31/linuxconfau-2017-review/</guid><category>Education</category><category>conferences</category></item><item><title>Extracting Early Boot Messages in QEMU</title><link>https://sthbrx.github.io/blog/2017/01/30/extracting-early-boot-messages-in-qemu/</link><description>&lt;p&gt;Be me, you're a kernel hacker, you make some changes to your kernel, you boot
test it in QEMU, and it fails to boot. Even worse is the fact that it just hangs
without any failure message, no stack trace, no nothing. "Now what?" you think
to yourself.&lt;/p&gt;
&lt;p&gt;You probably do the first thing you learnt in debugging101 and add abundant
print statements all over the place to try and make some sense of what's
happening and where it is that you're actually crashing. So you do this, you
recompile your kernel, boot it in QEMU and lo and behold, nothing... What
happened? You added all these shiny new print statements, where did the output
go? The kernel still failed to boot (obviously), but where you were hoping to
get some clue to go on you were again left with an empty screen. "Maybe I
didn't print early enough" or "maybe I got the code paths wrong" you think,
"maybe I just need more prints" even. So lets delve a bit deeper, why didn't
you see those prints, where did they go, and how can you get at them?&lt;/p&gt;
&lt;h1&gt;__log_buf&lt;/h1&gt;
&lt;p&gt;So what happens when you call printk()? Well what normally happens is,
depending on the log level you set, the output is sent to the console or logged
so you can see it in dmesg. But what happens if we haven't registered a console
yet? Well then we can't print the message can we, so its logged in a buffer,
kernel log buffer to be exact helpfully named __log_buf.&lt;/p&gt;
&lt;h1&gt;Console Registration&lt;/h1&gt;
&lt;p&gt;So how come I eventually see print statements on my screen? Well at some point
during the boot process a console is registered with the printk system, and any
buffered output can now be displayed. On ppc it happens that this occurs in
register_early_udbg_console() called in setup_arch() from start_kernel(),
which is the generic kernel entry point. From this point forward when you print
something it will be displayed on the console, but what if you crash before
this? What are you supposed to do then?&lt;/p&gt;
&lt;h1&gt;Extracting Early Boot Messages in QEMU&lt;/h1&gt;
&lt;p&gt;And now the moment you've all been waiting for, how do I extract those early
boot messages in QEMU if my kernel crashes before the console is registered?
Well it's quite simple really, QEMU is nice enough to allow us to dump guest
memory, and we know the log buffer is in there some where, so we just need to
dump the correct part of memory which corresponds to the log buffer.&lt;/p&gt;
&lt;h4&gt;Locating __log_buf&lt;/h4&gt;
&lt;p&gt;Before we can dump the log buffer we need to know where it is. Luckily for us
this is fairly simple, we just need to dump all the kernel symbols and look for
the right one.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;nm&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;vmlinux&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;grep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__log_buf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;c000000000f5e3dc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__log_buf&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;We use the nm tool to list all the kernel symbols and output this into some
temporary file, we can then grep this for the log buffer (which we know to be
named __log_buf), and presto we are told that it's at kernel address 0xf5e3dc.&lt;/p&gt;
&lt;h4&gt;Dumping Guest Memory&lt;/h4&gt;
&lt;p&gt;It's then simply a case of dumping guest memory from the QEMU console. So first
we press ^a+c to get us to the QEMU console, then we can use the aptly named
dump-guest-memory.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;help&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;guest&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;memory&lt;/span&gt;
&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;guest&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;memory&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;z&lt;/span&gt;&lt;span class="o"&gt;|-&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;|-&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;filename&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;begin&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;guest&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;memory&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;into&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;filename&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;paging&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;guest&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;memory&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mapping&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;immediately&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;not&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;wait&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;completion&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;z&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kdump&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;compressed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;with&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;zlib&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;compression&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kdump&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;compressed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;with&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lzo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;compression&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kdump&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;compressed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;with&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;snappy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;compression&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nl"&gt;begin&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;starting&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;physical&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;address&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nl"&gt;length&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;memory&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;We just give it a filename for where we want our output to go, we know the
starting address, we just don't know the length. We could choose some arbitrary
length, but inspection of the kernel code shows us that:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;#define __LOG_BUF_LEN (1 &amp;lt;&amp;lt; CONFIG_LOG_BUF_SHIFT)&lt;/span&gt;
&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__log_buf&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;__LOG_BUF_LEN&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__aligned&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;LOG_ALIGN&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Looking at the pseries_defconfig file shows us that the LOG_BUF_SHIFT is set to
18, and thus we know that the buffer is 2^18 bytes or 256kb. So now we run:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;guest&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;memory&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xf5e3dc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;262144&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And we now get our log buffer in the file tmp. This can simply be viewed with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hexdump&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;C&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This gives a readable, if poorly formatted output. I'm sure you can find
something better but I'll leave that as an exercise for the reader.&lt;/p&gt;
&lt;h1&gt;Conclusion&lt;/h1&gt;
&lt;p&gt;So if like me your kernel hangs somewhere early in the boot process and you're
left without your console output you are now fully equipped to extract the log
buffer in QEMU and hopefully therein lies the answer to why you failed to boot.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Suraj Jitindar Singh</dc:creator><pubDate>Mon, 30 Jan 2017 16:47:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-01-30:/blog/2017/01/30/extracting-early-boot-messages-in-qemu/</guid><category>Virtualisation and Emulation</category><category>QEMU</category><category>debug</category><category>virtualisation</category><category>kernel</category><category>dmesg</category><category>printk</category><category>boot</category><category>early</category><category>error</category></item><item><title>Installing Centos 7.2 on IBM Power System's S822LC for High Performance Computing (Minksy) with USB device</title><link>https://sthbrx.github.io/blog/2017/01/30/installing-centos-72-on-ibm-power-systems-s822lc-for-high-performance-computing-minksy-with-usb-device/</link><description>&lt;h2&gt;Introduction&lt;/h2&gt;
&lt;p&gt;If you are installing Linux on your IBM Power System's S822LC server then the instructions in this article will help you to start and run your system.  These instructions are specific to installing CentOS 7 on an IBM Power System S822LC for High Performance Computing (Minsky), but also work for RHEL 7 - just swap CentOS for RHEL.&lt;/p&gt;
&lt;h3&gt;Prerequisites&lt;/h3&gt;
&lt;p&gt;Before you power on the system, ensure that you have the following items:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Ethernet cables;&lt;/li&gt;
&lt;li&gt;USB storage device of 7G or greater;&lt;/li&gt;
&lt;li&gt;An installed ethernet network with a DHCP server;&lt;/li&gt;
&lt;li&gt;Access to the DHCP server's logs;&lt;/li&gt;
&lt;li&gt;Power cords and outlet for your system;&lt;/li&gt;
&lt;li&gt;PC or notebook that has IPMItool level 1.8.15 or greater; and &lt;/li&gt;
&lt;li&gt;a VNC client.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Download CentOS ISO file from the &lt;a href="http://mirror.centos.org/altarch/7/isos/ppc64le/"&gt;Centos Mirror&lt;/a&gt;. Select the "Everything" ISO file.&lt;/p&gt;
&lt;p&gt;Note: You must use the 1611 release (dated 2016-12-22) or later due to Linux Kernel support for the server hardware.&lt;/p&gt;
&lt;h2&gt;Step 1: Preparing to power on your system&lt;/h2&gt;
&lt;p&gt;Follow these steps to prepare your system:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;If your system belongs in a rack, install your system into that rack. For instructions, see IBM POWER8 Systems information.&lt;/li&gt;
&lt;li&gt;Connect an Ethernet cable to the left embedded Ethernet port next to the serial port on the back of your system and the other end to your network. This Ethernet port is used for the BMC/IPMI interface.&lt;/li&gt;
&lt;li&gt;Connect another Enternet cable to the right Ethernet port for network connection for the operating system.&lt;/li&gt;
&lt;li&gt;Connect the power cords to the system and plug them into the outlets. &lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;At this point, your firmware is booting.&lt;/p&gt;
&lt;h2&gt;Step 2: Determining the BMC firmware IP address&lt;/h2&gt;
&lt;p&gt;To determine the IP address of the BMC, examine the latest DHCP server logs for the network connected to the server. The IP address will be requested approximately 2 minutes after being powered on.&lt;/p&gt;
&lt;p&gt;It is possible to set the BMC to a static IP address by following the &lt;a href="https://www.ibm.com/support/knowledgecenter/en/TI0003H/p8eih/p8eih_managing_with_ipmi_ami.htm"&gt;IBM documentation on IPMI&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;Step 3: Connecting to the BMC firmware with IPMItool&lt;/h2&gt;
&lt;p&gt;After you have a network connection set up for your BMC firmware, you can connect using Intelligent Platform Management Interface (IPMI).  IPMI is the default console to use when connecting to the Open Power Abstraction Layer (OPAL) firmware.&lt;/p&gt;
&lt;p&gt;Use the default authentication for servers over IPMI is:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Default user: ADMIN &lt;/li&gt;
&lt;li&gt;Default password: admin &lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;To power on your server from a PC or notebook that is running Linux®, follow these steps:&lt;/p&gt;
&lt;p&gt;Open a terminal program on your PC or notebook with &lt;a href="#active-sol-ipmi"&gt;Activate Serial-Over-Lan using IPMI&lt;/a&gt;. Use other steps here as needed.&lt;/p&gt;
&lt;p&gt;For the following impitool commands, server_ip_address is the IP address of the BMC from Step 2, and ipmi_user and ipmi_password are the default user ID and password for IPMI.&lt;/p&gt;
&lt;h3&gt;Power On using IPMI&lt;/h3&gt;
&lt;p&gt;If your server is not powered on, run the following command to power the server on:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;ipmitool&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lanplus&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;H&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;server_ip_address&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;U&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipmi_user&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;P&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipmi_password&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;chassis&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;power&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3&gt;&lt;a name="active-sol-ipmi"&gt;&lt;/a&gt;Activate Serial-Over-Lan using IPMI&lt;/h3&gt;
&lt;p&gt;Activate your IPMI console by running this command:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;ipmitool&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lanplus&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;H&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;server_ip_address&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;U&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipmi_user&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;P&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipmi_password&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sol&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;activate&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After powering on your system, the Petitboot interface loads. If you do not interrupt the boot process by pressing any key within 10 seconds, Petitboot automatically boots the first option. At this point the IPMI console will be connected to the Operating Systems serial. If you get to this stage accidently you can deactivate and reboot as per the following two commands.&lt;/p&gt;
&lt;h3&gt;Deactivate Serial-Over-Lan using IPMI&lt;/h3&gt;
&lt;p&gt;If you need to power off or reboot your system, deactivate the console by running this command:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;ipmitool&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lanplus&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;H&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;server_ip_address&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;U&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;P&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipmi_password&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sol&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;deactivate&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3&gt;Reboot using IPMI&lt;/h3&gt;
&lt;p&gt;If you need to reboot the system, run this command: &lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;ipmitool&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lanplus&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;H&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;server_ip_address&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;U&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;P&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipmi_password&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;chassis&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;power&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;reset&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2&gt;Step 4: Creating a USB device and booting&lt;/h2&gt;
&lt;p&gt;At this point, your IPMI console should be contain a Petitboot bootloader menu as illustrated below and you are ready to install Centos 7 on your server.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Petitboot menu over IPMI" src="/images/centos7-minsky/petitboot-centos7-usb-topmenu.png"&gt; &lt;/p&gt;
&lt;p&gt;Use one of the following USB devices:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;USB attached DVD player with a single USB cable to stay under 1.0 Amps, or&lt;/li&gt;
&lt;li&gt;7 GB (or more) 2.0 (or later) USB flash drive. &lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Follow the following instructions:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;To create the bootable USB device, follow the instructions in the CentOS wiki &lt;a href="https://wiki.centos.org/HowTos/InstallFromUSBkey"&gt;Host to Set Up a USB to Install CentOS&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Insert your bootable USB device into the front USB port. CentOS AltArch installer will automatically appear as a boot option on the Petitboot main screen. If the USB device does not appear select &lt;em&gt;Rescan devices&lt;/em&gt;. If your device is not detected, you might have to try a different type.&lt;/li&gt;
&lt;li&gt;Arrow up to select the CentOS boot option. Press &lt;em&gt;e&lt;/em&gt; (Edit) to open the Petitboot Option Editor window&lt;/li&gt;
&lt;li&gt;Move the cursor to the Boot arguments section and to include the following information: &lt;code&gt;ro inst.stage2=hd:LABEL=CentOS_7_ppc64le:/ console=hvc0 ip=dhcp&lt;/code&gt; (if using RHEL the LABEL will be similar to &lt;code&gt;RHEL-7.3\x20Server.ppc64le:/&lt;/code&gt;)&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;img alt="Petitboot edited &amp;quot;Install CentOS AltArch 7 (64-bit kernel)" src="/images/centos7-minsky/petitboot-centos7-usb-option-editor-menu.png"&gt;&lt;/p&gt;
&lt;p&gt;Notes about the boot arguments:   &lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;ip=dhcp&lt;/code&gt; to ensure network is started for VNC installation.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;console hvc0&lt;/code&gt; is needed as this is not the default.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;inst.stage2&lt;/code&gt; is needed as the boot process won't automatically find the stage2 install on the install disk.&lt;/li&gt;
&lt;li&gt;append &lt;code&gt;inst.proxy=URL&lt;/code&gt; where URL is the proxy URL if installing in a network that requires a proxy to connect externally.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;You can find additional options at &lt;a href="https://rhinstaller.github.io/anaconda/boot-options.html"&gt;Anaconda Boot Options&lt;/a&gt;.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Select &lt;em&gt;OK&lt;/em&gt; to save your options and return to the Main menu &lt;/li&gt;
&lt;li&gt;On the Petitboot main screen, select the CentOS AltArch option and then press &lt;em&gt;Enter&lt;/em&gt;. &lt;/li&gt;
&lt;/ol&gt;
&lt;h2&gt;Step 5: Complete your installation&lt;/h2&gt;
&lt;p&gt;After you select to boot the CentOS installer, the installer wizard walks you through the steps.  &lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;If the CentOS installer was able to obtain a network address via DHCP, it will present an option to enable the VNC. If no option is presented check your network cables. &lt;img alt="VNC option" src="/images/centos7-minsky/anaconda-centos7-text-start.png"&gt;&lt;/li&gt;
&lt;li&gt;Select the &lt;em&gt;Start VNC&lt;/em&gt; option and it will provide an OS server IP adress. Note that this will be different to the BMC address previously optained. &lt;img alt="VNC option selected" src="/images/centos7-minsky/anaconda-centos7-vnc-selected.png"&gt;&lt;/li&gt;
&lt;li&gt;Run a VNC client program on your PC or notebook and connect to the OS server IP address.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;img alt="VNC of Installer" src="/images/centos7-minsky/anaconda-centos7-vnc-start.png"&gt;&lt;/p&gt;
&lt;p&gt;During the install over VNC, there are a couple of consoles active. To switch between them in the ipmitool terminal, press &lt;em&gt;ctrl-b&lt;/em&gt; and then between &lt;em&gt;1&lt;/em&gt;-&lt;em&gt;4&lt;/em&gt; as indicated.&lt;/p&gt;
&lt;p&gt;Using the VNC client program:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Select "Install Destination"&lt;/li&gt;
&lt;li&gt;Select a device from "Local Standard Disks"&lt;/li&gt;
&lt;li&gt;Select "Full disk summary and boot device"&lt;/li&gt;
&lt;li&gt;Select the device again from "Selected Disks" with the Boot enabled&lt;/li&gt;
&lt;li&gt;Select "Do not install boot loader" from device. &lt;img alt="Disabling install of boot loader" src="/images/centos7-minsky/anaconda-centos7-vnc-installation-destination-do-not-install-boot-loader.png"&gt; which results in &lt;img alt="Result after disabling boot loader install" src="/images/centos7-minsky/anaconda-centos7-vnc-installation-destination-do-not-install-boot-loader-result.png"&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Without disabling boot loader, the installer complains about &lt;code&gt;an invalid stage1 device&lt;/code&gt;. I suspect it needs a manual Prep partition of 10M to make the installer happy.&lt;/p&gt;
&lt;p&gt;If you have a local Centos repository  you can set this by selecting "Install Source" - the directories at this url should look like &lt;a href="http://mirror.centos.org/altarch/7/os/ppc64le/"&gt;CentOS's Install Source for ppc64le&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;Step 6: Before reboot and using the IPMI Serial-Over-LAN&lt;/h2&gt;
&lt;p&gt;Before reboot, generate the grub.cfg file as Petitboot uses this to generate its boot menu: &lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Using the ipmitool's shell (&lt;em&gt;ctrl-b 2&lt;/em&gt;):&lt;/li&gt;
&lt;li&gt;Enter the following commands to generate a grub.cfg file&lt;/li&gt;
&lt;/ol&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;chroot&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;mnt&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;sysimage&lt;/span&gt;
&lt;span class="nv"&gt;rm&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;grub&lt;/span&gt;.&lt;span class="nv"&gt;d&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="nv"&gt;_os&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;prober&lt;/span&gt;
&lt;span class="nv"&gt;grub2&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;mkconfig&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;o&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;boot&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;grub2&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;grub&lt;/span&gt;.&lt;span class="nv"&gt;cfg&lt;/span&gt;
&lt;span class="k"&gt;exit&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;/etc/grub.d/30_os-prober&lt;/code&gt; is removed as Petitboot probes the other devices anyway so including it would create lots of duplicate menu items.&lt;/p&gt;
&lt;p&gt;The last step is to restart your system.&lt;/p&gt;
&lt;p&gt;Note: While your system is restarting, remove the USB device. &lt;/p&gt;
&lt;p&gt;After the system restarts, Petitboot displays the option to boot CentOS 7.2. Select this option and press Enter. &lt;/p&gt;
&lt;h2&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;After you have booted CentOS, your server is ready to go!
For more information, see the following resources:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.ibm.com/support/knowledgecenter/"&gt;IBM Knowledge Center&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=fe313521-2e95-46f2-817d-44a4f27eba32"&gt;The Linux on Power Community&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://developer.ibm.com/linuxonpower/category/announcements/"&gt;The Linux on Power Developer Center&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://twitter.com/ibmpowerlinux"&gt;Follow us @ibmpowerlinux&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Black</dc:creator><pubDate>Mon, 30 Jan 2017 08:54:33 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2017-01-30:/blog/2017/01/30/installing-centos-72-on-ibm-power-systems-s822lc-for-high-performance-computing-minksy-with-usb-device/</guid><category>OpenPOWER</category><category>S822LC for hpc</category><category>hpc</category><category>centos</category><category>centos7</category><category>p8</category><category>bmc</category><category>RHEL</category></item><item><title>Getting In Sync</title><link>https://sthbrx.github.io/blog/2016/08/17/getting-in-sync/</link><description>&lt;p&gt;Since at least v1.0.0 Petitboot has used device-mapper snapshots to avoid
mounting block devices directly. Primarily this is so Petitboot can mount disks
and potentially perform filesystem recovery without worrying about messing it up
and corrupting a host's boot partition - all changes happen to the snapshot in
memory without affecting the actual device.&lt;/p&gt;
&lt;p&gt;This of course gets in the way if you actually &lt;em&gt;do&lt;/em&gt; want to make changes to a
block device. Petitboot will allow certain bootloader scripts to make changes
to disks if configured (eg, grubenv updates), but if you manually make changes
you would need to know the special sequence of &lt;code&gt;dmsetup&lt;/code&gt; commands to merge the
snapshots back to disk. This is particulary annoying if you're trying to copy
logs to a USB device!&lt;/p&gt;
&lt;p&gt;Depending on how recent a version of Petitboot you're running, there are two
ways of making sure your changes persist:&lt;/p&gt;
&lt;h2&gt;Before v1.2.2&lt;/h2&gt;
&lt;p&gt;If you really need to save changes from within Petitboot, the most
straightforward way is to disable snapshots. Drop to the shell and enter&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;nvram --update-config petitboot,snapshots?=false
reboot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Once you have rebooted you can remount the device as read-write and
modify it as normal.&lt;/p&gt;
&lt;h2&gt;After v1.2.2&lt;/h2&gt;
&lt;p&gt;To make this easier while keeping the benefit of snapshots, v1.2.2 introduces
a new user-event that will merge snapshots on demand. For example:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;mount&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;remount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;rw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;petitboot&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;mnt&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;sda2&lt;/span&gt;
&lt;span class="n"&gt;cp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;petitboot&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;mnt&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;sda2&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;
&lt;span class="n"&gt;pb&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sync&lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="n"&gt;sda2&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After calling &lt;code&gt;pb-event sync@yourdevice&lt;/code&gt;, Petitboot will remount the device back to
read-only and merge the current snapshot differences back to disk. You can also
run &lt;code&gt;pb-event sync@all&lt;/code&gt; to sync all existing snapshots if desired.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Wed, 17 Aug 2016 15:23:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-08-17:/blog/2016/08/17/getting-in-sync/</guid><category>Petitboot</category><category>skiroot</category><category>petitboot</category><category>goodposts</category><category>realcontent</category><category>devmapper</category><category>lvm</category><category>debug</category></item><item><title>Get off my lawn: separating Docker workloads using cgroups</title><link>https://sthbrx.github.io/blog/2016/07/27/get-off-my-lawn-separating-docker-workloads-using-cgroups/</link><description>&lt;p&gt;On my team, we do two different things in our Continuous Integration setup: build/functional tests, and performance tests. Build tests simply test whether a project builds, and, if the project provides a functional test suite, that the tests pass. We do a lot of MySQL/MariaDB testing this way. The other type of testing we do is performance tests: we build a project and then run a set of benchmarks against it. Python is a good example here.&lt;/p&gt;
&lt;p&gt;Build tests want as much grunt as possible. Performance tests, on the other hand, want a stable, isolated environment. Initially, we set up Jenkins so that performance and build tests never ran at the same time. Builds would get the entire machine, and performance tests would never have to share with anyone.&lt;/p&gt;
&lt;p&gt;This, while simple and effective, has some downsides. In POWER land, our machines are quite beefy. For example, one of the boxes I use - an S822L - has 4 sockets, each with 4 cores. At SMT-8 (an 8 way split of each core) that gives us 4 x 4 x 8 = 128 threads. It seems wasteful to lock this entire machine - all 128 threads - just so as to isolate a single-threaded test.&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1"&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;p&gt;So, &lt;strong&gt;can we partition our machine so that we can be running two different sorts of processes in a sufficiently isolated way?&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;What counts as 'sufficiently isolated'? Well, my performance tests are CPU bound, so I want CPU isolation. I also want memory, and in particular memory bandwith to be isolated. I don't particularly care about IO isolation as my tests aren't IO heavy. Lastly, I have a couple of tests that are very multithreaded, so I'd like to have enough of a machine for those test results to be interesting.&lt;/p&gt;
&lt;p&gt;For CPU isolation we have CPU affinity. We can also do something similar with memory. On a POWER8 system, memory is connected to individual P8s, not to some central point. This is a 'Non-Uniform Memory Architecture' (NUMA) setup: the directly attached memory will be very fast for a processor to access, and memory attached to other processors will be slower to access. An accessible guide (with very helpful diagrams!) is &lt;a href="http://www.redbooks.ibm.com/redpapers/pdfs/redp5098.pdf"&gt;the relevant RedBook (PDF)&lt;/a&gt;, chapter 2.&lt;/p&gt;
&lt;p&gt;We could achieve the isolation we want by dividing up CPUs and NUMA nodes between the competing workloads. Fortunately, all of the hardware NUMA information is plumbed nicely into Linux. Each P8 socket gets a corresponding NUMA node. &lt;code&gt;lscpu&lt;/code&gt; will tell you what CPUs correspond to which NUMA nodes (although what it calls a CPU we would call a hardware thread). If you install &lt;code&gt;numactl&lt;/code&gt;, you can use &lt;code&gt;numactl -H&lt;/code&gt; to get even more details.&lt;/p&gt;
&lt;p&gt;In our case, the relevant &lt;code&gt;lscpu&lt;/code&gt; output is thus:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;NUMA node0 CPU(s):     0-31
NUMA node1 CPU(s):     96-127
NUMA node16 CPU(s):    32-63
NUMA node17 CPU(s):    64-95
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now all we have to do is find some way to tell Linux to restrict a group of processes to a particular NUMA node and the corresponding CPUs. How? Enter control groups, or &lt;code&gt;cgroups&lt;/code&gt; for short. Processes can be put into a cgroup, and then a cgroup controller can control the resouces allocated to the cgroup. Cgroups are hierarchical, and there are controllers for a number of different ways you could control a group of processes. Most helpfully for us, there's one called &lt;code&gt;cpuset&lt;/code&gt;, which can control CPU affinity, and restrict memory allocation to a NUMA node.&lt;/p&gt;
&lt;p&gt;We then just have to get the processes into the relevant cgroup. Fortunately, Docker is incredibly helpful for this!&lt;sup id="fnref:2"&gt;&lt;a class="footnote-ref" href="#fn:2"&gt;2&lt;/a&gt;&lt;/sup&gt; Docker containers are put in the &lt;code&gt;docker&lt;/code&gt; cgroup. Each container gets it's own cgroup under the docker cgroup, and fortunately Docker deals well with the somewhat broken state of cpuset inheritance.&lt;sup id="fnref:3"&gt;&lt;a class="footnote-ref" href="#fn:3"&gt;3&lt;/a&gt;&lt;/sup&gt; So it suffices to create a cpuset cgroup for docker, and allocate some resources to it, and Docker will do the rest. Here we'll allocate the last 3 sockets and NUMA nodes to Docker containers:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cgcreate&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;cpuset:docker
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;32&lt;/span&gt;-127&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/sys/fs/cgroup/cpuset/docker/cpuset.cpus
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;,16-17&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/sys/fs/cgroup/cpuset/docker/cpuset.mems
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/sys/fs/cgroup/cpuset/docker/cpuset.mem_hardwall
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;mem_hardwall&lt;/code&gt; prevents memory allocations under docker from spilling over into the one remaining NUMA node.&lt;/p&gt;
&lt;p&gt;So, does this work? I created a container with sysbench and then ran the following:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@0d3f339d4181:/#&lt;span class="w"&gt; &lt;/span&gt;sysbench&lt;span class="w"&gt; &lt;/span&gt;--test&lt;span class="o"&gt;=&lt;/span&gt;cpu&lt;span class="w"&gt; &lt;/span&gt;--num-threads&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;128&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--max-requests&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;10000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;run
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now I've asked for 128 threads, but the cgroup only has CPUs/hwthreads 32-127 allocated. So If I run htop, I shouldn't see any load on CPUs 0-31. What do I actually see?&lt;/p&gt;
&lt;p&gt;&lt;img alt="htop screenshot, showing load only on CPUs 32-127" src="/images/dja/cgroup1.png"&gt;&lt;/p&gt;
&lt;p&gt;It works! Now, we create a cgroup for performance tests using the first socket and NUMA node:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cgcreate&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;cpuset:perf-cgroup
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;-31&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/sys/fs/cgroup/cpuset/perf-cgroup/cpuset.cpus
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/sys/fs/cgroup/cpuset/perf-cgroup/cpuset.mems
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/sys/fs/cgroup/cpuset/perf-cgroup/cpuset.mem_hardwall
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Docker conveniently lets us put new containers under a different cgroup, which means we can simply do:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;dja@p88&lt;span class="w"&gt; &lt;/span&gt;~&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;docker&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-it&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;--cgroup-parent&lt;span class="o"&gt;=&lt;/span&gt;/perf-cgroup/&lt;span class="w"&gt; &lt;/span&gt;ppc64le/ubuntu&lt;span class="w"&gt; &lt;/span&gt;bash
root@b037049f94de:/#&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# ... install sysbench&lt;/span&gt;
root@b037049f94de:/#&lt;span class="w"&gt; &lt;/span&gt;sysbench&lt;span class="w"&gt; &lt;/span&gt;--test&lt;span class="o"&gt;=&lt;/span&gt;cpu&lt;span class="w"&gt; &lt;/span&gt;--num-threads&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;128&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--max-requests&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;10000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;run
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And the result?&lt;/p&gt;
&lt;p&gt;&lt;img alt="htop screenshot, showing load only on CPUs 0-31" src="/images/dja/cgroup2.png"&gt;&lt;/p&gt;
&lt;p&gt;It works! My benchmark results also suggest this is sufficient isolation, and the rest of the team is happy to have more build resources to play with.&lt;/p&gt;
&lt;p&gt;There are some boring loose ends to tie up: if a build job does anything outside of docker (like clone a git repo), that doesn't come under the docker cgroup, and we have to interact with systemd. Because systemd doesn't know about cpuset, this is &lt;em&gt;quite&lt;/em&gt; fiddly. We also want this in a systemd unit so it runs on start up, and we want some code to tear it down. But I'll spare you the gory details.&lt;/p&gt;
&lt;p&gt;In summary, cgroups are surprisingly powerful and simple to work with, especially in conjunction with Docker and NUMA on Power!&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;It gets worse! Before the performance test starts, all the running build jobs must drain. If we have 8 Jenkins executors running on the box, and a performance test job is the next in the queue, we have to wait for 8 running jobs to clear. If they all started at different times and have different runtimes, we will inevitably spend a fair chunk of time with the machine at less than full utilisation while we're waiting.&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:2"&gt;
&lt;p&gt;At least, on Ubuntu 16.04. I haven't tested if this is true anywhere else.&amp;#160;&lt;a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:3"&gt;
&lt;p&gt;I hear this is getting better. It is also why systemd hasn't done cpuset inheritance yet.&amp;#160;&lt;a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Wed, 27 Jul 2016 13:30:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-07-27:/blog/2016/07/27/get-off-my-lawn-separating-docker-workloads-using-cgroups/</guid><category>Performance</category><category>cgroups</category><category>numa</category><category>p8</category></item><item><title>Where to Get a POWER8 Development VM</title><link>https://sthbrx.github.io/blog/2016/07/06/where-to-get-a-power8-development-vm/</link><description>&lt;p&gt;&lt;em&gt;POWER8 sounds great, but where the heck can I get a Power VM so I can test my code?&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;This is a common question we get at OzLabs from other open source developers looking to port their software to the Power Architecture. Unfortunately, most developers don't have one of our amazing servers just sitting around under their desk.&lt;/p&gt;
&lt;p&gt;Thankfully, there's a few IBM partners who offer free VMs for development use. If you're in need of a development VM, check out:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="http://openpower.ic.unicamp.br/minicloud/"&gt;MiniCloud&lt;/a&gt;, hosted by the State University of Campinas (Unicamp), Brazil&lt;/li&gt;
&lt;li&gt;&lt;a href="http://osuosl.org/services/powerdev"&gt;OSU Open Source Lab&lt;/a&gt;, hosted by the Oregon State University&lt;/li&gt;
&lt;li&gt;&lt;a href="https://ptopenlab.com/cloudlabconsole"&gt;SuperVessel Cloud for Power/OpenPOWER&lt;/a&gt;, hosted by IBM China&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;So, next time you wonder how you can test your project on POWER8, request a VM and get to it!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Donnellan</dc:creator><pubDate>Wed, 06 Jul 2016 16:00:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-07-06:/blog/2016/07/06/where-to-get-a-power8-development-vm/</guid><category>OpenPOWER</category><category>Development</category></item><item><title>Optical Action at a Distance</title><link>https://sthbrx.github.io/blog/2016/07/05/optical-action-at-a-distance/</link><description>&lt;p&gt;Generally when someone wants to install a Linux distro they start with an ISO
file. Now we could burn that to a DVD, walk into the server room, and put it in
our machine, but that's a pain. Instead let's look at how to do this over the
network with Petitboot!&lt;/p&gt;
&lt;p&gt;At the moment Petitboot won't be able to handle an ISO file unless it's
mounted in an expected place (eg. as a mounted DVD), so we need to unpack it
somewhere. Choose somewhere to host the result and unpack the ISO via whatever
method you prefer. (For example &lt;code&gt;bsdtar -xf /path/to/image.iso&lt;/code&gt;).&lt;/p&gt;
&lt;p&gt;You'll get a bunch of files but for our purposes we only care about a few; the
kernel, the initrd, and the bootloader configuration file. Using
the Ubuntu 16.04 ppc64el ISO as an example, these are:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;./install/vmlinux
./install/initrd.gz.
./boot/grub/grub.cfg
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In grub.cfg we can see that the boot arguments are actually quite simple:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;set timeout=-1

menuentry &amp;quot;Install&amp;quot; {
    linux   /install/vmlinux tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=false --- quiet
    initrd  /install/initrd.gz
}

menuentry &amp;quot;Rescue mode&amp;quot; {
    linux   /install/vmlinux rescue/enable=true --- quiet
    initrd  /install/initrd.gz
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;So all we need to do is create a PXE config file that points Petitboot towards
the correct files.&lt;/p&gt;
&lt;p&gt;We're going to create a PXE config file which you could serve from your DHCP
server, but that does not mean we need to use PXE - if you just want a quick
install you only need make these files accessible to Petitboot, and then we can
use the 'Retrieve config from URL' option to download the files.&lt;/p&gt;
&lt;p&gt;Create a petitboot.conf file somewhere accessible that contains (for Ubuntu):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;label Install Ubuntu 16.04 Xenial Xerus
    kernel http://myaccesibleserver/path/to/vmlinux
    initrd http://myaccesibleserver/path/to/initrd.gz
    append tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=false --- quiet
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then in Petitboot, select 'Retrieve config from URL' and enter
&lt;code&gt;http://myaccesibleserver/path/to/petitboot.conf&lt;/code&gt;. In the main menu your new
option should appear - select it and away you go!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Tue, 05 Jul 2016 15:23:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-07-05:/blog/2016/07/05/optical-action-at-a-distance/</guid><category>Petitboot</category><category>petitboot</category><category>goodposts</category><category>realcontent</category><category>netboot</category><category>pxe</category></item><item><title>A Taste of IBM</title><link>https://sthbrx.github.io/blog/2016/07/01/a-taste-of-ibm/</link><description>&lt;p&gt;As a hobbyist programmer and Linux user, I was pretty stoked to be able to experience real work in the IT field that interests me most, Linux. With a mainly disconnected understanding of computer hardware and software, I braced myself to entirely relearn everything and anything I thought I knew. Furthermore, I worried that my usefulness in a world of maintainers, developers and testers would not be enough to provide any real contribution to the company. In actual fact however, the employees at OzLabs (IBM ADL) put a really great effort into making use of my existing skills, were attentive to my current knowledge and just filled in the gaps! The knowledge they've given me is practical, interlinked with hardware and provided me with the foot-up that I'd been itching for to establish my own portfolio as a programmer. I was both honoured and astonished by their dedication to helping me make a truly meaningful contribution!&lt;/p&gt;
&lt;p&gt;On applying for the placement, I listed my skills and interests. Having a Mathematics, Science background, I listed among my greatest interests development of scientific simulation and graphics using libraries such as Python matplotlib and R. By the &lt;em&gt;first day&lt;/em&gt; they got me to work, researching and implementing a routine in R that would qualitatively model the ability of a system to perform common tasks - a benchmark. A series of these microbenchmarks were made; I was in my element and actually able to contribute to a corporation much larger than I could imagine. The team at IBM reinforced my knowledge from the ground up, introducing the rigorous hardware and corporate elements at a level I was comfortable with.&lt;/p&gt;
&lt;p&gt;I would say that my greatest single piece of take-home knowledge over the two weeks was knowledge of the Linux Kernel project, Git and GitHub. Having met the arch/powerpc and linux-next maintainers in person placed the Linux and Open Source development cycle in an entirely new perspective. I was introduced to the world of GitHub, and thanks to a few rigorous lessons of Git, I now have access to tools that empower me to safely and efficiently write code, and to build a public portfolio I can be proud of. Most members of the office donated their time to instruct me on all fronts, whether to do with career paths, programming expertise or conceptual knowledge, and the rest were all very good for a chat.&lt;/p&gt;
&lt;p&gt;Approaching the tail-end of Year Twelve, I was blessed with some really good feedback and recommendations regarding further study. If during the two weeks I had any query regarding anything ranging from work-life to programming expertise even to which code editor I should use (a source of much contention) the people in the office were very happy to help me. Several employees donated their time to teach me really very intensive and long lessons regarding the software development concepts, including (but not limited to!) a thorough and helpful lesson on Git that was just on my level of understanding.&lt;/p&gt;
&lt;p&gt;Working at IBM these past two weeks has not only bridged the gap between my hobby and my professional prospects, but more importantly established friendships with professionals in the field of Software Development. Without a doubt this really great experience of an environment that rewards my enthusiasm will fondly stay in my mind as I enter the next chapter of my life!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Rohan McLure</dc:creator><pubDate>Fri, 01 Jul 2016 11:45:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-07-01:/blog/2016/07/01/a-taste-of-ibm/</guid><category>Education</category><category>Education</category><category>work experience</category></item><item><title>Kernel interfaces and vDSO test</title><link>https://sthbrx.github.io/blog/2016/06/24/kernel-interfaces-and-vdso-test/</link><description>&lt;h3&gt;Getting Suckered&lt;/h3&gt;
&lt;p&gt;Last week a colleague of mine came up to me and showed me some of the
&lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt; on PowerPC and asked why on earth does it fail
&lt;a href="https://github.com/nlynch-mentor/vdsotest"&gt;vdsotest&lt;/a&gt;. I should come
clean at this point and admit that I knew very little about the &lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt;
and hadn't heard of vdsotest. I had to admit to this colleague that I
had no idea everything looked super sane.&lt;/p&gt;
&lt;p&gt;Unfortunately (for me) I got hooked, vdsotest was saying it was
getting '22' instead of '-1' and it was the case where the &lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt; would
call into the kernel. It plagued me all night, 22 is so suspicious.
Right before I got to work the next morning I had an epiphany, "I bet
22 is EINVAL".&lt;/p&gt;
&lt;h3&gt;Virtual Dynamically linked Shared Objects&lt;/h3&gt;
&lt;p&gt;The &lt;a href="https://en.wikipedia.org/wiki/VDSO"&gt;&lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt;&lt;/a&gt; is a mechanism to
expose some kernel functionality into userspace to avoid the cost of a
context switch into kernel mode. This is a great feat of engineering,
avoiding the context switch can have a dramatic speedup for userspace
code. Obviously not all kernel functionality can be placed into
userspace and even for the functionality which can,
there may be edge cases in which the &lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt; needs to ask the kernel.&lt;/p&gt;
&lt;p&gt;Who tests the &lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt;? For the portion that lies exclusively in userspace it
will escape all testing of the syscall interface which is really what
kernel developers are so focused on not breaking. Enter Nathan Lynch
with &lt;a href="https://github.com/nlynch-mentor/vdsotest"&gt;vdsotest&lt;/a&gt; who has
done some great work!&lt;/p&gt;
&lt;h3&gt;The Kernel&lt;/h3&gt;
&lt;p&gt;When the &lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt; can't get the correct value without the kernel, it
simply calls into the kernel because the kernel is the definitive
reference for every syscall. On PowerPC something like this happens
(sorry, our &lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt; is 100% asm):
&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1"&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cm"&gt;/*&lt;/span&gt;
&lt;span class="cm"&gt; * Exact prototype of clock_gettime()&lt;/span&gt;
&lt;span class="cm"&gt; *&lt;/span&gt;
&lt;span class="cm"&gt; * int __kernel_clock_gettime(clockid_t clock_id, struct timespec *tp);&lt;/span&gt;
&lt;span class="cm"&gt; *&lt;/span&gt;
&lt;span class="cm"&gt; */&lt;/span&gt;
&lt;span class="nf"&gt;V_FUNCTION_BEGIN&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;__kernel_clock_gettime&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;.cfi_startproc&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="cm"&gt;/* Check for supported clock IDs */&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nf"&gt;cmpwi&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="no"&gt;cr0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="no"&gt;r3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="no"&gt;CLOCK_REALTIME&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nf"&gt;cmpwi&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="no"&gt;cr1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="no"&gt;r3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="no"&gt;CLOCK_MONOTONIC&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nf"&gt;cror&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;cr0&lt;/span&gt;&lt;span class="p"&gt;*&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="err"&gt;+&lt;/span&gt;&lt;span class="no"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="no"&gt;cr0&lt;/span&gt;&lt;span class="p"&gt;*&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="err"&gt;+&lt;/span&gt;&lt;span class="no"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="no"&gt;cr1&lt;/span&gt;&lt;span class="p"&gt;*&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="err"&gt;+&lt;/span&gt;&lt;span class="no"&gt;eq&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nf"&gt;bne&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;cr0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;99&lt;/span&gt;&lt;span class="no"&gt;f&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="cm"&gt;/* [snip] */&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="cm"&gt;/*&lt;/span&gt;
&lt;span class="cm"&gt;     * syscall fallback&lt;/span&gt;
&lt;span class="cm"&gt;     */&lt;/span&gt;
&lt;span class="err"&gt;99:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nf"&gt;li&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="no"&gt;r0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="no"&gt;__NR_clock_gettime&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nf"&gt;sc&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nf"&gt;blr&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For those not familiar, this couldn't be more simple. The start checks
to see if it is a clock id that the &lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt; can handle and if not it jumps
to the 99 label. From here simply load the syscall number, jump to the
kernel and branch to link register aka 'return'.  In this case the
'return' statement would return to the userspace code which called the
&lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt; function.&lt;/p&gt;
&lt;p&gt;Wait, having the &lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt; calling into the kernel call gets us the wrong
result? Or course it should, vdsotest is assuming a C ABI with return
values and errno but the kernel doesn't do that, the kernel ABI is
different. How does this even work on x86? Ohhhhh vdsotest does &lt;sup id="fnref:2"&gt;&lt;a class="footnote-ref" href="#fn:2"&gt;2&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;inline&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;record_syscall_result&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;syscall_result&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;                     &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sr_ret&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sr_errno&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="cm"&gt;/* Calling the vDSO directly instead of through libc can lead to:&lt;/span&gt;
&lt;span class="cm"&gt;     * - The vDSO code punts to the kernel (e.g. unrecognized clock id).&lt;/span&gt;
&lt;span class="cm"&gt;     * - The kernel returns an error (e.g. -22 (-EINVAL))&lt;/span&gt;
&lt;span class="cm"&gt;     * So we need to recognize this situation and fix things up.&lt;/span&gt;
&lt;span class="cm"&gt;     * Fortunately we&amp;#39;re dealing only with syscalls that return -ve values&lt;/span&gt;
&lt;span class="cm"&gt;     * on error.&lt;/span&gt;
&lt;span class="cm"&gt;     */&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sr_ret&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sr_errno&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;sr_errno&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;sr_ret&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;sr_ret&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;-1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;syscall_result&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sr_ret&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sr_ret&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sr_errno&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sr_errno&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That little hack isn't working on PowerPC and here's why:&lt;/p&gt;
&lt;p&gt;The kernel puts the return value in the ABI specified return register
(r3) and uses a condition register bit (condition register field 0, SO
bit), so unlike x86 on error the return value isn't negative. To make
matters worse, the condition register is very difficult to access from
C. Depending on your definition of 'access from C' you might consider
it impossible, a fixup like that would be impossible.&lt;/p&gt;
&lt;h3&gt;Lessons learnt&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt; supplied functions aren't quite the same as their libc
counterparts. Unless you have very good reason, and to be fair,
vdsotest does have a very good reason, always access the &lt;abbr title="Virtual Dynamically linked Shared Objects"&gt;vDSO&lt;/abbr&gt; through
libc&lt;/li&gt;
&lt;li&gt;Kernel interfaces aren't C interfaces, yep, they're close but they
  aren't the same&lt;/li&gt;
&lt;li&gt;22 is in fact EINVAL&lt;/li&gt;
&lt;li&gt;Different architectures are... Different!&lt;/li&gt;
&lt;li&gt;Variety is the spice of life&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;P.S I have a hacky patch waiting review&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;arch/powerpc/kernel/vdso64/gettimeofday.S&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:2"&gt;
&lt;p&gt;src/vdsotest.h&amp;#160;&lt;a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Cyril Bur</dc:creator><pubDate>Fri, 24 Jun 2016 16:30:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-06-24:/blog/2016/06/24/kernel-interfaces-and-vdso-test/</guid><category>Development</category><category>kernel</category><category>asm</category><category>vdso</category></item><item><title>Introducing snowpatch: continuous integration for patches</title><link>https://sthbrx.github.io/blog/2016/06/15/introducing-snowpatch-continuous-integration-for-patches/</link><description>&lt;p&gt;Continuous integration has changed the way we develop software.  The ability to make a code change and be notified quickly and automatically whether or not it works allows for faster iteration and higher quality.  These processes and technologies allow products to quickly and consistently release new versions, driving continuous improvement to their users.  For a web app, it's all pretty simple: write some tests, someone makes a pull request, you build it and run the tests.  Tools like GitHub, Travis CI and Jenkins have made this process simple and efficient.&lt;/p&gt;
&lt;p&gt;Let's throw some spanners in the works.  What if instead of a desktop or web application, you're dealing with an operating system?  What if your tests can only be run when booted on physical hardware? What if instead of something like a GitHub pull request, code changes were sent as plain-text emails to a mailing list?  What if you didn't have control the development of this project, and you had to work with an existing, open community?&lt;/p&gt;
&lt;p&gt;These are some of the problems faced by the Linux kernel, and many other open source projects.  Mailing lists, along with tools like &lt;code&gt;git send-email&lt;/code&gt;, have become core development infrastructure for many large open source projects.  The idea of sending code via a plain-text email is simple and well-defined, not reliant on a proprietary service, and uses universal, well-defined technology.  It does have shortcomings, though.  How do you take a plain-text patch, which was sent as an email to a mailing list, and accomplish the continuous integration possibilities other tools have trivially?&lt;/p&gt;
&lt;p&gt;Out of this problem birthed &lt;a href="https://github.com/ruscur/snowpatch"&gt;snowpatch&lt;/a&gt;, a continuous integration tool designed to enable these practices for projects that use mailing lists and plain-text patches.  By taking patch metadata organised by &lt;a href="https://patchwork.ozlabs.org"&gt;Patchwork&lt;/a&gt;, performing a number of &lt;code&gt;git&lt;/code&gt; operations and shipping them off to &lt;a href="https://jenkins.io"&gt;Jenkins&lt;/a&gt;, snowpatch can enable continuous integration for any mailing list-based project.  At IBM &lt;a href="https://ozlabs.org"&gt;OzLabs&lt;/a&gt;, we're using snowpatch to automatically test new patches for &lt;a href="https://github.com/linuxppc/linux/"&gt;Linux on POWER&lt;/a&gt;, &lt;a href="https://github.com/open-power/skiboot.git"&gt;skiboot&lt;/a&gt;, snowpatch itself, and more.&lt;/p&gt;
&lt;p&gt;snowpatch is written in &lt;a href="https://rust-lang.org"&gt;Rust&lt;/a&gt;, an exciting new systems programming language with a focus on speed and safety.  Rust's amazing software ecosystem, enabled by its package manager &lt;a href="https://crates.io"&gt;Cargo&lt;/a&gt;, made development of snowpatch a breeze.  Using Rust has been a lot of fun, along with the practical benefits of (in our experience) faster development, and confidence in the runtime stability of our code.  It's still a young language, but it's quickly growing and has an amazing community that has always been happy to help.&lt;/p&gt;
&lt;p&gt;We still have a lot of ideas for snowpatch that haven't been implemented yet.  Once we've tested a patch and sent the results back to a patchwork instance, what if the project's maintainer (or a trusted contributor) could manually trigger some more intensive tests?  How would we handle it if the traffic on the mailing list of a project is too fast for us to test?  If we were running snowpatch on multiple machines on the same project, how would we avoid duplicating effort?  These are unsolved problems, and if you'd like to help us with these or anything else you think would be good for snowpatch, we take contributions and ideas via our mailing list, which you can subscribe to &lt;a href="https://lists.ozlabs.org/listinfo/snowpatch"&gt;here&lt;/a&gt;.  For more details, view our documentation on &lt;a href="https://github.com/ruscur/snowpatch"&gt;GitHub&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Thanks for taking your time to learn a bit about snowpatch.  In future, we'll be talking about how we tie all these technologies together to build a continuous integration workflow for the Linux kernel and OpenPOWER firmware.  Watch this space!&lt;/p&gt;
&lt;p&gt;&lt;em&gt;This article was originally posted on &lt;a href="https://developer.ibm.com/open/"&gt;IBM developerWorks Open&lt;/a&gt;.  Check that out for more open source from IBM, and look out for more content in their &lt;a href="https://developer.ibm.com/open/snowpatch"&gt;snowpatch&lt;/a&gt; section.&lt;/em&gt;&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Russell Currey</dc:creator><pubDate>Wed, 15 Jun 2016 15:33:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-06-15:/blog/2016/06/15/introducing-snowpatch-continuous-integration-for-patches/</guid><category>snowpatch</category><category>snowpatch</category><category>development</category><category>tools</category></item><item><title>Interning at Ozlabs</title><link>https://sthbrx.github.io/blog/2016/06/08/interning-at-ozlabs/</link><description>&lt;p&gt;I am sadly coming to the end of my six(ish) month internship with Ozlabs (funded by &lt;a href="https://www.acs.org.au"&gt;ACS&lt;/a&gt;). So here I am writing about my experience in the hopes that future prospective interns can read about how they should come and work with the previously dubbed Linux Gods.&lt;/p&gt;
&lt;h3&gt;What is your background?&lt;/h3&gt;
&lt;p&gt;Despite embracing being a nerd at school, my opinion of computers prior to starting my Engineering degree was that they were boring and for geeky boys who didn't want to interact with the 'real' world. However when having to choose a specialisation of Engineering I was drawn towards Computer Systems as everything else seemed obvious * but Computer Systems was this great mystical unknown. &lt;/p&gt;
&lt;p&gt;Fast forward three years, and I had seen glimpses into the workings of this magical computer world. I had learnt about transistors, logic gates and opamps; I had designed circuits that actually worked; and I had bashed my head against a wall trying to find obscure bugs. I had dabbled in a range of languages from the low levels of VHDL and embedded C, to the abstract world of Python and Java and delved into the obscure world of declarative prologs and relational reinforcement learning. Now it was time to solidify some of these concepts and get some experience under my belt so I could feel less like a monkey bashing random keys on my keyboard. Enter Ozlabs!&lt;/p&gt;
&lt;h3&gt;What did you do at Ozlabs?&lt;/h3&gt;
&lt;p&gt;After being handed a nice laptop and the root passwords, I faced the inevitable battle of getting everything setup. With the help of my mentor, the prestigious &lt;a href="http://mpe.github.io/"&gt;Michael Ellerman&lt;/a&gt;, and various other Ozlabs residents I picked off some low hanging fruit such as removing unused code and tidying up a few things. This allowed me to get familiar with the open-source workflow, the kernel building process, IRC, do more with Git then just push and pull, and &lt;strong&gt;finally&lt;/strong&gt; come face-to-face with the seemingly impossible: Vim and virtual machines.&lt;/p&gt;
&lt;p&gt;I then got to learn about Transactional Memory (TM) - a way of making a bunch of instructions on one processor appear to be one atomic operation to other processors. I took some old TM tests from Mikey and checked that they did indeed pass and fail when they were supposed to and refurbished them a little, learning how to run kernel self-tests and a bit about powerpc assembly along the way.&lt;/p&gt;
&lt;p&gt;Eventually my fear of shell scripts was no match for my desire to be able to build and install a kernel with one command and so I finally got around to writing a build script. Accidentally rebooting a bare-metal machine instead of my VM running on it may have had a significant contribution to this...&lt;/p&gt;
&lt;p&gt;The next interesting task I got to tackle was to implement a virtual memory dump that other architectures like x86 have, so we can see how the pages in memory are laid out along with information about these pages. This involved understanding x86's implementation and relating that to POWER's memory management. At Uni I never quite understood the fuss about pages and virtual memory and so it was great to be able to build up an appreciation and play around with page tables, virtual to real addresses, and hashtable.&lt;/p&gt;
&lt;p&gt;I then moved onto &lt;a href="https://sthbrx.github.io/blog/2016/05/13/srop-mitigation/"&gt;SROP mitigation&lt;/a&gt;! After a lot of reading and re-reading, I decided to first understand how to use SROP to make an exploit on POWER which meant some assembly, diving into the signal code and finally meeting and spending time with GDB.  Once again I had x86 code to port over to POWER, the main issue being making sure that I didn't break existing things - aka hours and hours of running the kernel self-tests and the Linux Test Project tests and some more scripting, with the help of &lt;a href="http://blog.christophersmart.com/"&gt;Chris Smart&lt;/a&gt;, to collate the results.&lt;/p&gt;
&lt;p&gt;You can judge all my submitted patches &lt;a href="https://patchwork.ozlabs.org/project/linuxppc-dev/list/?submitter=67695&amp;amp;state=*"&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;h3&gt;What was your overall experience like at Ozlabs?&lt;/h3&gt;
&lt;p&gt;I moved to Canberra shortly after finishing exams and so hadn't had the time to ponder expectations of Ozlabs. Everyone was super friendly and despite being, not just the only female but, the only kiwi among a whoooole lot of Aussies I experienced a distinct lack of discrimination (apart from a bit of banter about accents).&lt;/p&gt;
&lt;p&gt;Could I wear my normal clothes (and not stuffy business clothes)? Check. Did I get to work on interesting things? Check. Could I do my work without having to go through lots of unnecessary hoops and what not? Check. Could I develop my own workflow and learn all the things? Check. Did I get to delve into a few different areas? Check. Was I surrounded by super smart people who were willing to help me learn? Check. &lt;/p&gt;
&lt;p&gt;All in all, I have had a great time here, learnt so much and you should definitely come and work at Ozlabs! Hopefully you'll see me back on this blog in a few months :)&lt;/p&gt;
&lt;p&gt;* &lt;em&gt;My pre-university, perhaps somewhat naiive, opinion: Civil and Mechanical is just physics. Chemical and Materials is just chemistry. Electrical seems interesting but who wants to work with power lines? Biomedical is just math and biology. Software is just abstract high level nonsense. But how a computer works?? That is some magical stuff.&lt;/em&gt;&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Rashmica Gupta</dc:creator><pubDate>Wed, 08 Jun 2016 22:22:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-06-08:/blog/2016/06/08/interning-at-ozlabs/</guid><category>Education</category><category>intern</category><category>work experience</category></item><item><title>Using the Atom editor for Linux kernel development</title><link>https://sthbrx.github.io/blog/2016/06/07/using-the-atom-editor-for-linux-kernel-development/</link><description>&lt;p&gt;&lt;a href="https://atom.io"&gt;Atom&lt;/a&gt; is a text editor.  It's new, it's shiny, and it has a lot of good and bad sides.  I work in a lab full of kernel developers, and in the kernel, there are no IDEs.  There's no real metadata you can get out of your compiler (given the kernel isn't very clang friendly), there's certainly nothing like that you can get out of your build system, so "plain old" text editors reign supreme.  It's a vim or Emacs show.&lt;/p&gt;
&lt;p&gt;And so Atom comes along.  Unlike other shiny new text editors to emerge in the past 10 or so years, it's open source (unlike Sublime Text), it works well on Linux, and it's very configurable.  When it first came out, Atom was an absolute mess.  There was a noticeable delay whenever you typed a key.  That has gone, but the sour impression that comes from replacing a native application with a web browser in a frame remains.&lt;/p&gt;
&lt;p&gt;Like the curious person I am, I'm always trying out new things to see if they're any good.  I'm not particularly tied to any editor; I prefer modal editing, but I'm no vim wizard.  I eventually settled on using Emacs with evil-mode (which I assumed would make both Emacs and vim people like me, but the opposite happened), which was decent.  It was configurable, it was good, but it had issues.&lt;/p&gt;
&lt;p&gt;&lt;img alt="The Atom editor" src="/images/ruscur/36lOiMT.png"&gt;&lt;/p&gt;
&lt;p&gt;So, let's have a look at how Atom stacks up for low-level work.  First of all, it's X only.  You wouldn't use it to change one line of a file in /etc/, and a lot of kernel developers only edit code inside a terminal emulator.  Most vim people do this since gvim is a bit wonky, and Emacs people can double-dip; using Emacs without X for small things and Emacs with X for programming.  You don't want to do that with Atom, if nothing else because of its slow startup time.&lt;/p&gt;
&lt;p&gt;Now let's look at configurability.  In my opinion, no editor will ever match the level of configurability of Emacs, however the barrier to entry is much lower here.  Atom has lots of options exposed in a config file, and you can set them there or you can use an equivalent GUI.  In addition, a perk of being a browser in a frame is that you can customise a lot of UI things with CSS, for those inclined.  Overall, I'd say Emacs &amp;gt; Atom &amp;gt; vim here, but for a newbie, it's probably Atom &amp;gt; Emacs &amp;gt; vim.&lt;/p&gt;
&lt;p&gt;Okay, package management.  Atom is the clear winner here.  The package repository is very easy to use, for users and developers.  I wrote my own package, typed &lt;code&gt;apm publish&lt;/code&gt; and within a minute a friend could install it.  For kernel development though, you don't really need to install anything, Atom is pretty batteries-included.  This includes good syntax highlighting, ctags support, and a few themes.  In this respect, Atom feels like an editor that was created this century.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Atom's inbuilt package management" src="/images/ruscur/DAx7GqD.png"&gt;&lt;/p&gt;
&lt;p&gt;What about actually editing text?  Well, I only use modal editing, and Atom is very far from being the best vim.  I think evil-mode in Emacs is the best vim, followed closely by vim itself.  Atom has a vim-mode, and it's fine for insert/normal/visual mode, but anything involving a : is a no-go.  There's a plugin that's entirely useless.  If I tried to do a replacement with :s, Atom would lock up &lt;em&gt;and&lt;/em&gt; fail to replace the text.  vim replaced thousands of occurrences with in a second.  Other than that, Atom's pretty good.  I can move around pretty much just as well as I could in vim or Emacs, but not quite.  Also, it support ligatures!  The first kernel-usable editor that does.&lt;/p&gt;
&lt;p&gt;Autocompletions feel very good in Atom.  It completes within a local scope automatically, without any knowledge of the type of file you're working on.  As far as intelligence goes, Atom's support for tags outside of ctags is very lacking, and ctags is stupid.  Go-to definition &lt;em&gt;sometimes&lt;/em&gt; works, but it lags when dealing with something as big as the Linux kernel.  Return-from definition is very good, though.  Another downside is that it can complete from any open buffer, which is a huge problem if you're writing Rust in one tab and C in the other.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Atom's fuzzy file matching is pretty good" src="/images/ruscur/0PRiIUS.png"&gt;&lt;/p&gt;
&lt;p&gt;An experience I've had with Atom that I haven't had with other editors is actually writing a plugin.  It was really easy, mostly because I stole a lot of it from an existing plugin, but it was easy.  I wrote a syntax highlighting package for POWER assembly, which was much more fighting with regular expressions than it was fighting with anything in Atom.  Once I had it working, it was very easy to publish; just push to GitHub and run a command.&lt;/p&gt;
&lt;p&gt;Sometimes, Atom can get too clever for its own good.  For some completely insane reason, it automatically "fixes" whitespace in every file you open, leading to a huge amount of git changes you didn't intend.  That's easy to disable, but I don't want my editor doing that, it'd be much better if it highlighted whitespace it didn't like by default, like you can get vim and Emacs to do.  For an editor designed around git, I can't comprehend that decision.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Atom can also fuzzy match its commands" src="/images/ruscur/arbWXHx.png"&gt;&lt;/p&gt;
&lt;p&gt;Speaking of git, the editor pretty much has everything you'd expect for an editor written at GitHub.  The sidebar shows you what lines you've added, removed and modified, and the gutter shows you what branch you're on and how much you've changed all-up.  There's no in-built support for doing git things inside the editor, but there's a package for it.  It's pretty nice to get something "for free" that you'd have to tinker with in other editors.&lt;/p&gt;
&lt;p&gt;Overall, Atom has come a long way and still has a long way to go.  I've been using it for a few weeks and I'll continue to use it.  I'll encourage new developers to use it, but it needs to be better for experienced programmers who are used to their current workflow to consider switching.  If you're in the market for a new editor, Atom might just be for you.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Russell Currey</dc:creator><pubDate>Tue, 07 Jun 2016 17:03:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-06-07:/blog/2016/06/07/using-the-atom-editor-for-linux-kernel-development/</guid><category>Development</category><category>education</category><category>kernel</category><category>development</category><category>tools</category></item><item><title>SROP Mitigation</title><link>https://sthbrx.github.io/blog/2016/05/13/srop-mitigation/</link><description>&lt;h2&gt;What is SROP?&lt;/h2&gt;
&lt;p&gt;Sigreturn Oriented Programming - a general technique that can be used as an exploit, or as a backdoor to exploit another vulnerability.&lt;/p&gt;
&lt;h2&gt;Okay, but what is it?&lt;/h2&gt;
&lt;p&gt;Yeah... Let me take you through some relevant background info, where I skimp on the details and give you the general picture.&lt;/p&gt;
&lt;p&gt;In Linux, software interrupts are called signals. More about signals &lt;a href="http://www.thegeekstuff.com/2012/03/linux-signals-fundamentals/"&gt;here&lt;/a&gt;! Generally a signal will convey some information from the kernel and so most signals will have a specific signal handler (some code that deals with the signal) setup.&lt;/p&gt;
&lt;p&gt;Signals are asynchronous - ie they can be sent to a process/program at anytime. When a signal arrives for a process, the kernel suspends the process. The kernel then saves the 'context' of the process - all the general purpose registers (GPRs), the stack pointer, the next-instruction pointer etc - into a structure called a 'sigframe'. The sigframe is stored on the stack, and then the kernel runs the signal handler. At the very end of the signal handler, it calls a special system call called 'sigreturn' - indicating to the kernel that the signal has been dealt with. The kernel then grabs the sigframe from the stack, restores the process's context and resumes the execution of the process.&lt;/p&gt;
&lt;p&gt;This is the rough mental picture you should have:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Double Format" src="/images/rashmica/picture.png"&gt;&lt;/p&gt;
&lt;h2&gt;Okay... but you still haven't explained what SROP is..?&lt;/h2&gt;
&lt;p&gt;Well, if you insist...&lt;/p&gt;
&lt;p&gt;The above process was designed so that the kernel does not need to keep track of what signals it has delivered. The kernel assumes that the sigframe it takes off the stack was legitimately put there by the kernel because of a signal. This is where we can trick the kernel!&lt;/p&gt;
&lt;p&gt;If we can construct a fake sigframe, put it on the stack, and call sigreturn, the kernel will assume that the sigframe is one it put there before and will load the contents of the fake context into the CPU's registers and 'resume' execution from where the fake sigframe tells it to. And that is what SROP is!&lt;/p&gt;
&lt;h2&gt;Well that sounds cool, show me!&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Firstly&lt;/strong&gt; we have to set up a (valid) sigframe:&lt;/p&gt;
&lt;p&gt;By valid sigframe, I mean a sigframe that the kernel will not reject. Luckily most architectures only examine a few parts of the sigframe to determine the validity of it. Unluckily, you will have to dive into the source code to find out which parts of the sigframe you need to set up for your architecture. Have a look in the function which deals with the syscall sigreturn (probably something like sys_sigreturn() ).&lt;/p&gt;
&lt;p&gt;For a real time signal on a little endian powerpc 64bit machine, the sigframe looks something like this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;rt_sigframe&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;ucontext&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;uc&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;_unused&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tramp&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;TRAMP_SIZE&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;siginfo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__user&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;pinfo&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__user&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;puc&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;siginfo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;info&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;user_cookie&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="cm"&gt;/* New 64 bit little-endian ABI allows redzone of 512 bytes below sp */&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;abigap&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;USER_REDZONE_SIZE&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__attribute__&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;aligned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;)));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The most important part of the sigframe is the context or ucontext as this contains all the register values that will be written into the CPU's registers when the kernel loads in the sigframe. To minimise potential issues we can copy valid values from the current GPRs into our fake ucontext:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;register&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;r1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;asm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;r1&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;register&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;r13&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;asm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;r13&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;ucontext&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;};&lt;/span&gt;

&lt;span class="cm"&gt;/* We need a system thread id so copy the one from this process */&lt;/span&gt;
&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uc_mcontext&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;gp_regs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;PT_R13&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;r13&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="cm"&gt;/*  Set the context&amp;#39;s stack pointer to where the current stack pointer is pointing */&lt;/span&gt;
&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uc_mcontext&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;gp_regs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;PT_R1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;r1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;We also need to tell the kernel where to resume execution from. As this is just a test to see if we can successfully get the kernel to resume execution from a fake sigframe we will just point it to a function that prints out some text.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cm"&gt;/* Set the next instruction pointer (NIP) to the code that we want executed */&lt;/span&gt;
&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uc_mcontext&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;gp_regs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;PT_NIP&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;test_function&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For some reason the sys_rt_sigreturn() on little endian powerpc 64bit checks the endianess bit of the ucontext's MSR register, so we need to set that:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cm"&gt;/* Set MSR bit if LE */&lt;/span&gt;
&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uc_mcontext&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;gp_regs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;PT_MSR&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x01&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Fun fact: not doing this or setting it to 0 results in the CPU switching from little endian to big endian! For a powerpc machine sys_rt_sigreturn() only examines ucontext, so we do not need to set up a full sigframe.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Secondly&lt;/strong&gt; we have to put it on the stack:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cm"&gt;/* Set current stack pointer to our fake context */&lt;/span&gt;
&lt;span class="n"&gt;r1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Thirdly&lt;/strong&gt;, we call sigreturn:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cm"&gt;/* Syscall - NR_rt_sigreturn */&lt;/span&gt;
&lt;span class="k"&gt;asm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;li 0, 172&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;asm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;sc&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;When the kernel receives the sigreturn call, it looks at the userspace stack pointer for the ucontext and loads this in. As we have put valid values in the ucontext, the kernel assumes that this is a valid sigframe that it set up earlier and loads the contents of the ucontext in the CPU's registers "and resumes" execution of the process from the address we pointed the NIP to.&lt;/p&gt;
&lt;p&gt;Obviously, you need something worth executing at this address, but sadly that next part is not in my job description. This is a nice gateway into the kernel though and would pair nicely with another kernel vulnerability.  If you are interested in some more in depth examples, have a read of &lt;a href="http://www.cs.vu.nl/~herbertb/papers/srop_sp14.pdf"&gt;this&lt;/a&gt; paper.&lt;/p&gt;
&lt;h2&gt;So how can we mitigate this?&lt;/h2&gt;
&lt;p&gt;Well, I'm glad you asked. We need some way of distinguishing between sigframes that were put there legitimately by the kernel and 'fake' sigframes. The current idea that is being thrown around is cookies, and you can see the x86 discussion &lt;a href="https://lkml.org/lkml/2016/3/29/788"&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The proposed solution is to give every sighand struct a randomly generated value. When the kernel constructs a sigframe for a process, it stores a 'cookie' with the sigframe. The cookie is a hash of the cookie's location and the random value stored in the sighand struct for the process. When the kernel receives a sigreturn, it hashes the location where the cookie should be with the randomly generated number in sighand struct - if this matches the cookie, the cookie is zeroed,  the sigframe is valid and the kernel will restore this context.  If the cookies do not match, the sigframe is not restored.&lt;/p&gt;
&lt;p&gt;Potential issues:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Multithreading: Originally the random number was suggested to be stored in the task struct. However, this would break multi-threaded applications as every thread has its own task struct. As the sighand struct is shared by threads, this should not adversely affect multithreaded applications.&lt;/li&gt;
&lt;li&gt;Cookie location: At first I put the cookie on top of the sigframe. However some code in userspace assumed that all the space between the signal handler and the sigframe  was essentially up for grabs and would zero the cookie before I could read the cookie value. Putting the cookie below the sigframe was also a no-go due to the ABI-gap (a gap below the stack pointer that signal code cannot touch) being a part of the sigframe. Putting the cookie inside the sigframe, just above the ABI gap has been fine with all the tests I have run so far!&lt;/li&gt;
&lt;li&gt;Movement of sigframe: If you move the sigframe on the stack, the cookie value will no longer be valid... I don't think that this is something that you should be doing, and have not yet come across a scenario that does this. &lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For a more in-depth explanation of SROP, click &lt;a href="https://lwn.net/Articles/676803/"&gt;here&lt;/a&gt;.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Rashmica Gupta</dc:creator><pubDate>Fri, 13 May 2016 22:22:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-05-13:/blog/2016/05/13/srop-mitigation/</guid><category>Development</category><category>SROP</category><category>mitigation</category><category>kernel</category></item><item><title>Tell Me About Petitboot</title><link>https://sthbrx.github.io/blog/2016/05/13/tell-me-about-petitboot/</link><description>&lt;p&gt;A Google search for 'Petitboot' brings up results from a number of places, some describing its use on &lt;a href="https://www.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabppetitboot.htm"&gt;POWER servers&lt;/a&gt;, others talking about how to use it on the &lt;a href="http://jk.ozlabs.org/projects/petitboot/"&gt;PS3&lt;/a&gt;, in varying levels of detail. I tend to get a lot of general questions about Petitboot and its behaviour, and have had a few requests for a broad "Welcome to Petitboot" blog, suggesting that existing docs deal with more specific topics.. or that people just aren't reading them :)&lt;/p&gt;
&lt;p&gt;So today we're going to take a bit of a crash course in the what, why, and possibly how of Petitboot. I won't delve too much into technical details, and this will be focussed on Petitboot in POWER land since that's where I spend most of my time.
Here we go!&lt;/p&gt;
&lt;h2&gt;What&lt;/h2&gt;
&lt;p&gt;Aside from a whole lot of firmware and kernel logs flying past, the first thing you'll see when booting a POWER server&lt;sup&gt;In OPAL mode at least...&lt;/sup&gt; is Petitboot's main menu:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Main Menu" src="/images/sammj/mainmenu.png"&gt;&lt;/p&gt;
&lt;p&gt;Petitboot is the first interact-able component a user will see. The word 'BIOS' gets thrown around a lot when discussing this area, but that is wrong, and the people using that word are wrong.&lt;/p&gt;
&lt;p&gt;When the OPAL firmware layer &lt;a href="https://github.com/open-power/skiboot"&gt;Skiboot&lt;/a&gt; has finished its own set up, it loads a certain binary (stored on the BMC) into memory and jumps into it. This could hypothetically be anything, but for any POWER server right now it is 'Skiroot'. Skiroot is a full Linux kernel and userspace, which runs Petitboot. People often say Petitboot when they mean Skiroot - technically Petitboot is the server and UI processes that happen to run within Skiroot, and Skiroot is the full kernel and rootfs package. This is more obvious when you look at the &lt;a href="https://github.com/open-power/op-build"&gt;op-build&lt;/a&gt; project - Petitboot is a package built as part of the kernel and rootfs created by Buildroot.&lt;/p&gt;
&lt;p&gt;Petitboot is made of two major parts - the UI processes (one for each available console), and the 'discover' server. The discover server updates the UI processes, manages and scans available disks and network devices, and performs the actual booting of host operating systems. The UI, running in ncurses, displays these options, allows the user to edit boot options and system configuration, and tells the server which boot option to kexec.&lt;/p&gt;
&lt;h2&gt;Why&lt;/h2&gt;
&lt;p&gt;The 'why' delves into some of the major architectural differences between a POWER machine and your average x86 machine which, as always, could spread over several blog posts and/or a textbook.&lt;/p&gt;
&lt;p&gt;POWER processors don't boot themselves, instead the attached Baseboard Management Controller (BMC) does a lot of low-level poking that gets the primary processor into a state where it is ready to execute instructions. PowerVM systems would then jump directly into the PHYP hypervisor - any subsequent OS, be it AIX or Linux, would then run as a 'partition' under this hypervisor.&lt;/p&gt;
&lt;p&gt;What we all really want though is to run Linux directly on the hardware, which meant a new boot process would have to be thought up while still maintaining compatibility with PowerVM so systems could be booted in either mode. Thus became OPAL, and its implementation &lt;a href="https://github.com/open-power/skiboot"&gt;Skiboot&lt;/a&gt;. Skipping over &lt;em&gt;so much&lt;/em&gt; detail, the system ends up booting into Skiboot which acts as our firmware layer. Skiboot isn't interactive and doesn't really care about things like disks, so it loads another binary into memory and executes it - Skiroot!&lt;/p&gt;
&lt;p&gt;Skiroot exists as an alternative to writing a whole new bootloader just for POWER in OPAL mode, or going through the effort to port an existing bootloader to understand the specifics of POWER. Why do all that when Linux already exists and already knows how to handle disks, network interfaces, and a thousand other things? Not to mention that when Linux gains support for fancy new devices so do we, and adding new features of our own is as simple as writing your average Linux program.&lt;/p&gt;
&lt;p&gt;Skiroot itself (not including Skiboot) is roughly comparable to UEFI, or at least much more so than legacy BIOS implementations. But whereas UEFI tends to be a monolithic blob of fairly platform-specific code (in practice), Skiroot is simply a small Linux environment that anyone could put together with Buildroot.&lt;/p&gt;
&lt;p&gt;A much better insight into the development and motivation behind Skiroot and Petitboot is available in Jeremy's &lt;a href="https://www.youtube.com/watch?v=oxmMJMibZQ8"&gt;LCA2013 talk&lt;/a&gt;&lt;/p&gt;
&lt;h2&gt;Back to Petitboot&lt;/h2&gt;
&lt;p&gt;Petitboot is the part of the 'bootloader' that &lt;em&gt;did&lt;/em&gt; need to be written, because users probably wouldn't be too thrilled if they had to manually mount disks and kexec their kernels every time they booted the machine. The Petitboot server process mounts any available disk devices and scans them for available operating systems. That's not to say that it scans the entire disk, because otherwise you could be waiting for quite some time, but rather it looks in a list of common locations for bootloader configuration files. This is handy because it means the operating system doesn't need to have any knowledge of Petitboot - it just uses its usual install scripts and Petitboot reads them to know what is available.
At the same time Petitboot makes PXE requests on configured network interfaces so we can netboot, and allows these various sources to be given relative priorities for auto-boot, plus a number of other ways to specially configure booting behaviour.&lt;/p&gt;
&lt;p&gt;A particularly neat feature of existing in a Linux environment is the ability to easily recover from boot problems; whereas on another system you might need to use a Live CD to fix a misconfiguration or recover a broken filesystem, in Skiroot you can just drop to the shell and fix the issue right there.&lt;/p&gt;
&lt;p&gt;In summary, Petitboot/Skiroot is a small but capable Linux environment that every OPAL POWER machine boots into, gathering up all the various local and remote boot possibilities, and presenting them to you in a state-of-the-art ncurses interface. Petitboot updates all the time, and if you come across a feature that you think Petitboot is missing, patches are very welcome at &lt;a href="mailto:petitboot@lists.ozlabs.org"&gt;petitboot@lists.ozlabs.org&lt;/a&gt; (or hassle me on IRC)!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Fri, 13 May 2016 15:23:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-05-13:/blog/2016/05/13/tell-me-about-petitboot/</guid><category>Petitboot</category><category>petitboot</category><category>goodposts</category><category>realcontent</category></item><item><title>Doubles in hex and why Kernel addresses ~= -2</title><link>https://sthbrx.github.io/blog/2016/05/12/doubles-in-hex-and-why-kernel-addresses-2/</link><description>&lt;p&gt;It started off a regular Wednesday morning when I hear from my desk a colleague
muttering about doubles and their hex representation. "But that doesn't look
right", "How do I read this as a float", and "&lt;del&gt;redacted&lt;/del&gt; you're the engineer,
you do it". My interest piqued, I headed over to his desk to enquire about the
great un-solvable mystery of the double and its hex representation. The number
which would consume me for the rest of the morning: 0xc00000001568fba0.&lt;/p&gt;
&lt;h2&gt;That's a Perfectly Valid hex Number!&lt;/h2&gt;
&lt;p&gt;I hear you say. And you're right, if we were to treat this as a long it
would simply be 13835058055641365408 (or -4611686018068186208 if we assume
a signed value). But we happen to know that this particular piece of data which
we have printed is supposed to represent a double (-2 to be precise). "Well
print it as a double" I hear from the back, and once again we &lt;em&gt;should&lt;/em&gt; all know
that this can be achieved rather easily by using the %f/%e/%g specifiers in our
print statement. The only problem is that in kernel land (where we use printk)
we are limited to printing fixed point numbers, hence why our only &lt;em&gt;easy&lt;/em&gt;
option was to print our double in it's raw hex format.&lt;/p&gt;
&lt;p&gt;This is the point where we all think back to that university course where
number representations were covered in depth, and terms like 'mantissa' and
'exponent' surface in our minds. Of course as we rack our brains we realise
there's no way that we're going to remember exactly how a double is represented
and bring up the &lt;a href="https://en.wikipedia.org/wiki/Double-precision_floating-point_format"&gt;IEEE 754 Wikipedia page&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;What is a Double?&lt;/h2&gt;
&lt;p&gt;Taking a step back for a second, a double (or a double-precision floating-point)
is a number format used to represent floating-point numbers (those with a
decimal component). They are made up of a sign bit, an exponent and a fraction
(or mantissa):&lt;/p&gt;
&lt;p&gt;&lt;img alt="Double Format" src="/images/surajjs/doubles_in_hex/double.png"&gt;&lt;/p&gt;
&lt;p&gt;Where the number they represent is defined by:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Double Formula" src="/images/surajjs/doubles_in_hex/formula.png"&gt;&lt;/p&gt;
&lt;p&gt;So this means that a 1 in the MSB (sign bit) represents a negative number, and
we have some decimal component (the fraction) which we multiply by some power
of 2 (as determined by the exponent) to get our value.&lt;/p&gt;
&lt;h2&gt;Alright, so what's 0xc00000001568fba0?&lt;/h2&gt;
&lt;p&gt;The reason we're all here to be begin with, so what's 0xc00000001568fba0 if we
treat it as a double? We can first split it into the three components:&lt;/p&gt;
&lt;h5&gt;0xc00000001568fba0:&lt;/h5&gt;
&lt;p&gt;Sign bit: 1             -&amp;gt; Negative&lt;br&gt;
Exponent: 0x400         -&amp;gt; 2&lt;sup&gt;(1024 - 1023)&lt;/sup&gt;&lt;br&gt;
Fraction: 0x1568fba0    -&amp;gt; 1.&lt;em&gt;something&lt;/em&gt;&lt;br&gt;&lt;/p&gt;
&lt;p&gt;And then use the formula above to get our number:&lt;/p&gt;
&lt;p&gt;(-1)&lt;sup&gt;&lt;strong&gt;1&lt;/strong&gt;&lt;/sup&gt; x 1.&lt;em&gt;&lt;strong&gt;something&lt;/strong&gt;&lt;/em&gt; x 2&lt;sup&gt;&lt;strong&gt;(1024 - 1023)&lt;/strong&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;But there's a much easier way!&lt;/strong&gt; Just write ourselves a little program in
userspace (where we are capable of printing floats) and we can save ourselves
&lt;em&gt;most&lt;/em&gt; of the trouble.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;stdio.h&amp;gt;&lt;/span&gt;

&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;val&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xc00000001568fba0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;val: %lf&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="kt"&gt;double&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;val&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;So all we're doing is taking our hex value and storing it in a long (val), then
getting a pointer to val, casting it to a double pointer, and dereferencing it
and printing it as a float. &lt;em&gt;&lt;strong&gt;Drum Roll&lt;/strong&gt;&lt;/em&gt; And the answer is?&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;"val: -2.000000"&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;"Wait a minute, that doesn't quite sound right". You're right, it does seem a
bit strange that this is exactly -2. Well it may be that we are not printing
enough decimal places to see the full result, so update our print statement to:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;val: %.64lf&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="kt"&gt;double&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;val&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And now we get:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;"val: -2.0000001595175973534423974342644214630126953125000000"&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Much better... But still where did this number come from and why wasn't it the
-2 that we were expecting?&lt;/p&gt;
&lt;h2&gt;Kernel Pointers&lt;/h2&gt;
&lt;p&gt;At this point suspicions had been raised that what was being printed by my
colleague was not what he expected and that this was in fact a Kernel pointer.
How do you know? Lets take a step back for a second...&lt;/p&gt;
&lt;p&gt;In the PowerPC architecture, the address space which can be seen by an
application is known as the &lt;em&gt;effective&lt;/em&gt; address space. We can take this
and translate it into a &lt;em&gt;virtual&lt;/em&gt; address which when mapped through the
HPT (hash page table) gives us a &lt;em&gt;real&lt;/em&gt; address (or the hardware memory address).&lt;/p&gt;
&lt;p&gt;The &lt;em&gt;effective&lt;/em&gt; address space is divided into 5 regions:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Effective Address Table" src="/images/surajjs/doubles_in_hex/effective_address.png"&gt;&lt;/p&gt;
&lt;p&gt;As you may notice, Kernel addresses begin with 0xc. This has the advantage that
we can map a &lt;em&gt;virtual&lt;/em&gt; address without the need for a table by simply
masking the top nibble.&lt;/p&gt;
&lt;p&gt;Thus it would be reasonable to assume that our value (0xc00000001568fba0) was
indeed a pointer to a Kernel address (and further code investigation confirmed
this).&lt;/p&gt;
&lt;h2&gt;But What is -2 as a Double in hex?&lt;/h2&gt;
&lt;p&gt;Well lets modify the above program and find out:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;stdio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;

&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;double&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;val&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;-2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;val: 0x%lx&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;val&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Result?&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;"val: 0xc000000000000000"&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Now that sounds much better. Lets take a closer look:&lt;/p&gt;
&lt;h5&gt;0xc000000000000000:&lt;/h5&gt;
&lt;p&gt;Sign Bit: 1     -&amp;gt; Negative&lt;br&gt;
Exponent: 0x400 -&amp;gt; 2&lt;sup&gt;(1024 - 1023)&lt;/sup&gt;&lt;br&gt;
Fraction: 0x0   -&amp;gt; Zero&lt;br&gt;&lt;/p&gt;
&lt;p&gt;So if you remember from above, we have:&lt;/p&gt;
&lt;p&gt;(-1)&lt;sup&gt;&lt;strong&gt;1&lt;/strong&gt;&lt;/sup&gt; x 1.&lt;em&gt;&lt;strong&gt;0&lt;/strong&gt;&lt;/em&gt; x 2&lt;sup&gt;&lt;strong&gt;(1024 - 1023)&lt;/strong&gt;&lt;/sup&gt; = -2&lt;/p&gt;
&lt;p&gt;What about -1? -3?&lt;/p&gt;
&lt;h4&gt;-1:&lt;/h4&gt;
&lt;h5&gt;0xbff0000000000000:&lt;/h5&gt;
&lt;p&gt;Sign Bit: 1     -&amp;gt; Negative&lt;br&gt;
Exponent: 0x3ff -&amp;gt; 2&lt;sup&gt;(1023 - 1023)&lt;/sup&gt;&lt;br&gt;
Fraction: 0x0   -&amp;gt; Zero&lt;br&gt;&lt;/p&gt;
&lt;p&gt;(-1)&lt;sup&gt;&lt;strong&gt;1&lt;/strong&gt;&lt;/sup&gt; x 1.&lt;em&gt;&lt;strong&gt;0&lt;/strong&gt;&lt;/em&gt; x 2&lt;sup&gt;&lt;strong&gt;(1023 - 1023)&lt;/strong&gt;&lt;/sup&gt; = -1&lt;/p&gt;
&lt;h4&gt;-3:&lt;/h4&gt;
&lt;h5&gt;0xc008000000000000:&lt;/h5&gt;
&lt;p&gt;Sign Bit: 1                     -&amp;gt; Negative&lt;br&gt;
Exponent: 0x400                 -&amp;gt; 2&lt;sup&gt;(1024 - 1023)&lt;/sup&gt;&lt;br&gt;
Fraction: 0x8000000000000       -&amp;gt; 0.5&lt;br&gt;&lt;/p&gt;
&lt;p&gt;(-1)&lt;sup&gt;&lt;strong&gt;1&lt;/strong&gt;&lt;/sup&gt; x 1.&lt;em&gt;&lt;strong&gt;5&lt;/strong&gt;&lt;/em&gt; x 2&lt;sup&gt;&lt;strong&gt;(1024 - 1023)&lt;/strong&gt;&lt;/sup&gt; = -3&lt;/p&gt;
&lt;h2&gt;So What Have We Learnt?&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Firstly&lt;/strong&gt;, make sure that what you're printing is what you think you're printing.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Secondly&lt;/strong&gt;, if it looks like a Kernel pointer then you're probably not printing
what you think you're printing.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Thirdly&lt;/strong&gt;, all Kernel pointers ~= -2 if you treat them as a double.&lt;/p&gt;
&lt;p&gt;And &lt;strong&gt;Finally&lt;/strong&gt;, &lt;em&gt;with my morning gone&lt;/em&gt;, I can say for certain that if we treat it as
a double, 0xc00000001568fba0 =
-2.0000001595175973534423974342644214630126953125.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Suraj Jitindar Singh</dc:creator><pubDate>Thu, 12 May 2016 22:22:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-05-12:/blog/2016/05/12/doubles-in-hex-and-why-kernel-addresses-2/</guid><category>Education</category><category>double</category><category>float</category><category>hex</category><category>kernel</category></item><item><title>Getting logs out of things</title><link>https://sthbrx.github.io/blog/2016/03/22/getting-logs-out-of-things/</link><description>&lt;p&gt;Here at OzLabs, we have an unfortunate habit of making our shiny Power computers very sad, which is a common problem in systems programming and kernel hacking. When this happens, we like having logs. In particular, we like to have the kernel log and the OPAL firmware log, which are, very surprisingly, rather helpful when debugging kernel and firmware issues.&lt;/p&gt;
&lt;p&gt;Here's how to get them.&lt;/p&gt;
&lt;h2&gt;From userspace&lt;/h2&gt;
&lt;p&gt;You're lucky enough that your machine is still up, yay! As every Linux sysadmin knows, you can just grab the kernel log using &lt;code&gt;dmesg&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;As for the OPAL log: we can simply ask OPAL to tell us where its log is located in memory, copy it from there, and hand it over to userspace. In Linux, as per standard Unix conventions, we do this by exposing the log as a file, which can be found in &lt;code&gt;/sys/firmware/opal/msglog&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Annoyingly, the &lt;code&gt;msglog&lt;/code&gt; file reports itself as size 0 (I'm not sure exactly why, but I &lt;em&gt;think&lt;/em&gt; it's due to limitations in sysfs), so if you try to copy the file with &lt;code&gt;cp&lt;/code&gt;, you end up with just a blank file. However, you can read it with &lt;code&gt;cat&lt;/code&gt; or &lt;code&gt;less&lt;/code&gt;.&lt;/p&gt;
&lt;h2&gt;From &lt;code&gt;xmon&lt;/code&gt;&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;xmon&lt;/code&gt; is a really handy in-kernel debugger for PowerPC that allows you to do basic debugging over the console without hooking up a second machine to use with &lt;code&gt;kgdb&lt;/code&gt;. On our development systems, we often configure &lt;code&gt;xmon&lt;/code&gt; to automatically begin debugging whenever we hit an oops or panic (using &lt;code&gt;xmon=on&lt;/code&gt; on the kernel command line, or the &lt;code&gt;XMON_DEFAULT&lt;/code&gt; Kconfig option). It can also be manually triggered:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="nv"&gt;@p86&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;proc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;sysrq&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="k"&gt;trigger&lt;/span&gt;
&lt;span class="nl"&gt;sysrq&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;SysRq&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Entering&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;xmon&lt;/span&gt;
&lt;span class="n"&gt;cpu&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x7&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;Vector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="k"&gt;at&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c000000fcd717a80&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
&lt;span class="nl"&gt;pc&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;c000000000085ad8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sysrq_handle_xmon&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x68&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x80&lt;/span&gt;
&lt;span class="nl"&gt;lr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;c000000000085ad8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sysrq_handle_xmon&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mh"&gt;0x68&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mh"&gt;0x80&lt;/span&gt;
&lt;span class="nl"&gt;sp&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;c000000fcd717be0&lt;/span&gt;
&lt;span class="nl"&gt;msr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9000000000009033&lt;/span&gt;
&lt;span class="k"&gt;current&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xc000000fcd689200&lt;/span&gt;
&lt;span class="n"&gt;paca&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xc00000000fe01c00&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="nl"&gt;softe&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nl"&gt;irq_happened&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x01&lt;/span&gt;
&lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;7127&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;comm&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bash&lt;/span&gt;
&lt;span class="n"&gt;Linux&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;4.5.0&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;ajd&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;11118&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;g968f3e3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ajd&lt;/span&gt;&lt;span class="nv"&gt;@ka1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gcc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;5.2.1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;20150930&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;GCC&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;#1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;SMP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Tue&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Mar&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;17&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;01&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;58&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;AEDT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2016&lt;/span&gt;
&lt;span class="n"&gt;enter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="vm"&gt;?&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;help&lt;/span&gt;
&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="n"&gt;mon&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;From &lt;code&gt;xmon&lt;/code&gt;, simply type &lt;code&gt;dl&lt;/code&gt; to dump out the kernel log. If you'd like to page through the log rather than dump the entire thing at once, use &lt;code&gt;#&amp;lt;n&amp;gt;&lt;/code&gt; to split it into groups of &lt;code&gt;n&lt;/code&gt; lines.&lt;/p&gt;
&lt;p&gt;Until recently, it wasn't as easy to extract the OPAL log without knowing magic offsets. A couple of months ago, I was debugging a nasty CAPI issue and got rather frustrated by this, so one day when I had a couple of hours free I &lt;a href="http://patchwork.ozlabs.org/patch/581775/"&gt;refactored&lt;/a&gt; the existing sysfs interface and &lt;a href="http://patchwork.ozlabs.org/patch/581774/"&gt;added&lt;/a&gt; the &lt;code&gt;do&lt;/code&gt; command to &lt;code&gt;xmon&lt;/code&gt;. These patches will be included from kernel 4.6-rc1 onwards.&lt;/p&gt;
&lt;p&gt;When you're done, &lt;code&gt;x&lt;/code&gt; will attempt to recover the machine and continue, &lt;code&gt;zr&lt;/code&gt; will reboot, and &lt;code&gt;zh&lt;/code&gt; will halt.&lt;/p&gt;
&lt;h2&gt;From the FSP&lt;/h2&gt;
&lt;p&gt;Sometimes, not even &lt;code&gt;xmon&lt;/code&gt; will help you. In production environments, you're not generally going to start a debugger every time you have an incident. Additionally, a serious hardware error can cause a 'checkstop', which completely halts the system. (Thankfully, end users don't see this very often, but kernel developers, on the other hand...)&lt;/p&gt;
&lt;p&gt;This is where the Flexible Service Processor, or FSP, comes in. The FSP is an IBM-developed baseboard management controller used on most IBM-branded Power Systems machines, and is responsible for a whole range of things, including monitoring system health. Among its many capabilities, the FSP can automatically take "system dumps" when fatal errors occur, capturing designated regions of memory for later debugging. System dumps can be configured and triggered via the FSP's web interface, which is beyond the scope of this post but is &lt;a href="https://www.ibm.com/support/knowledgecenter/POWER8/p8ha5/mainstoragedump.htm?cp=POWER8%2F1-3-14-2"&gt;documented&lt;/a&gt; in IBM Power Systems user manuals.&lt;/p&gt;
&lt;p&gt;How does the FSP know what to capture? As it turns out, skiboot (the firmware which implements OPAL) maintains a &lt;a href="https://github.com/open-power/skiboot/blob/master/hw/fsp/fsp-mdst-table.c"&gt;Memory Dump Source Table&lt;/a&gt; which tells the FSP which memory regions to dump. MDST updates are recorded in the OPAL log:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2690088026&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Max&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;entries&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;256&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2690090666&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Addr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x31000000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x100000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;added&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2690093767&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Addr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x31100000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x100000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;added&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2750378890&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;updated&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;11199672771&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Addr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x1fff772780&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x200000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;added&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;11215193760&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;updated&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;28031311971&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;updated&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;28411709421&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Addr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x1fff830000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x100000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;added&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;28417251110&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MDST&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;updated&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In the above log, we see four entries: the skiboot/OPAL log, the &lt;a href="https://github.com/open-power/hostboot"&gt;hostboot&lt;/a&gt; runtime log, the petitboot Linux kernel log (which doesn't make it into the final dump) and the real Linux kernel log. skiboot obviously adds the OPAL and hostboot logs to the MDST early in boot, but it also exposes the &lt;a href="https://github.com/open-power/skiboot/blob/master/doc/opal-api/opal-register-dump-region-101.txt"&gt;&lt;code&gt;OPAL_REGISTER_DUMP_REGION&lt;/code&gt;&lt;/a&gt; call which can be used by the operating system to register additional regions. Linux uses this to &lt;a href="https://github.com/torvalds/linux/blob/master/arch/powerpc/platforms/powernv/opal.c#L608"&gt;register the kernel log buffer&lt;/a&gt;. If you're a kernel developer, you could potentially use the OPAL call to register your own interesting bits of memory.&lt;/p&gt;
&lt;p&gt;So, the MDST is all set up, we go about doing our business, and suddenly we checkstop. The FSP does its sysdump magic and a few minutes later it reboots the system. What now?&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;After we come back up, the FSP notifies OPAL that a new dump is available. Linux exposes the dump to userspace under &lt;code&gt;/sys/firmware/opal/dump/&lt;/code&gt;.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;a href="https://sourceforge.net/projects/linux-diag/files/ppc64-diag/"&gt;ppc64-diag&lt;/a&gt; is a suite of utilities that assist in manipulating FSP dumps, including the &lt;code&gt;opal_errd&lt;/code&gt; daemon. &lt;code&gt;opal_errd&lt;/code&gt; monitors new dumps and saves them in &lt;code&gt;/var/log/dump/&lt;/code&gt; for later analysis.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;opal-dump-parse&lt;/code&gt; (also in the &lt;code&gt;ppc64-diag&lt;/code&gt; suite) can be used to extract the sections we care about from the dump:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="n"&gt;p86&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="c1"&gt;# opal-dump-parse -l SYSDUMP.842EA8A.00000001.20160322063051 &lt;/span&gt;
&lt;span class="o"&gt;|---------------------------------------------------------|&lt;/span&gt;
&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;ID&lt;/span&gt;&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="n"&gt;SECTION&lt;/span&gt;&lt;span class="w"&gt;                              &lt;/span&gt;&lt;span class="n"&gt;SIZE&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
&lt;span class="o"&gt;|---------------------------------------------------------|&lt;/span&gt;
&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="n"&gt;Opal&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt;                           &lt;/span&gt;&lt;span class="mi"&gt;1048576&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="n"&gt;HostBoot&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;Runtime&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="mi"&gt;1048576&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="n"&gt;printk&lt;/span&gt;&lt;span class="w"&gt;                             &lt;/span&gt;&lt;span class="mi"&gt;1048576&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
&lt;span class="o"&gt;|---------------------------------------------------------|&lt;/span&gt;
&lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;completed&lt;/span&gt;
&lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="n"&gt;p86&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="c1"&gt;# opal-dump-parse -s 1 SYSDUMP.842EA8A.00000001.20160322063051 &lt;/span&gt;
&lt;span class="n"&gt;Captured&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Opal&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;842&lt;/span&gt;&lt;span class="n"&gt;EA8A&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;00000001.20160322063051&lt;/span&gt;
&lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="n"&gt;p86&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="c1"&gt;# opal-dump-parse -s 2 SYSDUMP.842EA8A.00000001.20160322063051 &lt;/span&gt;
&lt;span class="n"&gt;Captured&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;HostBoot&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;Runtime&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;842&lt;/span&gt;&lt;span class="n"&gt;EA8A&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;00000001.20160322063051&lt;/span&gt;
&lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="n"&gt;p86&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="c1"&gt;# opal-dump-parse -s 128 SYSDUMP.842EA8A.00000001.20160322063051 &lt;/span&gt;
&lt;span class="n"&gt;Captured&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;printk&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;842&lt;/span&gt;&lt;span class="n"&gt;EA8A&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;00000001.20160322063051&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There's various other types of dumps and logs that I won't get into here. I'm probably obliged to say that if you're having problems out in the wild, you should probably contact your friendly local IBM Service Representative...&lt;/p&gt;
&lt;h2&gt;Acknowledgements&lt;/h2&gt;
&lt;p&gt;Thanks to &lt;a href="https://flamingspork.com"&gt;Stewart Smith&lt;/a&gt; for pointing me in the right direction regarding FSP sysdumps and related tools.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Donnellan</dc:creator><pubDate>Tue, 22 Mar 2016 18:00:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-03-22:/blog/2016/03/22/getting-logs-out-of-things/</guid><category>OpenPOWER</category><category>debugging</category><category>skiboot</category><category>OPAL</category><category>FSP</category><category>kernel</category><category>development</category></item><item><title>The Elegance of the Plaintext Patch</title><link>https://sthbrx.github.io/blog/2016/03/22/the-elegance-of-the-plaintext-patch/</link><description>&lt;p&gt;I've only been working on the Linux kernel for a few months.  Before that, I worked with proprietary source control at work and common tools like GitHub at home.  The concept of the mailing list seemed obtuse to me.  If I noticed a problem with some program, I'd be willing to open an issue on GitHub but not to send an email to a mailing list.  Who still uses those, anyway?&lt;/p&gt;
&lt;p&gt;Starting out with the kernel meant I had to figure this email thing out.  &lt;code&gt;git format-patch&lt;/code&gt; and &lt;code&gt;git send-email&lt;/code&gt; take most of the pain out of formatting and submitting a patch, which is nice.  The patch files generated by &lt;code&gt;format-patch&lt;/code&gt; open nicely in Emacs by default, showing all whitespace and letting you pick up any irregularities.  &lt;code&gt;send-email&lt;/code&gt; means you can send it to yourself or a friend first, finding anything that looks stupid before being exposed to the public.&lt;/p&gt;
&lt;p&gt;And then what?  You've sent an email.  It gets sent to hundreds or thousands of people.  Nowhere near that many will read it.  Some might miss it due to their mail server going down, or the list flagging your post as spam, or requiring moderation.  Some recipients will be bots that archive mail on the list, or publish information about the patch.  If you haven't formatted it correctly, someone will let you know quickly.  If your patch is important or controversial, you'll have all sorts of responses.  If your patch is small or niche, you might not ever hear anything back.&lt;/p&gt;
&lt;p&gt;I remember when I sent my first patch.  I was talking to a former colleague who didn't understand the patch/mailing list workflow at all.  I sent him a link to my patch on a mail archive.  I explained it like a pull request - here's my code, you can find the responses.  What's missing from a GitHub-esque pull request?  We don't know what tests it passed.  We don't know if it's been merged yet, or if the maintainer has looked at it.  It takes a bit of digging around to find out who's commented on it.  If it's part of a series, that's awkward to find out as well.  What about revisions of a series?  That's another pain point.&lt;/p&gt;
&lt;p&gt;Luckily, these problems do have solutions.  &lt;a href="http://jk.ozlabs.org/projects/patchwork/"&gt;Patchwork&lt;/a&gt;, written by fellow OzLabs member &lt;a href="http://jk.ozlabs.org"&gt;Jeremy Kerr&lt;/a&gt;, changes the way we work with patches.  Project maintainers rely on Pathwork instances, such as &lt;a href="https://patchwork.ozlabs.org"&gt;https://patchwork.ozlabs.org&lt;/a&gt;, for their day-to-day workflow: tagging reviewers, marking the status of patches, keeping track of tests, acks, reviews and comments in one place.  Missing from this picture is support for series and revisions, which is a feature that's being developed by the &lt;a href="https://www.freedesktop.org/wiki/"&gt;freedesktop&lt;/a&gt; project.  You can check out their changes in action &lt;a href="https://patchwork.freedesktop.org"&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;So, Patchwork helps patches and email catch up to what GitHub has in terms of ease of information.  We're still missing testing and other hooks.  What about review?  What can we do with email, compared to GitHub and the like?&lt;/p&gt;
&lt;p&gt;In my opinion, the biggest feature of email is the ease of review.  Just reply inline and you're done.  There's inline commenting on GitHub and GitLab, which works well but is a bit tacky, people commenting on the same thing overlap and conflict, each comment generates a notification (which can be an email until you turn that off).  Plus, since it's email, it's really easy to bring in additional people to the conversation as necessary.  If there's a super lengthy technical discussion in the kernel, it might just take Linus to resolve.&lt;/p&gt;
&lt;p&gt;There are alternatives to just replying to email, too, such as &lt;a href="https://www.gerritcodereview.com/"&gt;Gerrit&lt;/a&gt;.  Gerrit's pretty popular, and has a huge amount of features.  I understand why people use it, though I'm not much of a fan.  Reason being, it doesn't add to the email workflow, it replaces it.  Plaintext email is supported on pretty much any device, with a bunch of different programs.  From the goals of Patchwork: "patchwork should supplement mailing lists, not replace them".&lt;/p&gt;
&lt;p&gt;Linus Torvalds famously explained why he prefers email over GitHub pull requests &lt;a href="https://github.com/torvalds/linux/pull/17"&gt;here&lt;/a&gt;, using &lt;a href="https://groups.google.com/forum/#!topic/linux.kernel/w957vpu3PPU"&gt;this&lt;/a&gt; pull request from Ben Herrenschmidt as an example of why git's own pull request format is superior to that of GitHub.  Damien Lespiau, who is working on the freedesktop Patchwork fork, &lt;a href="http://damien.lespiau.name/2016/02/augmenting-mailing-lists-with-patchwork.html"&gt;outlines on his blog&lt;/a&gt; all the issues he has with mailing list workflows and why he thinks mailing lists are a relic of the past.  His work on Patchwork has gone a long way to help fix those problems, however I don't think mailing lists are outdated and superceded, I think they are timeless.  They are a technology-agnostic, simple and free system that will still be around if GitHub dies or alienates its community.&lt;/p&gt;
&lt;p&gt;That said, there's still the case of the missing features.  What about automated testing?  What about developer feedback?  What about making a maintainer's life easier?  We've been working on improving these issues, and I'll outline how we're approaching them in a future post.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Russell Currey</dc:creator><pubDate>Tue, 22 Mar 2016 13:53:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-03-22:/blog/2016/03/22/the-elegance-of-the-plaintext-patch/</guid><category>Development</category><category>development</category><category>education</category><category>kernel</category><category>patches</category></item><item><title>No Network For You</title><link>https://sthbrx.github.io/blog/2016/03/21/no-network-for-you/</link><description>&lt;p&gt;In POWER land &lt;a href="https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface"&gt;IPMI&lt;/a&gt; is mostly known as the method to access the machine's console and start interacting with Petitboot. However it also has a plethora of other features, handily described in the 600ish page &lt;a href="http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html"&gt;IPMI specification&lt;/a&gt; (which you can go read yourself).&lt;/p&gt;
&lt;p&gt;One especially relevant feature to Petitboot however is the 'chassis bootdev' command, which you can use to tell Petitboot to ignore any existing boot order, and only consider boot options of the type you specify (eg. 'network', 'disk', or 'setup' to not boot at all). Support for this has been in Petitboot for a while and should work on just about any machine you can get your hands on.&lt;/p&gt;
&lt;h2&gt;Network Overrides&lt;/h2&gt;
&lt;p&gt;Over in OpenPOWER&lt;sup&gt;1&lt;/sup&gt; land however, someone took this idea and pushed it further - why not allow the network configuration to be overwritten too? This isn't in the IPMI spec, but if you cast your gaze down to page 398 where the spec lays out the entire format of the IPMI request, there is a certain field named "OEM Parameters". This is an optional amount of space set aside for whatever you like, which in this case is going to be data describing an override of the network config.&lt;/p&gt;
&lt;p&gt;This allows a user to tell Petitboot over IPMI to either;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Disable the network completely,&lt;/li&gt;
&lt;li&gt;Set a particular interface to use DHCP, or&lt;/li&gt;
&lt;li&gt;Set a particular interface to use a specific static configuration.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Any of these options will cause any existing network configurations to be ignored.&lt;/p&gt;
&lt;h2&gt;Building the Request&lt;/h2&gt;
&lt;p&gt;Since this is an OEM-specific command, your average ipmitool package isn't going to have a nice way of making this request, such as 'chassis bootdev network'. Rather you need to do something like this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;ipmitool&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lanplus&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;H&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;yourbmc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;U&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;P&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x08&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x61&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x80&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x21&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x70&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x62&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x21&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x01&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x06&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x04&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xf4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x52&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x14&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xf3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x01&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xdf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x01&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x0a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x3d&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xa1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x42&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x10&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x0a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x3d&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Horrific right? In the near future the Petitboot tree will include a helper program to format this request for you, but in the meantime (and for future reference), lets lay out how to put this together:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nx"&gt;Specify&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;chassis bootdev&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;command&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;field&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;96&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;field&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x6&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x8&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;

&lt;span class="nx"&gt;Unique&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;that&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Petitboot&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;recognises&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="mh"&gt;0x2&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x7&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x6&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x2&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="nx"&gt;Version&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;field&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;

&lt;span class="nx"&gt;Size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;hardware&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;address&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;

&lt;span class="nx"&gt;Size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;address&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;IPv4&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;IPv6&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;

&lt;span class="nx"&gt;Hardware&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;MAC&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;address&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="mh"&gt;0xf&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x5&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x1&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xf&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xd&lt;/span&gt;&lt;span class="nx"&gt;f&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;

&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;Ignore&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;flag&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;and&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;DHCP&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;Static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;flag&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;DHCP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;Below&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;fields&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;only&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;required&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;setting&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IP&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nx"&gt;IP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Address&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x3&lt;/span&gt;&lt;span class="nx"&gt;d&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xa&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x4&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;

&lt;span class="nx"&gt;Subnet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Mask&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;eg&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="mh"&gt;0x1&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;
&lt;span class="nx"&gt;Gateway&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Address&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;..&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x3&lt;/span&gt;&lt;span class="nx"&gt;d&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="mh"&gt;0x0&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Clearing a network override is as simple as making a request empty aside from the header:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x08&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x61&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x80&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x21&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x70&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x62&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x21&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x01&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x00&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You can also read back the request over IPMI with this request:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x09&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x61&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="n"&gt;x00&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That's it! Ideally this is something you would be scripting rather than bashing out on the keyboard - the main use case at the moment is as a way to force a machine to netboot against a known good source, rather than whatever may be available on its other interfaces.&lt;/p&gt;
&lt;p&gt;[1] The reason this is only available on OpenPOWER machines at the moment is that support for the IPMI command itself depends on the BMC firmware, and non-OpenPOWER machines use an FSP which is a different platform.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Mon, 21 Mar 2016 15:23:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-03-21:/blog/2016/03/21/no-network-for-you/</guid><category>Petitboot</category><category>petitboot</category><category>goodposts</category><category>realcontent</category><category>ipmi</category><category>bmc</category><category>based16</category></item><item><title>And now for something completely different: approximate computing</title><link>https://sthbrx.github.io/blog/2016/03/15/and-now-for-something-completely-different-approximate-computing/</link><description>&lt;p&gt;In early February I had the opportunity to go the the NICTA Systems Summer School, where Cyril and I were invited to represent IBM. There were a number of excellent talks across a huge range of systems related subjects, but the one that has stuck with me the most was a talk given by &lt;a href="http://homes.cs.washington.edu/~luisceze/"&gt;Luis Ceze&lt;/a&gt;  on a topic called approximate computing. So here, in hopes that you too find it interesting, is a brief run-down on what I learned.&lt;/p&gt;
&lt;p&gt;Approximate computing is fundamentally about trading off accuracy for something else - often speed or power consumption. Initially this sounded like a very weird proposition: computers do things like 'running your operating system' and 'reading from and writing to disks': things you need to always be absolutely correct if you want anything vaguely resembling reliability. It turns out that this is actually not as big a roadblock as I had assumed - you can work around it fairly easily.&lt;/p&gt;
&lt;p&gt;The model proposed for approximate computing is as follows. You divide your computation up into two classes: 'precise', and 'approximate'. You use 'precise' computations when you need to get exact answers: so for example if you are constructing a JPEG file, you want the JPEG header to be exact. Then you have approximate computations: so for example the contents of your image can be approximate.&lt;/p&gt;
&lt;p&gt;For correctness, you have to establish some boundaries: you say that precise data can be used in approximate calculations, but that approximate data isn't allowed to cross back over and pollute precise calculations. This, while intuitively correct, poses some problems in practise: when you want to write out your approximate JPEG data, you need an operation that allows you to 'bless' (or in their terms 'endorse') some approximate data so it can be used in the precise file system operations.&lt;/p&gt;
&lt;p&gt;In the talk we were shown an implementation of this model in Java, called &lt;a href="http://sampa.cs.washington.edu/research/approximation/enerj.html"&gt;EnerJ&lt;/a&gt;. EnerJ allows you to label variables with either &lt;code&gt;@Precise&lt;/code&gt; if you're dealing with precise data, or &lt;code&gt;@Approx&lt;/code&gt; if you're dealing with approximate data. The compiler was modified so that it would do all sorts of weird things when it knew it was dealing with approximate data: for example, drop loop iterations entirely, do things in entirely non-determistic ways - all sorts of fun stuff. It turns out this works surprisingly well.&lt;/p&gt;
&lt;p&gt;However, the approximate computing really shines when you can bring it all the way down to the hardware level. The first thing they tried was a CPU with both 'approximate' and precise execution engines, but this turned out not to have the power savings hoped for. What seemed to work really well was a model where some approximate calculations could be identified ahead of time, and then replaced with neural networks in hardware. These neural networks approximated the calculations, but did so at significantly lower power levels. This sounded like a really promising concept, and it will be interesting to see if this goes anywhere over the next few years.&lt;/p&gt;
&lt;p&gt;There's a lot of work evaluating the quality of the approximate result, for cases where the set of inputs is known, and when the inputs is not known. This is largely beyond my understanding, so I'll simply refer you to some of the papers &lt;a href="http://sampa.cs.washington.edu/research/approximation/enerj.html"&gt;listed on the website&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The final thing covered in the talk was bringing approximate computing into current paradigms by just being willing to accept higher user-visible error rates. For example, they hacked up a network stack to accept packets with invalid checksums. This has had mixed results so far. A question I had (but didn't get around to asking!) would be whether the mathematical properties of checksums (i.e. that they can correct a certain number of bit errors) could be used to correct some of the errors, rather than just accepting/rejecting them blindly. Perhaps by first attempting to correct errors using the checksums, we will be able to fix the simpler errors, reducing the error rate visible to the user.&lt;/p&gt;
&lt;p&gt;Overall, I found the NICTA Systems Summer School to be a really interesting experience (and I hope to blog more about it soon). If you're a university student in Australia, or an academic, see if you can make it in 2017!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Tue, 15 Mar 2016 11:30:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-03-15:/blog/2016/03/15/and-now-for-something-completely-different-approximate-computing/</guid><category>Education</category><category>conferences</category></item><item><title>linux.conf.au 2016: A set of brief thoughts</title><link>https://sthbrx.github.io/blog/2016/03/15/linuxconfau-2016-a-set-of-brief-thoughts/</link><description>&lt;p&gt;Recently most of us attended LCA2016. This is one set of reflections on what we heard and what we've thought since. (Hopefully not the only set of reflections that will be posted on this blog either!)&lt;/p&gt;
&lt;p&gt;LCA was 2 days of miniconferences plus 3 days of talks. Here, I've picked some of the more interesting talks I attended, and I've written down some thoughts. If you find the thoughts interesting, you can click through and watch the whole talk video, because LCA is awesome like that.&lt;/p&gt;
&lt;h4&gt;Life is better with Rust's community automation&lt;/h4&gt;
&lt;p&gt;&lt;a href="https://www.youtube.com/watch?v=dIageYT0Vgg"&gt;This talk&lt;/a&gt; is probably the one that's had the biggest impact on our team so far. We were really impressed by the community automation that Rust has: the way they can respond to pull requests from new community members in a way that lets them keep their code quality high and be nice to everyone at the same time.&lt;/p&gt;
&lt;p&gt;The system that they've developed is fascinating (and seems fantastic). However, their system uses pull requests, while we use mailing lists. Pull requests are easy, because github has good hook support, but how do we link mailing lists to an automatic test system?&lt;/p&gt;
&lt;p&gt;As it turns out, this is something we're working on: we already have &lt;a href="http://patchwork.ozlabs.org/"&gt;Patchwork&lt;/a&gt;, and &lt;a href="https://openpower.xyz/"&gt;Jenkins&lt;/a&gt;: how do we link them? We have something brewing, which we'll open source real soon now - stay tuned!&lt;/p&gt;
&lt;h4&gt;Usable formal methods - are we there yet?&lt;/h4&gt;
&lt;p&gt;I liked &lt;a href="https://www.youtube.com/watch?v=RxHjhBVOCSU"&gt;this talk&lt;/a&gt;, as I have a soft spot for formal methods (as I have a soft spot for maths). It covers applying a bunch of static analysis and some of the less intrusive formal methods (in particular &lt;a href="http://www.cprover.org/cbmc/"&gt;cbmc&lt;/a&gt;) to an operating system kernel. They were looking at eChronos rather than Linux, but it's still quite an interesting set of results.&lt;/p&gt;
&lt;p&gt;We've also tried to increase our use of static analysis, which has already found a &lt;a href="http://patchwork.ozlabs.org/patch/580629/"&gt;real bug&lt;/a&gt;. We're hoping to scale this up, especially the use of sparse and cppcheck, but we're a bit short on developer cycles for it at the moment.&lt;/p&gt;
&lt;h4&gt;Adventures in OpenPower Firmware&lt;/h4&gt;
&lt;p&gt;Stewart Smith - another OzLabber - gave &lt;a href="https://www.youtube.com/watch?v=a4XGvssR-ag"&gt;this talk&lt;/a&gt; about, well, OpenPOWER firmware. This is a large part of our lives in OzLabs, so it's a great way to get a picture of what we do each day. It's also a really good explanation of the open source stack we have: a POWER8 CPU runs open-source from the first cycle.&lt;/p&gt;
&lt;h4&gt;What Happens When 4096 Cores &lt;code&gt;All Do synchronize_rcu_expedited()&lt;/code&gt;?&lt;/h4&gt;
&lt;p&gt;Paul McKenney is a parallel programming genius - he literally &lt;a href="https://www.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html"&gt;'wrote the book'&lt;/a&gt; (or at least, wrote &lt;em&gt;a&lt;/em&gt; book!) on it. &lt;a href="https://www.youtube.com/watch?v=1nfpjHTWaUc"&gt;His talk&lt;/a&gt; is - as always - a brain-stretching look at parallel programming within the RCU subsystem of the Linux kernel. In particular, the tree structure for locking that he presents is really interesting and quite a clever way of scaling what at first seems to be a necessarily global lock.&lt;/p&gt;
&lt;p&gt;I'd also really recommed &lt;a href="https://www.youtube.com/watch?v=tFmajPt0_hI"&gt;RCU Mutation Testing&lt;/a&gt;, from the kernel miniconf, also by Paul.&lt;/p&gt;
&lt;h4&gt;What I've learned as the kernel docs maintainer&lt;/h4&gt;
&lt;p&gt;As an extra bonus: I mention &lt;a href="https://www.youtube.com/watch?v=gsJXf6oSbAE"&gt;this talk&lt;/a&gt;, just to say "why on earth have we still not fixed the Linux kernel &lt;a href="https://www.kernel.org/doc/linux/README"&gt;README&lt;/a&gt;"?!!?&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Tue, 15 Mar 2016 11:30:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-03-15:/blog/2016/03/15/linuxconfau-2016-a-set-of-brief-thoughts/</guid><category>Education</category><category>conferences</category></item><item><title>Learning From the Best</title><link>https://sthbrx.github.io/blog/2016/03/03/learning-from-the-best/</link><description>&lt;p&gt;When I first started at IBM I knew how to alter Javascript and compile it. This is because of my many years playing Minecraft (yes I am a nerd). Now I have leveled up! I can understand and use Bash, Assembly, Python, Ruby and C! Writing full programs in any of these languages is a very difficult prospect but none the less achievable with what I know now. Whereas two weeks ago it would have been impossible. Working here even for a short time has been an amazing Learning experience for me, plus it looks great on a resume! Learning how to write C has been one of the most useful things I have learnt. I have already written programs for use both in and out of IBM. The first program I wrote was the standard newbie 'hello world' exercise. I have now expanded on that program so that it now says, "Hello world! This is Callum Scarvell". This is done using strings that recognise my name as a set character. Then I used a header file called conio.h or curses.h to recognise 'cal' as the short form of my name. This is so now I can abbreviate my name easier. Heres what the code looks like:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;stdio.h&amp;gt;&lt;/span&gt;
&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;string.h&amp;gt;&lt;/span&gt;
&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;curses.h&amp;gt;&lt;/span&gt;

&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Hello, World! This Is cal&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;first_name&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Callum&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;last_name&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Scarvell&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="cm"&gt;/* testing code */&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;strncmp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;first_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Callum&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;strncmp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;last_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Scarvell&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;last_name&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sc"&gt;&amp;#39;S&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;sprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;%s %s&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;first_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;last_name&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;strncmp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Callum Scarvell&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="n"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;This is %s&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="cm"&gt;/*printf(&amp;quot;actual string is -%s-\n&amp;quot;,name);*/&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;Name_Rec&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;

&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;clrscr&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;puts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Callum Scarvell : &lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;gets&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s"&gt;cal : &lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s"&gt;%c&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="sc"&gt;&amp;#39;\0&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The last two lines have been left out to make it a challenge to recreate. Feel free to test your own knowledge of C to finish the program! My ultimate goal for this program is to make it generate the text 'Hello World! This is Callum Scarvell's computer. Everybody else beware!'(which is easy) then import it into the Linux kernel to the profile login screen. Then I will have my own unique copy of the kernel. And I could call myself an LSD(Linux system developer). That's just a small pet project I have been working on in my time here. Another pet project of mine is my own very altered copy of the open source game NetHack. It's written in C as well and is very easy to tinker with. I have been able to do things like set my characters starting hit points to 40, give my character awesome starting gear and keep save files even after the death of a character. These are just a couple small projects that made learning C so much easier and a lot more fun. And the whole time I was learning C, Ruby, or Python I had some of the best system developers in the world showing me the ropes. This made things even easier, and much more comprehensive. So really its no surprise that in three short weeks I managed to learn almost four different languages and how to run a blog from the raw source code. The knowledge given to me by the OzLabs team is priceless and invaluable. I will forever remember all the new faces and what they taught me. And the &lt;em&gt;Linux Gods&lt;/em&gt; will answer your prayers whether e-mail or in person because they walk among us! So if you ever get an opportunity to do work experience, internship or a graduate placement take the chance to do it because you will learn many things that are not taught in school.&lt;/p&gt;
&lt;p&gt;If you would like to reveiw the source code for the blog or my work in general you can find me at &lt;a href="https://github.com/CallumScar"&gt;CallumScar.github.com&lt;/a&gt; or find me on facebook, &lt;a href="https://www.facebook.com/callum.scarvell/about"&gt;Callum Scarvell&lt;/a&gt;. &lt;/p&gt;
&lt;p&gt;And a huge thankyou to the OzLabs team for taking me on for the three weeks and for teaching me so much! I am forever indebted to everyone here.   &lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Callum Scarvell</dc:creator><pubDate>Thu, 03 Mar 2016 00:00:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-03-03:/blog/2016/03/03/learning-from-the-best/</guid><category>Education</category><category>education</category><category>work experience</category><category>Linux Gods</category></item><item><title>Work Experience At Ozlabs</title><link>https://sthbrx.github.io/blog/2016/02/25/work-experience-at-ozlabs/</link><description>&lt;p&gt;As a recent year twelve graduate my knowledge of computer science was very limited and my ability to write working programs was all but none. So you can imagine my excitement when I heard of an opening for work experience with IBM's internationally renowned Ozlabs team, or as I knew them the &lt;em&gt;Linux Gods&lt;/em&gt;. My first day of working at Ozlabs I learnt more about programing then in six years of secondary education. I met most of the Ozlabs team and made connections that will certainly help with my pursuit of a career in IT. Because in business its who you know more than what you know, and now I know the guys at Ozlabs I know how to write code and run it on my own Linux Distro. And on top of all the extremely valuable knowledge I am on a first name basis with the &lt;em&gt;Linux Gods&lt;/em&gt; at the LTC.&lt;/p&gt;
&lt;p&gt;After my first week at Ozlabs I cloned this blog from Octopress and reformatted it for pelican static site generator.For those who don't know Octopress is a ruby based static site generator so converting the embedded ruby gems to pelicans python code was no easy task for this newbie. Luckily I had a team of some of the best software developers in the world to help and teach me their ways. After we sorted the change from ruby to python and I was able to understand both languages, I presented my work to the team. They then decided to throw me a curve ball as they did not like any of pelicans default themes, instead they wanted the original Octopress theme on the new blog. This is how I learnt GitHub is my bestest friend, because some kind soul had already converted the ruby theme into python and it ran perfectly!&lt;/p&gt;
&lt;p&gt;Now it was a simple task of re-formatting the ruby-gem text files into markdown which is pelican compatible(which is why we chose pelican in the first place). So now we had a working pelican blog on the Octopress theme, one issue it was very annoying to navigate. Using my newly learned skills and understanding of python I inserted tags, categories, web-links, navigation bar and I started learning how to code C. And it all worked fine! That was what I a newbie could accomplish in one week. I still have two more weeks left here and I have plenty of really interesting work left to do. This has been one of the greatest learning experiences of my life and I would do it again if I could! So if you are looking for experience in it or software development look no further because you could be learning to code from the people who wrote the language itself. The &lt;em&gt;Linux Gods&lt;/em&gt;.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Callum Scarvell</dc:creator><pubDate>Thu, 25 Feb 2016 00:00:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-02-25:/blog/2016/02/25/work-experience-at-ozlabs/</guid><category>Education</category><category>Work Experience</category><category>Ozlabs</category></item><item><title>Panic, flushing and compromise</title><link>https://sthbrx.github.io/blog/2016/02/15/panic/</link><description>&lt;p&gt;This is a tale of a simple problem, with a relatively simple solution, that ended up being pretty complicated.&lt;/p&gt;
&lt;p&gt;The BMC of OpenPOWER machines expose a serial console.  It's pretty useful for getting information as the system is booting, or when it's having issues and the network is down.  OpenPOWER machines also have runtime firmware, namely &lt;a href="https://github.com/open-power/skiboot"&gt;skiboot&lt;/a&gt;, which the Linux kernel calls to make certain things happen.  One of those is writing to the serial console.  There's a function that &lt;a href="https://github.com/open-power/skiboot/blob/master/core/opal.c"&gt;skiboot exposes&lt;/a&gt;, &lt;code&gt;opal_poll_events()&lt;/code&gt; (which then calls &lt;code&gt;opal_run_pollers()&lt;/code&gt;), which the kernel calls frequently.  Among other things, it performs a partial flush of the serial console.  And that all works fine...until the kernel panics.&lt;/p&gt;
&lt;p&gt;Well, the kernel is in panic.  Who cares if it flushes the console?  It's dead.  It doesn't need to do anything else.&lt;/p&gt;
&lt;p&gt;Oh, right.  It prints the reason it panicked.  Turns out that's pretty useful.&lt;/p&gt;
&lt;p&gt;There's a pretty simple fix here that we can push into the firmware.  Most kernels are configured to reboot after panic, typically with some delay.  In OpenPOWER, the kernel reboots by calling into skiboot with the &lt;code&gt;opal_cec_reboot()&lt;/code&gt; function.  So all we need to do is flush out the console buffer:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;int64&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;opal_cec_reboot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;OPAL: Reboot request...&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;console_complete_flush&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;// &amp;lt;-- what I added&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// rebooting stuff happens here...&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;OPAL_SUCCESS&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Writing a complete flushing function was pretty easy, then call it from the power down and reboot functions.  Easy, all nicely contained in firmware.&lt;/p&gt;
&lt;p&gt;Now, what if the kernel isn't configured to reboot after panic.  Or, what if the reboot timer is really long?  Do you want to wait 3 minutes to see your panic output?  Probably not.  We need to call the pollers after panic.&lt;/p&gt;
&lt;p&gt;First, I had to figure out what the kernel actually &lt;em&gt;does&lt;/em&gt; when it panics.  Let's have a look at the &lt;a href="https://github.com/torvalds/linux/blob/master/kernel/panic.c"&gt;panic function itself&lt;/a&gt; to figure out where we could work some code in.&lt;/p&gt;
&lt;p&gt;In the &lt;code&gt;panic()&lt;/code&gt; function, the easiest place I found to put in some code was &lt;code&gt;panic_blink()&lt;/code&gt;.  This is supposed to be a function to blink the LEDs on your keyboard when the kernel is panicking, but we could set it to &lt;code&gt;opal_poll_events()&lt;/code&gt; and it'd work fine.  There, problem solved!&lt;/p&gt;
&lt;p&gt;Oh, wait.  That will never get accepted upstream, ever.  Let's try again.&lt;/p&gt;
&lt;p&gt;Well, there are &lt;code&gt;#ifdef&lt;/code&gt;s in the code that are architecture specific, for s390 and SPARC.  I could add an &lt;code&gt;#ifdef&lt;/code&gt; to check if we're an OpenPOWER machine, and if so, run the pollers a bunch of times.  That would also involve including architecture specific code from &lt;code&gt;arch/powerpc&lt;/code&gt;, and that's somewhat gross.  Maybe I could upstream this, but it'd be difficult.  There must be a better way.&lt;/p&gt;
&lt;p&gt;As a kernel noob, I found myself digging into what every function called by &lt;code&gt;panic()&lt;/code&gt; actually did, to see if there's a way I could use it.  I looked over it at first, but eventually I started looking harder at this line:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;kmsg_dump&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;KMSG_DUMP_PANIC&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It turns out &lt;code&gt;kmsg_dump()&lt;/code&gt; does what it says: dumps messages from the kernel.  Different parts of the kernel can register their own dumpers, so the kernel can have a variety of dumpers for different purposes.  One existing example in OpenPOWER is a kmsg dumper that stores messages in &lt;code&gt;nvram&lt;/code&gt; (non-volatile RAM), so you can find it after you reboot.&lt;/p&gt;
&lt;p&gt;Well, we don't really want to dump any output, it's already been sent to the output buffer.  We just need to flush it.  Pretty simple, just call &lt;code&gt;opal_poll_events()&lt;/code&gt; a whole bunch of times, right?  That &lt;em&gt;would&lt;/em&gt; work, though it'd be nice to have a better way than just calling the pollers.  Instead, we can add a new API call to skiboot specifically for console flushing, and call it from the kmsg dumper.&lt;/p&gt;
&lt;p&gt;Initially, I wired up the skiboot complete console flushing function to a new OPAL API call, and called that from the kernel.  After some feedback, this was refactored into a partial, incremental flush so it was more generic.  I also had to consider what happened if the machine was running a newer kernel and an older skiboot, so if the skiboot version didn't have my new flushing call it would fall back to calling the pollers an arbitrary amount of times.&lt;/p&gt;
&lt;p&gt;In the end, it looks like this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cm"&gt;/*&lt;/span&gt;
&lt;span class="cm"&gt; * Console output is controlled by OPAL firmware.  The kernel regularly calls&lt;/span&gt;
&lt;span class="cm"&gt; * OPAL_POLL_EVENTS, which flushes some console output.  In a panic state,&lt;/span&gt;
&lt;span class="cm"&gt; * however, the kernel no longer calls OPAL_POLL_EVENTS and the panic message&lt;/span&gt;
&lt;span class="cm"&gt; * may not be completely printed.  This function does not actually dump the&lt;/span&gt;
&lt;span class="cm"&gt; * message, it just ensures that OPAL completely flushes the console buffer.&lt;/span&gt;
&lt;span class="cm"&gt; */&lt;/span&gt;
&lt;span class="k"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;force_opal_console_flush&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;kmsg_dumper&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;dumper&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;                                     &lt;/span&gt;&lt;span class="k"&gt;enum&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kmsg_dump_reason&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kt"&gt;int64_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ret&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="cm"&gt;/*&lt;/span&gt;
&lt;span class="cm"&gt;     * Outside of a panic context the pollers will continue to run,&lt;/span&gt;
&lt;span class="cm"&gt;     * so we don&amp;#39;t need to do any special flushing.&lt;/span&gt;
&lt;span class="cm"&gt;     */&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reason&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;KMSG_DUMP_PANIC&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;opal_check_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;OPAL_CONSOLE_FLUSH&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;ret&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;opal_console_flush&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ret&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;OPAL_UNSUPPORTED&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ret&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;OPAL_PARAMETER&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="cm"&gt;/* Incrementally flush until there&amp;#39;s nothing left */&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;while&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;opal_console_flush&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;OPAL_SUCCESS&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="cm"&gt;/*&lt;/span&gt;
&lt;span class="cm"&gt;         * If OPAL_CONSOLE_FLUSH is not implemented in the firmware,&lt;/span&gt;
&lt;span class="cm"&gt;         * the console can still be flushed by calling the polling&lt;/span&gt;
&lt;span class="cm"&gt;         * function enough times to flush the buffer.  We don&amp;#39;t know&lt;/span&gt;
&lt;span class="cm"&gt;         * how much output still needs to be flushed, but we can be&lt;/span&gt;
&lt;span class="cm"&gt;         * generous since the kernel is in panic and doesn&amp;#39;t need&lt;/span&gt;
&lt;span class="cm"&gt;         * to do much else.&lt;/span&gt;
&lt;span class="cm"&gt;         */&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;printk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;KERN_NOTICE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;opal: OPAL_CONSOLE_FLUSH missing.&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="n"&gt;opal_poll_events&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You can find the full code in-tree &lt;a href="https://github.com/torvalds/linux/blob/master/arch/powerpc/platforms/powernv/opal-kmsg.c"&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;And thus, panic messages now roam free 'cross the countryside, causing developer frustration around the world.  At least now they know why they're frustrated.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Russell Currey</dc:creator><pubDate>Mon, 15 Feb 2016 14:22:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2016-02-15:/blog/2016/02/15/panic/</guid><category>OpenPOWER</category></item><item><title>Evolving into a systems programmer</title><link>https://sthbrx.github.io/blog/2015/11/06/evolving-into-a-systems-programmer/</link><description>&lt;p&gt;In a previous life I tutored first year computing. The university I
attended had a policy of using C to introduce first years to programming.
One of the most rewarding aspects of teaching is opening doors of
possibility to people by sharing my knowledge.&lt;/p&gt;
&lt;p&gt;Over the years I had a mixture of computer science or computer engineering
students as well as other disciplines of engineering who were required to
learn the basics (notably electrical and mechanical). Each class was
different and the initial knowledge always varied greatly. The beauty of
teaching C meant that there was never someone who truly knew it all, heck,
I didn't and still don't. The other advantage of teaching C is that I could
very quickly spot the hackers, the shy person at the back of the room who's
eyes light up when you know you've correctly explained pointers (to them
anyway) or when asked "What happens if you use a negative index into an
array" and the smile they would make upon hearing "What do you think happens".&lt;/p&gt;
&lt;p&gt;Right there I would see the makings of a hacker, and this post is dedicated
to you or to anyone who wants to be a hacker. I've been asked "What did you
do to get where you are?", "How do I get into Linux?" (vague much) at
careers fairs. I never quite know what to say, here goes a braindump.&lt;/p&gt;
&lt;p&gt;Start with the basics, one of the easiest way we tested the first years was
to tell them they can't use parts of libc. That was a great exam, taking
aside those who didn't read the question and used &lt;code&gt;strlen()&lt;/code&gt; when they were
explicitly told they couldn't &lt;code&gt;#include &amp;lt;string.h&amp;gt;&lt;/code&gt; a true hacker doesn't
need libc, understand it won't always be there. I thought of this example
because only two weeks ago I was writing code in an environment where I
didn't have libc. Ok sure, if you've got it, use it, just don't crumble
when you don't. Oh how I wish I could have told those students who argued
that it was a pointless question that they were objectively wrong.&lt;/p&gt;
&lt;p&gt;Be a fan of assembly, don't be afraid of it, it doesn't bite and it can be
a lot of fun. I wouldn't encourage you to dive right into the PowerISA,
it's intense but perhaps understand the beauty of GCC, know what it's doing
for you. There is a variety of little 8 bit processors you can play with
these days.&lt;/p&gt;
&lt;p&gt;At all levels of my teaching I saw almost everyone get something which
'worked', and that's fine, it probably does but I'm here to tell you that
it doesn't work until you know why it works. I'm all for the 'try it and
see' approach but once you've tried it you have to explain why the
behaviour changed otherwise you didn't fix it. As an extension to that,
know how your tools work, I don't think anyone would expect you to be able
to write tools to the level of complexity of GCC or GDB or Valgrind but
have a rough idea as to how they achieve their goals.&lt;/p&gt;
&lt;p&gt;A hacker is paranoid, yes, &lt;code&gt;malloc()&lt;/code&gt; fails. Linux might just decide now
isn't a good time for you to &lt;code&gt;open()&lt;/code&gt; and your &lt;code&gt;fopen()&lt;/code&gt; calling function had
better be cool with that. A hacker also doesn't rely on the kindness of the
operating system theres an &lt;code&gt;munmap()&lt;/code&gt; for a reason. Nor should you even
completely trust it, what are you leaving around in memory?&lt;/p&gt;
&lt;p&gt;Above all do a it for the fun of it, so many of my students asked how I
knew everything I knew (I was only a year ahead of them in my first year of
teaching) and put simply, write code on a Saturday night.&lt;/p&gt;
&lt;p&gt;None of these things do or don't make you a hacker, being a hacker is a
frame of mind and a way of thinking but all of the above helps.&lt;/p&gt;
&lt;p&gt;Unfortunately there isn't a single path, I might even say it is a path that
chooses you. Odds are you're here because you approached me at some point
and asked me one of those questions I never quite know how to answer.
Perhaps this is the path, at the very least you're asking questions and
approaching people. I'm hope I did on the day, but once again, all the very
best with your endeavours into the future&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Cyril Bur</dc:creator><pubDate>Fri, 06 Nov 2015 11:13:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2015-11-06:/blog/2015/11/06/evolving-into-a-systems-programmer/</guid><category>Education</category><category>education</category><category>offtopic</category></item><item><title>What the HILE is this?</title><link>https://sthbrx.github.io/blog/2015/11/03/what-the-hile-is-this/</link><description>&lt;p&gt;One of the cool features of POWER8 processors is the ability to run in either big- or little-endian mode. Several distros are already available in little-endian, but up until recently Petitboot has remained big-endian. While it has no effect on the OS, building Petitboot little-endian has its advantages, such as making support for vendor tools easier.
So it should just be a matter of compiling Petitboot LE right? Well...&lt;/p&gt;
&lt;h3&gt;Switching Endianess&lt;/h3&gt;
&lt;p&gt;Endianess, and several other things besides, are controlled by the Machine State Register (MSR). Each processor in a machine has an MSR, and each bit of the MSR controls some aspect of the processor such as 64-bit mode or enabling interrupts. To switch endianess we set the LE bit (63) to 1.&lt;/p&gt;
&lt;p&gt;When a processor first starts up it defaults to big-endian (bit 63 = 0). However the processor doesn't actually know the endianess of the kernel code it is about to execute - either it is big-endian and everything is fine, or it isn't and the processor will very quickly try to execute an illegal instruction.&lt;/p&gt;
&lt;p&gt;The solution to this is an amazing little snippet of code in &lt;a href="https://github.com/torvalds/linux/blob/master/arch/powerpc/boot/ppc_asm.h#L65"&gt;arch/powerpc/boot/ppc_asm.h&lt;/a&gt; (follow the link to see some helpful commenting):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;#define FIXUP_ENDIAN&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;tdi&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x48&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="n"&gt;$&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;36&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x05009f42&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xa602487d&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x1c004a39&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xa600607d&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x01006b69&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xa6035a7d&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xa6037b7d&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x2400004c&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;By some amazing coincidence if you take the opcode for &lt;code&gt;tdi 0, 0, 0x48&lt;/code&gt; and flip the order of the bytes it forms the opcode for &lt;code&gt;b . + 8&lt;/code&gt;. So if the kernel is big-endian, the processor will jump to the next instruction after this snippet. However if the kernel is little-endian we execute the next 8 instructions. These are written in reverse so that if the processor isn't in the right endian it interprets them backwards, executing the instructions shown in the linked comments above, resulting in MSR&lt;sub&gt;LE&lt;/sub&gt; being set to 1.&lt;/p&gt;
&lt;p&gt;When booting a little-endian kernel all of the above works fine - but there is a problem for Petitboot that will become apparent a little further down...&lt;/p&gt;
&lt;h3&gt;Petitboot's Secret Sauce&lt;/h3&gt;
&lt;p&gt;The main feature of Petitboot is that it is a full (but small!) Linux kernel and userspace which scans all available devices and presents possible boot options. To boot an available operating system Petitboot needs to start executing the OS's kernel, which it accomplishes via &lt;a href="https://en.wikipedia.org/wiki/Kexec"&gt;kexec&lt;/a&gt;. Simply speaking kexec loads the target kernel into memory, shuts the current system down most of the way, and at the last moment sets the instruction pointer to the start of the target kernel. From there it's like booting any other kernel, including the FIXUP_ENDIAN section above.&lt;/p&gt;
&lt;h3&gt;We've Booted! Wait...&lt;/h3&gt;
&lt;p&gt;So our LE Petitboot kernel boots fine thanks to FIXUP_ENDIAN, we kexec into some other kernel.. and everything falls to pieces.&lt;br&gt;
The problem is we've unwittingly changed one of the assumptions of booting a kernel; namely that MSR&lt;sub&gt;LE&lt;/sub&gt; defaults to zero. When kexec-ing from an LE kernel we start executing the next kernel in LE mode. This itself is ok, the FIXUP_ENDIAN macro will handle the switch if needed. The problem is that the FIXUP_ENDIAN macro is relatively recent, first entering the kernel in early 2014. So if we're booting, say, an old Fedora 19 install with a v3.9 kernel - things go very bad, very quickly.&lt;/p&gt;
&lt;h3&gt;Fix #1&lt;/h3&gt;
&lt;p&gt;The solution seems pretty straightforward: find where we jump into the next kernel, and just before that make sure we reset the LE bit in the MSR. That's exactly what &lt;a href="https://github.com/antonblanchard/kexec-lite/commit/150b14e76a4b51f865b929ad9a9bf4133e2d3af7"&gt;this patch&lt;/a&gt; to kexec-lite does.&lt;br&gt;
That worked up until I tested on a machine with more than one CPU. Remembering that the MSR is processor-specific, we also have to &lt;a href="https://github.com/torvalds/linux/commit/ffebf5f391dfa9da3e086abad3eef7d3e5300249"&gt;reset the endianess of each secondary CPU&lt;/a&gt;&lt;br&gt;
Now things are looking good! All the CPUs are reset to big-endian, the target kernel boots fine, and then... 'recursive interrupts?!'&lt;/p&gt;
&lt;h3&gt;HILE&lt;/h3&gt;
&lt;p&gt;Skipping the debugging process that led to this (hint: &lt;a href="https://www.flamingspork.com/blog/2014/12/03/running-skiboot-opal-on-the-power8-simulator/"&gt;mambo&lt;/a&gt; is actually a pretty cool tool), these were the sequence of steps leading up to the problem:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Little-endian Petitboot kexecs into a big-endian kernel&lt;/li&gt;
&lt;li&gt;All CPUs are reset to big-endian&lt;/li&gt;
&lt;li&gt;The big-endian kernel begins to boot successfully&lt;/li&gt;
&lt;li&gt;Somewhere in the device-tree parsing code we take an exception&lt;/li&gt;
&lt;li&gt;Execution jumps to the exception handler at &lt;a href="https://github.com/torvalds/linux/blob/master/arch/powerpc/kernel/exceptions-64s.S#L199"&gt;0x300&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;I notice that MSR&lt;sub&gt;LE&lt;/sub&gt; is set to 1&lt;/li&gt;
&lt;li&gt;WHAT WHY IS THE LE BIT IN THE MSR SET TO 1&lt;/li&gt;
&lt;li&gt;We fail to read the first instruction at 0x300 because it's written in big-endian, so we jump to the exception handler at 0x300... oh no.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;And then we very busily execute nothing until the machine is killed. I spend some time staring incredulously at my screen, then appeal to a &lt;a href="https://github.com/torvalds/linux/blob/master/MAINTAINERS"&gt;higher authority&lt;/a&gt; who replies with "What is the HILE set to?"  &lt;/p&gt;
&lt;p&gt;..the WHAT?&lt;br&gt;
Cracking open the &lt;a href="https://www.power.org/documentation/power-isa-v-2-07b/"&gt;PowerISA&lt;/a&gt; reveals this tidbit:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;The Hypervisor Interrupt Little-Endian (HILE) bit is a bit
in an implementation-dependent register or similar
mechanism. The contents of the HILE bit are copied
into MSR&lt;sub&gt;LE&lt;/sub&gt; by interrupts that set MSR&lt;sub&gt;HV&lt;/sub&gt; to 1 (see Section
6.5), to establish the Endian mode for the interrupt
handler. The HILE bit is set, by an implementation-dependent
method, during system initialization,
and cannot be modified after system initialization.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;To be fair, there are use cases for taking exceptions in a different endianess. The problem is that while HILE gets switched on when setting MSR&lt;sub&gt;LE&lt;/sub&gt; to 1, it &lt;em&gt;doesn't&lt;/em&gt; get turned off when MSR&lt;sub&gt;LE&lt;/sub&gt; is set to zero. In particular the line "...cannot be modified after system initialization." led to a fair amount of hand wringing from myself and whoever would listen; if we can't reset the HILE bit, we simply can't use little-endian kernels for Petitboot.  &lt;/p&gt;
&lt;p&gt;Luckily while on some other systems the machinations of the firmware might be a complete black box, Petitboot runs on OPAL systems - which means the firmware source is &lt;a href="https://github.com/open-power/skiboot"&gt;right here&lt;/a&gt;. In particular we can see here the OPAL call to &lt;a href="https://github.com/open-power/skiboot/blob/master/core/cpu.c#L702"&gt;opal_reinit_cpus&lt;/a&gt; which among other things resets the HILE bit.&lt;br&gt;
This is actually what turns on the HILE bit in the first place, and is meant to be called early on in boot since it also clobbers a large amount of state. Luckily for us we don't need to hold onto any state since we're about to jump into a new kernel. We just need to choose an appropriate place where we can be sure we won't take an exception before we get into the next kernel: thus the &lt;a href="https://github.com/torvalds/linux/commit/e72bb8a5a884d022231149d407653923a1d79e53"&gt;final patch to support PowerNV machines.&lt;/a&gt;&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Tue, 03 Nov 2015 15:02:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2015-11-03:/blog/2015/11/03/what-the-hile-is-this/</guid><category>Petitboot</category><category>petitboot</category><category>p8</category><category>goodposts</category><category>autoboot</category><category>realcontent</category><category>kexec</category><category>kernel</category></item><item><title>Docker: Just Stop Using AUFS</title><link>https://sthbrx.github.io/blog/2015/10/30/docker-just-stop-using-aufs/</link><description>&lt;p&gt;Docker's default storage driver on most Ubuntu installs is AUFS.&lt;/p&gt;
&lt;p&gt;Don't use it. Use Overlay instead. Here's why.&lt;/p&gt;
&lt;p&gt;First, some background. I'm testing the performance of the basic LAMP
stack on POWER. (LAMP is Linux + Apache + MySQL/MariaDB + PHP, by the
way.) To do more reliable and repeatable tests, I do my builds and
tests in Docker containers. (See &lt;a href="/blog/2015/10/12/a-tale-of-two-dockers/"&gt;my previous post&lt;/a&gt; for more info.)&lt;/p&gt;
&lt;p&gt;Each test downloads the source of Apache, MariaDB and PHP, and builds
them. This should be quick: the POWER8 system I'm building on has 160
hardware threads and 128 GB of memory. But I was finding that it was
only just keeping pace with a 2 core Intel VM on BlueMix.&lt;/p&gt;
&lt;p&gt;Why? Well, my first point of call was to observe a compilation under
&lt;code&gt;top&lt;/code&gt;. The header is below.&lt;/p&gt;
&lt;p&gt;&lt;img alt="top header, showing over 70 percent of CPU time spent in the kernel" src="/images/dja/aufs/top-bad.png"&gt;&lt;/p&gt;
&lt;p&gt;Over 70% of CPU time is spent in the kernel?! That's weird. Let's dig
deeper.&lt;/p&gt;
&lt;p&gt;My next port of call for analysis of CPU-bound workloads is
&lt;code&gt;perf&lt;/code&gt;. &lt;code&gt;perf top&lt;/code&gt; reports astounding quantities of time in
spin-locks:&lt;/p&gt;
&lt;p&gt;&lt;img alt="display from perf top, showing 80 percent of time in a spinlock" src="/images/dja/aufs/perf-top-spinlock.png"&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;perf top -g&lt;/code&gt; gives us some more information: the time is in system
calls. &lt;code&gt;open()&lt;/code&gt; and &lt;code&gt;stat()&lt;/code&gt; are the key culprits, and we can see a
number of file system functions are in play in the call-chains of the
spinlocks.&lt;/p&gt;
&lt;p&gt;&lt;img alt="display from perf top -g, showing syscalls and file ops" src="/images/dja/aufs/perf-top-syscalls.png"&gt;&lt;/p&gt;
&lt;p&gt;Why are open and stat slow? Well, I know that the files are on an AUFS
mount. (&lt;code&gt;docker info&lt;/code&gt; will tell you what you're using if you're not
sure.) So, being something of a kernel hacker, I set out to find out
why. This did not go well. AUFS isn't upstream, it's a separate patch
set. Distros have been trying to deprecate it for years. Indeed, RHEL
doesn't ship it. (To it's credit, Docker seems to be trying to move
away from it.)&lt;/p&gt;
&lt;p&gt;Wanting to avoid the minor nightmare that is an out-of-tree patchset,
I looked at other storage drivers for Docker. &lt;a href="https://jpetazzo.github.io/assets/2015-03-03-not-so-deep-dive-into-docker-storage-drivers.html"&gt;This presentation is particularly good.&lt;/a&gt;
My choices are pretty simple: AUFS, btrfs, device-mapper or
Overlay. Overlay was an obvious choice: it doesn't need me to set up
device mapper on a cloud VM, or reformat things as btrfs.&lt;/p&gt;
&lt;p&gt;It's also easy to set up on Ubuntu:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;export/save any docker containers you care about.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;add &lt;code&gt;--storage-driver=overlay&lt;/code&gt; option to &lt;code&gt;DOCKER_OPTS&lt;/code&gt; in &lt;code&gt;/etc/default/docker&lt;/code&gt;, and restart docker (&lt;code&gt;service docker restart&lt;/code&gt;)&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;import/load the containters you exported&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;verify that things work, then clear away your old storage directory (&lt;code&gt;/var/lib/docker/aufs&lt;/code&gt;). &lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Having moved my base container across, I set off another build.&lt;/p&gt;
&lt;p&gt;The first thing I noticed is that images are much slower to create with Overlay. But once that finishes, and a compile starts, things run much better:&lt;/p&gt;
&lt;p&gt;&lt;img alt="top, showing close to zero system time, and around 90 percent user time" src="/images/dja/aufs/top-good.png"&gt;&lt;/p&gt;
&lt;p&gt;The compiles went from taking painfully long to astonishingly fast. Winning.&lt;/p&gt;
&lt;p&gt;So in conclusion:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;If you use Docker for something that involves open()ing or stat()ing files&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;If you want your machine to do real work, rather than spin in spinlocks&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;If you want to use code that's upstream and thus much better supported&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;If you want something less disruptive than the btrfs or dm storage drivers&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;...then drop AUFS and switch to Overlay today.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Fri, 30 Oct 2015 13:30:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2015-10-30:/blog/2015/10/30/docker-just-stop-using-aufs/</guid><category>Performance</category><category>aufs</category><category>overlay</category><category>performance</category></item><item><title>A tale of two Dockers</title><link>https://sthbrx.github.io/blog/2015/10/12/a-tale-of-two-dockers/</link><description>&lt;p&gt;(This was published in an internal technical journal last week, and is now being published here. If you already know what Docker is, feel free to skim the first half.)&lt;/p&gt;
&lt;p&gt;Docker seems to be the flavour of the month in IT. Most attention is focussed on using Docker for the deployment of production services. But that's not all Docker is good for. Let's explore Docker, and two ways I use it as a software developer.&lt;/p&gt;
&lt;p&gt;Docker: what is it?&lt;/p&gt;
&lt;p&gt;Docker is essentially a set of tools to deal with &lt;em&gt;containers&lt;/em&gt; and &lt;em&gt;images&lt;/em&gt;. &lt;/p&gt;
&lt;p&gt;To make up an artificial example, say you are developing a web app. You first build an &lt;em&gt;image&lt;/em&gt;: a file system which contains the app, and some associated metadata. The app has to run on something, so you also install things like Python or Ruby and all the necessary libraries, usually by installing a minimal Ubuntu and any necessary packages.&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1"&gt;1&lt;/a&gt;&lt;/sup&gt; You then run the image inside an isolated environment called a &lt;em&gt;container&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;You can have multiple containers running the same image, (for example, your web app running across a fleet of servers) and the containers don't affect each other.  Why? Because Docker is designed around the concept of &lt;em&gt;immutability&lt;/em&gt;. Containers can write to the image they are running, but the changes are specific to that container, and aren't preserved beyond the life of the container.&lt;sup id="fnref:2"&gt;&lt;a class="footnote-ref" href="#fn:2"&gt;2&lt;/a&gt;&lt;/sup&gt; Indeed, once built, images can't be changed at all, only rebuilt from scratch.&lt;/p&gt;
&lt;p&gt;However, as well as enabling you to easily run multiple copies, another upshot of immutability is that if your web app allows you to upload photos, and you restart the container, your photos will be gone. Your web app needs to be designed to store all of the data outside of the container, sending it to a dedicated database or object store of some sort.&lt;/p&gt;
&lt;p&gt;Making your application Docker friendly is significantly more work than just spinning up a virtual machine and installing stuff. So what does all this extra work get you? Three main things: isolation, control and, as mentioned, immutability. &lt;/p&gt;
&lt;p&gt;&lt;em&gt;Isolation&lt;/em&gt; makes containers easy to migrate and deploy, and easy to update. Once an image is built, it can be copied to another system and launched. Isolation also makes it easy to update software your app depends on: you rebuild the image with software updates, and then just deploy it. You don't have to worry about service A relying on version X of a library while service B depends on version Y; it's all self contained. &lt;/p&gt;
&lt;p&gt;&lt;em&gt;Immutability&lt;/em&gt; also helps with upgrades, especially when deploying them across multiple servers. Normally, you would upgrade your app on each server, and have to make sure that every server gets all the same sets of updates. With Docker, you don't upgrade a running container. Instead, you rebuild your Docker image and re-deploy it, and you then know that the same version of everything is running everywhere. This immutability also guards against the situation where you have a number of different servers that are all special snowflakes with their own little tweaks, and you end up with a fractal of complexity.&lt;/p&gt;
&lt;p&gt;Finally, Docker offers a lot of &lt;em&gt;control&lt;/em&gt; over containers, and for a low performance penalty. Docker containers can have their CPU, memory and network controlled easily, without the overhead of a full virtual machine. This makes it an attractive solution for running untrusted executables.&lt;sup id="fnref:3"&gt;&lt;a class="footnote-ref" href="#fn:3"&gt;3&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;p&gt;As an aside: despite the hype, very little of this is actually particularly new. Isolation and control are not new problems. All Unixes, including Linux, support 'chroots'. The name comes from “change root”: the system call changes the processes idea of what the file system root is, making it impossible for it to access things outside of the new designated root directory.  FreeBSD has jails, which are more powerful, Solaris has Zones, and AIX has WPARs. Chroots are fast and low overhead. However, they offer much lower ability to control the use of system resources. At the other end of the scale, virtual machines (which have been around since ancient IBM mainframes) offer isolation much better than Docker, but with a greater performance hit.&lt;/p&gt;
&lt;p&gt;Similarly, immutability isn't really new: Heroku and AWS Spot Instances are both built around the model that you get resources in a known, consistent state when you start, but in both cases your changes won't persist. In the development world, modern CI systems like Travis CI also have this immutable or disposable model – and this was originally built on VMs. Indeed, with a little bit of extra work, both chroots and VMs can give the same immutability properties that Docker gives.&lt;/p&gt;
&lt;p&gt;The control properties that Docker provides are largely as a result of leveraging some Linux kernel concepts, most notably something called namespaces.&lt;/p&gt;
&lt;p&gt;What Docker does well is not something novel, but the engineering feat of bringing together fine-grained control, isolation and immutability, and – importantly – a tool-chain that is easier to use than any of the alternatives. Docker's tool-chain eases a lot of pain points with regards to building containers: it's vastly simpler than chroots, and easier to customise than most VM setups. Docker also has a number of engineering tricks to reduce the disk space overhead of isolation.&lt;/p&gt;
&lt;p&gt;So, to summarise: Docker provides a toolkit for isolated, immutable, finely controlled containers to run executables and services.&lt;/p&gt;
&lt;h2&gt;Docker in development: why?&lt;/h2&gt;
&lt;p&gt;I don't run network services at work; I do performance work. So how do I use Docker?&lt;/p&gt;
&lt;p&gt;There are two things I do with Docker: I build PHP 5, and do performance regression testing on PHP 7. They're good case studies of how isolation and immutability provide real benefits in development and testing, and how the Docker tool chain makes life a lot nicer that previous solutions.&lt;/p&gt;
&lt;h3&gt;PHP 5 builds&lt;/h3&gt;
&lt;p&gt;I use the &lt;em&gt;isolation&lt;/em&gt; that Docker provides to make building PHP 5 easier. PHP 5 depends on an old version of Bison, version 2. Ubuntu and Debian long since moved to version 3. There are a few ways I could have solved this:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;I could just install the old version directly on my system in &lt;code&gt;/usr/local/&lt;/code&gt;, and hope everything still works and nothing else picks up Bison 2 when it needs Bison 3. Or I could install it somewhere else and remember to change my path correctly before I build PHP 5.&lt;/li&gt;
&lt;li&gt;I could roll a chroot by hand. Even with tools like debootstrap and schroot, working in chroots is a painful process.&lt;/li&gt;
&lt;li&gt;I could spin up a virtual machine on one of our development boxes and install the old version on that. That feels like overkill: why should I need to run an entire operating system? Why should I need to copy my source tree over the network to build it?&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Docker makes it easy to have a self-contained environment that has Bison 2 built from source, and to build my latest source tree in that environment. Why is Docker so much easier?&lt;/p&gt;
&lt;p&gt;Firstly, Docker allows me to base my container on an existing container, and there's an online library of containers to build from.&lt;sup id="fnref:4"&gt;&lt;a class="footnote-ref" href="#fn:4"&gt;4&lt;/a&gt;&lt;/sup&gt; This means I don't have to roll a base image with &lt;code&gt;debootstrap&lt;/code&gt; or the RHEL/CentOS/Fedora equivalent.&lt;/p&gt;
&lt;p&gt;Secondly, unlike a chroot build process, which ultimately is just copying files around, a docker build process includes the ability to both copy files from the host and &lt;em&gt;run commands&lt;/em&gt; in the context of the image. This is defined in a file called a &lt;code&gt;Dockerfile&lt;/code&gt;, and is kicked off by a single command: &lt;code&gt;docker build&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;So, my PHP 5 build container loads an Ubuntu Vivid base container, uses apt-get to install the compiler, tool-chain and headers required to build PHP 5, then installs old bison from source, copies in the PHP source tree, and builds it. The vast majority of this process – the installation of the compiler, headers and bison, can be cached, so they don't have to be downloaded each time. And once the container finishes building, I have a fully built PHP interpreter ready for me to interact with.&lt;/p&gt;
&lt;p&gt;I do, at the moment, rebuild PHP 5 from scratch each time. This is a bit sub-optimal from a performance point of view. I could alleviate this with a Docker volume, which is a way of sharing data persistently between a host and a guest, but I haven't been sufficiently bothered by the speed yet. However, Docker volumes are also quite fiddly, leading to the development of tools like &lt;code&gt;docker compose&lt;/code&gt; to deal with them. They also are prone to subtle and difficult to debug permission issues.&lt;/p&gt;
&lt;h3&gt;PHP 7 performance regression testing&lt;/h3&gt;
&lt;p&gt;The second thing I use docker for takes advantage of the throwaway nature of docker environments to prevent cross-contamination.&lt;/p&gt;
&lt;p&gt;PHP 7 is the next big version of PHP, slated to be released quite soon. I care about how that runs on POWER, and I preferably want to know if it suddenly deteriorates (or improves!). I use Docker to build a container with a daily build of PHP 7, and then I run a benchmark in it. This doesn't give me a particularly meaningful absolute number, but it allows me to track progress over time. Building it inside of Docker means that I can be sure that nothing from old runs persists into new runs, thus giving me more reliable data. However, because I do want the timing data I collect to persist, I send it out of the container over the network.&lt;/p&gt;
&lt;p&gt;I've now been collecting this data for almost 4 months, and it's plotted below, along with a 5-point moving average. The most notable feature of the graph is a the drop in benchmark time at about the middle. Sure enough, if you look at the PHP repository, you will see that a set of changes to improve PHP performance were merged on July 29: changes submitted by our very own Anton Blanchard.&lt;sup id="fnref:5"&gt;&lt;a class="footnote-ref" href="#fn:5"&gt;5&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;p&gt;&lt;img alt="Graph of PHP 7 performance over time" src="/images/dja/php7-perf.png"&gt;&lt;/p&gt;
&lt;h2&gt;Docker pain points&lt;/h2&gt;
&lt;p&gt;Docker provides a vastly improved experience over previous solutions, but there are still a few pain points. For example:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;Docker was apparently written by people who had no concept that platforms other than x86 exist. This leads to major issues for cross-architectural setups. For instance, Docker identifies images by a name and a revision. For example, &lt;code&gt;ubuntu&lt;/code&gt; is the name of an image, and &lt;code&gt;15.04&lt;/code&gt; is a revision. There's no ability to specify an architecture. So, how you do specify that you want, say, a 64-bit, little-endian PowerPC build of an image versus an x86 build? There have been a couple of approaches, both of which are pretty bad. You could name the image differently: say &lt;code&gt;ubuntu_ppc64le&lt;/code&gt;. You can also just cheat and override the &lt;code&gt;ubuntu&lt;/code&gt; name with an architecture specific version. Both of these break some assumptions in the Docker ecosystem and are a pain to work with.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Image building is incredibly inflexible. If you have one system that requires a proxy, and one that does not, you need different Dockerfiles. As far as I can tell, there are no simple ways to hook in any changes between systems into a generic Dockerfile. This is largely by design, but it's still really annoying when you have one system behind a firewall and one system out on the public cloud (as I do in the PHP 7 setup).&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Visibility into a Docker server is poor. You end up with lots of different, anonymous images and dead containers, and you end up needing scripts to clean them up. It's not clear what Docker puts on your file system, or where, or how to interact with it.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Docker is still using reasonably new technologies. This leads to occasional weird, obscure and difficult to debug issues.&lt;sup id="fnref:6"&gt;&lt;a class="footnote-ref" href="#fn:6"&gt;6&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;h2&gt;Final words&lt;/h2&gt;
&lt;p&gt;Docker provides me with a lot of useful tools in software development: both in terms of building and testing. Making use of it requires a certain amount of careful design thought, but when applied thoughtfully it can make life significantly easier.&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;There's some debate about how much stuff from the OS installation you should be using. You need to have key dynamic libraries available, but I would argue that you shouldn't be running long running processes other than your application. You shouldn't, for example, be running a SSH daemon in your container. (The one exception is that you must handle orphaned child processes appropriately: see &lt;a href="https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/"&gt;https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/&lt;/a&gt;) Considerations like debugging and monitoring the health of docker containers mean that this point of view is not universally shared.&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:2"&gt;
&lt;p&gt;Why not simply make them read only? You may be surprised at how many things break when running on a read-only file system. Things like logs and temporary files are common issues.&amp;#160;&lt;a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:3"&gt;
&lt;p&gt;It is, however, easier to escape a Docker container than a VM. In Docker, an untrusted executable only needs a kernel exploit to get to root on the host, whereas in a VM you need a guest-to-host vulnerability, which are much rarer.&amp;#160;&lt;a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:4"&gt;
&lt;p&gt;Anyone can upload an image, so this does require running untrusted code from the Internet. Sadly, this is a distinctly retrograde step when compared to the process of installing binary packages in distros, which are all signed by a distro's private key.&amp;#160;&lt;a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:5"&gt;
&lt;p&gt;See &lt;a href="https://github.com/php/php-src/pull/1326"&gt;https://github.com/php/php-src/pull/1326&lt;/a&gt;&amp;#160;&lt;a class="footnote-backref" href="#fnref:5" title="Jump back to footnote 5 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:6"&gt;
&lt;p&gt;I hit this last week: &lt;a href="https://github.com/docker/docker/issues/16256"&gt;https://github.com/docker/docker/issues/16256&lt;/a&gt;, although maybe that's my fault for running systemd on my laptop.&amp;#160;&lt;a class="footnote-backref" href="#fnref:6" title="Jump back to footnote 6 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Mon, 12 Oct 2015 14:14:00 +1100</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2015-10-12:/blog/2015/10/12/a-tale-of-two-dockers/</guid><category>Development</category><category>php</category><category>performance</category></item><item><title>Running ppc64le_hello on real hardware</title><link>https://sthbrx.github.io/blog/2015/06/03/ppc64le-hello-on-real-hardware/</link><description>&lt;p&gt;So today I saw &lt;a href="https://github.com/andreiw/ppc64le_hello"&gt;Freestanding “Hello World” for OpenPower&lt;/a&gt; on &lt;a href="https://news.ycombinator.com/item?id=9649490"&gt;Hacker News&lt;/a&gt;. Sadly Andrei hadn't been able to test it on real hardware, so I set out to get it running on a real OpenPOWER box. Here's what I did.&lt;/p&gt;
&lt;p&gt;Firstly, clone the repo, and, as mentioned in the README, comment out &lt;code&gt;mambo_write&lt;/code&gt;. Build it.&lt;/p&gt;
&lt;p&gt;Grab &lt;a href="https://github.com/open-power/op-build"&gt;op-build&lt;/a&gt;, and build a Habanero defconfig. To save yourself a fair bit of time, first edit &lt;code&gt;openpower/configs/habanero_defconfig&lt;/code&gt; to answer &lt;code&gt;n&lt;/code&gt; about a custom kernel source. That'll save you hours of waiting for git.&lt;/p&gt;
&lt;p&gt;This will build you a PNOR that will boot a linux kernel with Petitboot. This is almost what you want: you need Skiboot, Hostboot and a bunch of the POWER specific bits and bobs, but you don't actually want the Linux boot kernel.&lt;/p&gt;
&lt;p&gt;Then, based on &lt;code&gt;op-build/openpower/package/openpower-pnor/openpower-pnor.mk&lt;/code&gt;, we look through the output of &lt;code&gt;op-build&lt;/code&gt; for a  &lt;code&gt;create_pnor_image.pl&lt;/code&gt; command, something like this monstrosity:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;PATH="/scratch/dja/public/op-build/output/host/bin:/scratch/dja/public/op-build/output/host/sbin:/scratch/dja/public/op-build/output/host/usr/bin:/scratch/dja/public/op-build/output/host/usr/sbin:/home/dja/bin:/home/dja/bin:/home/dja/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/openpower/common/x86_64/bin" /scratch/dja/public/op-build/output/build/openpower-pnor-ed1682e10526ebd85825427fbf397361bb0e34aa/create_pnor_image.pl -xml_layout_file /scratch/dja/public/op-build/output/build/openpower-pnor-ed1682e10526ebd85825427fbf397361bb0e34aa/"defaultPnorLayoutWithGoldenSide.xml" -pnor_filename /scratch/dja/public/op-build/output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/pnor/"habanero.pnor" -hb_image_dir /scratch/dja/public/op-build/output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/hostboot_build_images/ -scratch_dir /scratch/dja/public/op-build/output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/openpower_pnor_scratch/ -outdir /scratch/dja/public/op-build/output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/pnor/ -payload /scratch/dja/public/op-build/output/images/"skiboot.lid" -bootkernel /scratch/dja/public/op-build/output/images/zImage.epapr -sbe_binary_filename "venice_sbe.img.ecc" -sbec_binary_filename "centaur_sbec_pad.img.ecc" -wink_binary_filename "p8.ref_image.hdr.bin.ecc" -occ_binary_filename /scratch/dja/public/op-build/output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/occ/"occ.bin" -targeting_binary_filename "HABANERO_HB.targeting.bin.ecc" -openpower_version_filename /scratch/dja/public/op-build/output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/openpower_version/openpower-pnor.version.txt&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;Replace the &lt;code&gt;-bootkernel&lt;/code&gt; arguement with the path to ppc64le_hello, e.g.: &lt;code&gt;-bootkernel /scratch/dja/public/ppc64le_hello/ppc64le_hello&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;Don't forget to move it into place! &lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mv&lt;span class="w"&gt; &lt;/span&gt;output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/pnor/habanero.pnor&lt;span class="w"&gt; &lt;/span&gt;output/images/habanero.pnor
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then we can use skiboot's boot test script (written by Cyril and me, coincidentally!) to flash it.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ppc64le_hello/skiboot/external/boot-tests/boot_test.sh&lt;span class="w"&gt; &lt;/span&gt;-vp&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;hab2-bmc&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;path&lt;span class="w"&gt; &lt;/span&gt;to&amp;gt;/habanero.pnor
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It's not going to get into Petitboot, so just interrupt it after it powers up the box and connect with IPMI. It boots, kinda:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[11012941323,5] INIT: Starting kernel at 0x20010000, fdt at 0x3044db68 (size 0x11cc3)
Hello OPAL!
           _start = 0x20010000
                              _bss   = 0x20017E28
                                                 _stack = 0x20018000
                                                                    _end   = 0x2001A000
                                                                                       KPCR   = 0x20017E50
                                                                                                          OPAL   = 0x30000000
                                                                                                                             FDT    = 0x3044DB68
                                                                                                                                                CPU0 not found?

                                                                                                                                                               Pick your poison:
                                                                                                                                                                                Choices: (MMU = disabled):
                                                                                                                                                                                                             (d) 5s delay
                                                                                                                                                                                                                            (e) test exception
    (n) test nested exception
                                (f) dump FDT
                                               (M) enable MMU
                                                                (m) disable MMU
                                                                                  (t) test MMU
                                                                                                 (u) test non-priviledged code
                                                                                                                                 (I) enable ints
                                                                                                                                                   (i) disable ints
                                                                                                                                                                      (H) enable HV dec
                                                                                                                                                                                          (h) disable HV dec
                                                                                                                                                                                                               (q) poweroff
                                                                                                                                                                                                                             1.42486|ERRL|Dumping errors reported prior to registration
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Yes, it does wrap horribly. However, the big issue here (which you'll have to scroll to see!) is the "CPU0 not found?". Fortunately, we can fix this with a little patch to &lt;code&gt;cpu_init&lt;/code&gt; in main.c to test for a PowerPC POWER8:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;cpu0_node&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fdt_path_offset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fdt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/cpus/cpu@0&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cpu0_node&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;cpu0_node&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fdt_path_offset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fdt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/cpus/PowerPC,POWER8@20&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cpu0_node&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;printk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;CPU0 not found?&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is definitely the &lt;em&gt;wrong&lt;/em&gt; way to do this, but it works for now.&lt;/p&gt;
&lt;p&gt;Now, correcting for weird wrapping, we get:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Hello OPAL!
_start = 0x20010000
_bss   = 0x20017E28
_stack = 0x20018000
_end   = 0x2001A000
KPCR   = 0x20017E50
OPAL   = 0x30000000
FDT    = 0x3044DB68
Assuming default SLB size
SLB size = 0x20
TB freq = 512000000
[13205442015,3] OPAL: Trying a CPU re-init with flags: 0x2
Unrecoverable exception stack top @ 0x20019EC8
HTAB (2048 ptegs, mask 0x7FF, size 0x40000) @ 0x20040000
SLB entries:
1: E 0x8000000 V 0x4000000000000400
EA 0x20040000 -&amp;gt; hash 0x20040 -&amp;gt; pteg 0x200 = RA 0x20040000
EA 0x20041000 -&amp;gt; hash 0x20041 -&amp;gt; pteg 0x208 = RA 0x20041000
EA 0x20042000 -&amp;gt; hash 0x20042 -&amp;gt; pteg 0x210 = RA 0x20042000
EA 0x20043000 -&amp;gt; hash 0x20043 -&amp;gt; pteg 0x218 = RA 0x20043000
EA 0x20044000 -&amp;gt; hash 0x20044 -&amp;gt; pteg 0x220 = RA 0x20044000
EA 0x20045000 -&amp;gt; hash 0x20045 -&amp;gt; pteg 0x228 = RA 0x20045000
EA 0x20046000 -&amp;gt; hash 0x20046 -&amp;gt; pteg 0x230 = RA 0x20046000
EA 0x20047000 -&amp;gt; hash 0x20047 -&amp;gt; pteg 0x238 = RA 0x20047000
EA 0x20048000 -&amp;gt; hash 0x20048 -&amp;gt; pteg 0x240 = RA 0x20048000
...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The weird wrapping seems to be caused by NULLs getting printed to OPAL, but I haven't traced what causes that.&lt;/p&gt;
&lt;p&gt;Anyway, now it largely works! Here's a transcript of some things it can do on real hardware.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Choices: (MMU = disabled):
   (d) 5s delay
   (e) test exception
   (n) test nested exception
   (f) dump FDT
   (M) enable MMU
   (m) disable MMU
   (t) test MMU
   (u) test non-priviledged code
   (I) enable ints
   (i) disable ints
   (H) enable HV dec
   (h) disable HV dec
   (q) poweroff
&amp;lt;press e&amp;gt;
Testing exception handling...
sc(feed) =&amp;gt; 0xFEEDFACE
Choices: (MMU = disabled):
   (d) 5s delay
   (e) test exception
   (n) test nested exception
   (f) dump FDT
   (M) enable MMU
   (m) disable MMU
   (t) test MMU
   (u) test non-priviledged code
   (I) enable ints
   (i) disable ints
   (H) enable HV dec
   (h) disable HV dec
   (q) poweroff
&amp;lt;press t&amp;gt;
EA 0xFFFFFFF000 -&amp;gt; hash 0xFFFFFFF -&amp;gt; pteg 0x3FF8 = RA 0x20010000
mapped 0xFFFFFFF000 to 0x20010000 correctly
EA 0xFFFFFFF000 -&amp;gt; hash 0xFFFFFFF -&amp;gt; pteg 0x3FF8 = unmap
EA 0xFFFFFFF000 -&amp;gt; hash 0xFFFFFFF -&amp;gt; pteg 0x3FF8 = RA 0x20011000
mapped 0xFFFFFFF000 to 0x20011000 incorrectly
EA 0xFFFFFFF000 -&amp;gt; hash 0xFFFFFFF -&amp;gt; pteg 0x3FF8 = unmap
Choices: (MMU = disabled):
   (d) 5s delay
   (e) test exception
   (n) test nested exception
   (f) dump FDT
   (M) enable MMU
   (m) disable MMU
   (t) test MMU
   (u) test non-priviledged code
   (I) enable ints
   (i) disable ints
   (H) enable HV dec
   (h) disable HV dec
   (q) poweroff
&amp;lt;press u&amp;gt;
EA 0xFFFFFFF000 -&amp;gt; hash 0xFFFFFFF -&amp;gt; pteg 0x3FF8 = RA 0x20080000
returning to user code
returning to kernel code
EA 0xFFFFFFF000 -&amp;gt; hash 0xFFFFFFF -&amp;gt; pteg 0x3FF8 = unmap
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I also tested the other functions and they all seem to work. Running non-priviledged code with the MMU on works. Dumping the FDT and the 5s delay both worked, although they tend to stress IPMI a &lt;em&gt;lot&lt;/em&gt;. The delay seems to correspond well with real time as well.&lt;/p&gt;
&lt;p&gt;It does tend to error out and reboot quite often, usually on the menu screen, for reasons that are not clear to me. It usually starts with something entirely uninformative from Hostboot, like this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;1.41801|ERRL|Dumping errors reported prior to registration
  2.89873|Ignoring boot flags, incorrect version 0x0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That may be easy to fix, but again I haven't had time to trace it.&lt;/p&gt;
&lt;p&gt;All in all, it's very exciting to see something come out of the simulator and in to real hardware. Hopefully with the proliferation of OpenPOWER hardware, prices will fall and these sorts of systems will become increasingly accessible to people with cool low level projects like this!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Wed, 03 Jun 2015 12:16:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2015-06-03:/blog/2015/06/03/ppc64le-hello-on-real-hardware/</guid><category>OpenPOWER</category></item><item><title>Petitboot Autoboot Changes</title><link>https://sthbrx.github.io/blog/2015/06/02/autoboot/</link><description>&lt;p&gt;The way autoboot behaves in Petitboot has undergone some significant changes recently, so in order to ward off any angry emails lets take a quick tour of how the new system works.&lt;/p&gt;
&lt;h2&gt;Old &amp;amp; Busted&lt;/h2&gt;
&lt;p&gt;For some context, here is the old (or current depending on what you're running) section of the configuration screen.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Old Autoboot" src="/images/sammj/oldstyle.jpg"&gt;&lt;/p&gt;
&lt;p&gt;This gives you three main options: don't autoboot, autoboot from anything, or autoboot only from a specific device. For the majority of installations this is fine, such as when you have only one default option, or know exactly which device you'll be booting from.&lt;/p&gt;
&lt;p&gt;A side note about default options: it is important to note that not all boot options are valid &lt;em&gt;autoboot&lt;/em&gt; options. A boot option is only considered for auto-booting if it is marked default, eg. 'set default' in GRUB and 'default' in PXE options.&lt;/p&gt;
&lt;h2&gt;New Hotness&lt;/h2&gt;
&lt;p&gt;Below is the new autoboot configuration.&lt;/p&gt;
&lt;p&gt;&lt;img alt="New Autoboot" src="/images/sammj/newstyle.jpg"&gt;&lt;/p&gt;
&lt;p&gt;The new design allows you to specify an ordered list of autoboot options.
The last two of the three buttons are self explanatory - clear the list and autoboot any device, or clear the list completely (no autoboot).&lt;/p&gt;
&lt;p&gt;Selecting the first button, 'Add Device' brings up the following screen:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Device Selection" src="/images/sammj/devices.jpg"&gt;&lt;/p&gt;
&lt;p&gt;From here you can select any device or &lt;em&gt;class&lt;/em&gt; of device to add to the boot order. Once added to the boot order, the order of boot options can be changed with the left and right arrow keys, and removed from the list with the minus key ('-').&lt;/p&gt;
&lt;p&gt;This allows you to create additional autoboot configurations such as "Try to boot from sda2, otherwise boot from the network", or "Give priority to PXE options from eth0, otherwise try any other netboot option".
You can retain the original behaviour by only putting one option into the list (either 'Any Device' or a specific device).&lt;/p&gt;
&lt;p&gt;Presently you can add any option into the list and order them how you like - which means you can do silly things like this:&lt;/p&gt;
&lt;p&gt;&lt;img alt="If you send me a bug report with this in it I may laugh at you" src="/images/sammj/redundant.jpg"&gt;&lt;/p&gt;
&lt;h2&gt;IPMI&lt;/h2&gt;
&lt;p&gt;Slightly prior to the boot order changes Petitboot also received an update to its IPMI handling. IPMI 'bootdev' commands allow you to override the current autoboot configuration remotely, either by specifying a device type to boot (eg. PXE), or by forcing Petitboot to boot into the 'setup' or 'safe' modes. IPMI overrides are either persistent or non-persistent. A non-persistent override will disappear after a successful boot - that is, a successful boot of a boot option, not booting to Petitboot itself - whereas a persistent override will, well, persist!&lt;/p&gt;
&lt;p&gt;If there is an IPMI override currently active, it will appear in the configuration screen with an option to manually clear it:&lt;/p&gt;
&lt;p&gt;&lt;img alt="IPMI Overrides" src="/images/sammj/ipmi.jpg"&gt;&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;That sums up the recent changes to autoboot; a bit more flexibility in assigning priority, and options for more detailed autoboot order if you need it. New versions of Petitboot are backwards compatible and will recognise older saved settings, so updating your firmware won't cause your machines to start booting things at random.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Tue, 02 Jun 2015 08:11:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2015-06-02:/blog/2015/06/02/autoboot/</guid><category>Petitboot</category><category>petitboot</category><category>goodposts</category><category>autoboot</category><category>realcontent</category></item><item><title>Joining the CAPI project</title><link>https://sthbrx.github.io/blog/2015/05/27/joining-the-capi-project/</link><description>&lt;p&gt;(I wrote this blog post a couple of months ago, but it's still quite relevant.)&lt;/p&gt;
&lt;p&gt;Hi, I'm Daniel! I work in OzLabs, part of IBM's Australian Development Labs. Recently, I've been assigned to the CAPI project, and I've been given the opportunity to give you an idea of what this is, and what I'll be up to in the future!&lt;/p&gt;
&lt;h2&gt;What even is CAPI?&lt;/h2&gt;
&lt;p&gt;To help you understand CAPI, think back to the time before computers. We had a variety of machines: machines to build things, to check things, to count things, but they were all specialised --- good at one and only one thing.&lt;/p&gt;
&lt;p&gt;Specialised machines, while great at their intended task, are really expensive to develop. Not only that, it's often impossible to change how they operate, even in very small ways.&lt;/p&gt;
&lt;p&gt;Computer processors, on the other hand, are generalists. They are cheap. They can do a lot of things. If you can break a task down into simple steps, it's easy to get them to do it. The trade-off is that computer processors are incredibly inefficient at everything.&lt;/p&gt;
&lt;p&gt;Now imagine, if you will, that a specialised machine is a highly trained and experienced professional, a computer processor is a hungover university student.&lt;/p&gt;
&lt;p&gt;Over the years, we've tried lots of things to make student faster. Firstly, we gave the student lots of caffeine to make them go as fast as they can. That worked for a while, but you can only give someone so much caffeine before they become unreliable. Then we tried teaming the student up with another student, so they can do two things at once. That worked, so we added more and more students. Unfortunately, lots of tasks can only be done by one person at a time, and team-work is complicated to co-ordinate. We've also recently noticed that some tasks come up often, so we've given them some tools for those specific tasks. Sadly, the tools are only useful for those specific situations.&lt;/p&gt;
&lt;p&gt;Sometimes, what you really need is a professional.&lt;/p&gt;
&lt;p&gt;However, there are a few difficulties in getting a professional to work with uni students. They don't speak the same way; they don't think the same way, and they don't work the same way. You need to teach the uni students how to work with the professional, and vice versa.&lt;/p&gt;
&lt;p&gt;Previously, developing this interface – this connection between a generalist processor and a specialist machine – has been particularly difficult. The interface between processors and these specialised machines – known as &lt;em&gt;accelerators&lt;/em&gt; – has also tended to suffer from bottlenecks and inefficiencies.&lt;/p&gt;
&lt;p&gt;This is the problem CAPI solves. CAPI provides a simpler and more optimised way to interface specialised hardware accelerators with IBM's most recent line of processors, POWER8. It's a common 'language' that the processor and the accelerator talk, that makes it much easier to build the hardware side and easier to program the software side. In our Canberra lab, we're working primarily on the operating system side of this. We are working with some external companies who are building CAPI devices and the optimised software products which use them.&lt;/p&gt;
&lt;p&gt;From a technical point of view, CAPI provides &lt;em&gt;coherent&lt;/em&gt; access to system memory and processor caches, eliminating a major bottleneck in using external devices as accelerators. This is illustrated really well by the following graphic from &lt;a href="https://www.youtube.com/watch?v=4ZyXc12J6FA"&gt;an IBM promotional video&lt;/a&gt;. In the non-CAPI case, you can see there's a lot of data (the little boxes) stalled in the PCIe subsystem, whereas with CAPI, the accelerator has direct access to the memory subsystem, which makes everything go faster.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Slide showing CAPI's memory access" src="/images/dja/capi-memory.png"&gt;&lt;/p&gt;
&lt;h2&gt;Uses of CAPI&lt;/h2&gt;
&lt;p&gt;CAPI technology is already powering a few really cool products.&lt;/p&gt;
&lt;p&gt;Firstly, we have an implementation of Redis that sits on top of flash storage connected over CAPI. Or, to take out the buzzwords, CAPI lets us do really, really fast NoSQL databases. There's &lt;a href="https://www.youtube.com/watch?v=cCmFc_0xsvA"&gt;a video online&lt;/a&gt; giving more details.&lt;/p&gt;
&lt;p&gt;Secondly, our partner &lt;a href="http://www.mellanox.com/page/products_dyn?product_family=201&amp;amp;mtag=connectx_4_vpi_card"&gt;Mellanox&lt;/a&gt; is using CAPI to make network cards that run at speeds of up to 100Gb/s.&lt;/p&gt;
&lt;p&gt;CAPI is also part of IBM's OpenPOWER initiative, where we're trying to grow a community of companies around our POWER system designs. So in many ways, CAPI is both a really cool technology, and a brand new ecosystem that we're growing here in the Canberra labs. It's very cool to be a part of!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Wed, 27 May 2015 15:08:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2015-05-27:/blog/2015/05/27/joining-the-capi-project/</guid><category>OpenPOWER</category><category>capi</category></item><item><title>OpenPOWER Powers Forward</title><link>https://sthbrx.github.io/blog/2015/05/21/openpower-powers-forward/</link><description>&lt;p&gt;I wrote this blog post late last year, it is very relevant for this blog though so I'll repost it here.&lt;/p&gt;
&lt;p&gt;With the launch of &lt;a href="http://www.tyan.com/campaign/openpower/"&gt;TYAN's OpenPOWER reference system&lt;/a&gt; now is a good time to reflect on the team responsible for so much of the research, design and development behind this very first ground breaking step of &lt;a href="http://openpowerfoundation.org/"&gt;OpenPOWER&lt;/a&gt; with their start to finish involvement of this new Power platform.&lt;/p&gt;
&lt;p&gt;ADL Canberra have been integral to the success of this launch providing the Open Power Abstraction Layer (OPAL) firmware. OPAL breathes new life into Linux on Power finally allowing Linux to run on directly on the hardware.
While OPAL harnesses the hardware, ADL Canberra significantly improved Linux to sit on top and take direct control of IBMs new Power8 processor without needing to negotiate with a hypervisor. With all the Linux expertise present at ADL Canberra it's no wonder that a Linux based bootloader was developed to make this system work. Petitboot leverage's all the resources of the Linux kernel to create a light, fast and yet extremely versatile bootloader. Petitboot provides a massive amount of tools for debugging and system configuration without the need to load an operating system.&lt;/p&gt;
&lt;p&gt;TYAN have developed great and highly customisable hardware. ADL Canberra have been there since day 1 performing vital platform enablement (bringup) of this new hardware. ADL Canberra have put all the work into the entire software stack, low level work to get OPAL and Linux to talk to the new BMC chip as well as the higher level, enabling to run Linux in either endian and Linux is even now capable of virtualising KVM guests in either endian irrespective of host endian. Furthermore a subset of ADL Canberra have been key to getting the Coherent Accelerator Processor Interface (CAPI) off the ground, enabling more almost endless customisation and greater diversity within the OpenPOWER ecosystem.&lt;/p&gt;
&lt;p&gt;ADL Canberra is the home for Linux on Power and the beginning of the OpenPOWER hardware sees much of the hard work by ADL Canberra come to fruition.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Cyril Bur</dc:creator><pubDate>Thu, 21 May 2015 11:29:00 +1000</pubDate><guid isPermaLink="false">tag:sthbrx.github.io,2015-05-21:/blog/2015/05/21/openpower-powers-forward/</guid><category>OpenPOWER</category></item></channel></rss>