Skip to content

Commit

Permalink
✨ (post): Added a post about my Lox bytecode VM
Browse files Browse the repository at this point in the history
  • Loading branch information
theobori committed Oct 10, 2024
1 parent 82b8011 commit b4e6d0b
Show file tree
Hide file tree
Showing 30 changed files with 2,044 additions and 132 deletions.
76 changes: 76 additions & 0 deletions posts/clox.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
---
title: My bytecode VM Lox interpreter
date: "2024-11-06"
---

The aim of this post is to describe the general operation of the program and some of the mechanisms that we consider to be of interest. For full details, a link to the source code is available at the bottom of the page.

So I continued with “Crafting Interpreters” by Robert Nystrom after making [My Tree-Walker Lox interpreter](/posts/jlox). In this part I tried to do as many challenges as possible and really understand how a VM bytecode works.

This version is written in C, which means we have to write a lot of code ourselves, but we don't use any external libraries.

## Compiler

The primary purpose of our compiler is to generate a chunk of code in bytecode form for interpretation by our bytecode virtual machine. Here are a few interesting features of the front end.

### Scanner

The token scanner is very classic, with only one thing to say: the function responsible for identifying the language's native keywords is very dirty. The author has chosen to use a large switch statement instead of implementing a sorting function, which is certainly powerful but not very elegant.

### Parser

An interesting point to note is that the author chose not to use a syntax tree for the front end. We therefore implemented a single-pass compiler (directly converts compile units into bytecode).

We also implemented a Vaughan Pratt's parser, in our case a “top-down operator precedence parser”. This means we have to define operator precedence in advance. Here's what it looks like in code.

```c
typedef enum {
PREC_NONE,
PREC_ASSIGNMENT, // =
PREC_OR, // or
PREC_AND, // and
PREC_EQUALITY, // == !=
PREC_COMPARISON, // < > <= >=
PREC_TERM, // + -
PREC_FACTOR, // * /
PREC_UNARY, // ! -
PREC_CALL, // . ()
PREC_PRIMARY
} Precedence;
```

This precedence is simply used to control the parsing of expressions. A rule with a lower precedence than the last parsed expression is not allowed.

## Bytecode

To manage conditions, we emit `OP_JUMP` operation code for conditions. If a condition expression is evaluated to false, it jumps to the end of the conditionnal block / expression. To do this, we use the concept of backpatching: we overwrite the immediate value of the instruction in the chunk during compilation.

In my implementation, all immediate values are encoded on 8 bits, with the exception of constants, which have a size of 24 bits.

## Virtual Machine

The VM is centered on a stack where we push operands, local variables, etc..

Everything at runtime is managed by callframes, even the top-level code is embed within a function object.

## Example

Here is a simple Lox example that can be evaluated by my interpreter.

```text
fun fib(n) {
if (n < 2) {
return n;
}
return fib(n - 2) + fib(n - 1);
}
print fib(10);
```

## Links

[https://github.com/theobori/lox-virtual-machine](https://github.com/theobori/lox-virtual-machine)

&nbsp;
90 changes: 35 additions & 55 deletions posts/lox.md → posts/jlox.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,13 @@
---
title: Yet another Lox interpreter
title: My Tree-Walker Lox interpreter
date: "2024-03-22"
---

I wanted to learn more about designing an interpreter, so I looked around and found the free "Crafting Interpreters" by Robert Nystrom.

I read parts I and II, which focus on concepts, common techniques and language behavior. Since I have recently read these parts, writing helps me to better understand and even re-understand certain things.

For the moment I'm not quite done, I've implemented the features below.

- *Tokens and lexing*
- *Abstract syntax trees*
- *Recursive descent parsing*
- *Prefix and infix expressions*
- *Runtime representation of objects*
- *Interpreting code using the Visitor pattern*
- *Lexical scope*
- *Environment chains for storing variables*
- *Control flow*
- *Functions with parameters*
- *Closures*
- *Static variable resolution and error detection*
&nbsp;
The aim was to have a Lox interpreter that at least supported functions and closures, so we could have a taste of the basics.

## What is lox ?

Expand All @@ -48,45 +34,6 @@ Scanning is also known as lexing or lexical analysis. It takes a linear stream o

The scanner must group characters into the smalles possible sequence that represents something. This blobs of characters are called lexemes.

Here are some examples of token kinds.

```python
...
from enum import Enum
...

class TokenKind(Enum):
"""
Represents every available token kinds
"""

# Single-character tokens
LEFT_PAREN = "left_paren",
RIGHT_PAREN = "right_paren",
LEFT_BRACE = "left_brace",
RIGHT_BRACE = "right_brace",
...

# One or two character tokens
BANG = "bang",
BANG_EQUAL = "bang_equal",
EQUAL = "equal",
...

# Literals
IDENTIFIER = "identifier",
STRING = "string",
NUMBER = "number",

# Keywords
AND = "and",
CLASS = "class",
ELSE = "else",
FALSE = "false",
...

EOF = "eof"
```
&nbsp;

### Parsing
Expand Down Expand Up @@ -146,7 +93,40 @@ So here, a valid strings could be the one below.
The best explanation here is probably the one in the book.

> *Recursive descent is considered a top-down parser because it starts from the top or outermost grammar rule (here expression ) and works its way down into the nested subexpressions before finally reaching the leaves of the syntax tree.*
&nbsp;

## Examples

Here are some Lox examples that can be evaluated by my interpreter.

```text
var b = 1;
var a = "hello";
{
var a = b + b;
print a;
}
print a;
fun fibonacci(n) {
if (n <= 1) return n;
return fibonacci(n - 1) + fibonacci(n - 2);
}
print fibonacci(5);
print "helo" + "world";
fun echo(n) {
print n;
return n;
}
print echo(echo(1) + echo(2)) + echo(echo(4) + echo(5));
```

&nbsp;

Expand Down
65 changes: 36 additions & 29 deletions posts/nix.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ date: "2024-06-24"
&nbsp;

Here I share some notes and other things I've learned about Nix that I find interesting. The content of this post is mainly about me learning Nix, it's not about understanding the whole tool and language.

Also, it's important to note that I use Nix as a non-NixOS user.
&nbsp;

## What is Nix?
Expand Down Expand Up @@ -314,7 +316,8 @@ Below is a Nix expression I wrote for the Python module [callviz](https://pypi.o
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
};
outputs = { self, nixpkgs }:
outputs =
{ self, nixpkgs }:
let
supportedSystems = [
"x86_64-linux"
Expand All @@ -323,26 +326,31 @@ Below is a Nix expression I wrote for the Python module [callviz](https://pypi.o
"aarch64-darwin"
];
forEachSupportedSystem = f: nixpkgs.lib.genAttrs supportedSystems (system: f {
pkgs = import nixpkgs { inherit system; };
});
forEachSupportedSystem =
f: nixpkgs.lib.genAttrs supportedSystems (system: f { pkgs = import nixpkgs { inherit system; }; });
in
{
packages = forEachSupportedSystem ({ pkgs }: {
default = pkgs.callPackage ./. { inherit (pkgs) python311; };
});
devShells = forEachSupportedSystem ({ pkgs }: {
default = pkgs.mkShell {
venvDir = ".venv";
packages = with pkgs; [ python311 ] ++
(with pkgs.python311Packages; [
pip
venvShellHook
graphviz
]);
};
});
# ...
# I usually also declare a default package, a code checker and formatter
devShells = forEachSupportedSystem (
{ pkgs }:
{
default = pkgs.mkShell {
venvDir = ".venv";
packages =
with pkgs;
[
python3
graphviz
]
++ (with pkgs.python3Packages; [
pip
venvShellHook
graphviz
]);
};
}
);
};
}
```
Expand Down Expand Up @@ -376,23 +384,24 @@ Here's what the package looks like.
SDL2_mixer,
zlib,
unstableGitUpdater,
makeWrapper,
}:
stdenv.mkDerivation (finalAttrs: {
pname = "supermariowar";
version = "2.0-unstable-2024-06-22";
version = "2023-unstable-2024-09-17";
src = fetchFromGitHub {
owner = "mmatyas";
repo = "supermariowar";
rev = "e646679c119a3b6c93c48e505564e8d24441fe4e";
hash = "sha256-bA/Pu47Rm1MrnJHIrRvOevI3LXj207GFcJloP94/LOA=";
rev = "6b8ff8c669ca31a116754d23b6ff65e42ac50733";
hash = "sha256-P0jV7G81thj0UJoYLd5+H5SjjaVu4goJxc9IkbzxJgs=";
fetchSubmodules = true;
};
nativeBuildInputs = [
cmake
pkg-config
makeWrapper
];
buildInputs = [
Expand All @@ -410,17 +419,15 @@ stdenv.mkDerivation (finalAttrs: {
mkdir -p $out/bin
for app in smw smw-leveledit smw-worldedit; do
chmod +x $out/games/$app
cat << EOF > $out/bin/$app
$out/games/$app --datadir $out/share/games/smw
EOF
chmod +x $out/bin/$app
makeWrapper $out/games/$app $out/bin/$app \
--add-flags "--datadir $out/share/games/smw"
done
ln -s $out/games/smw-server $out/bin/smw-server
'';
passthru.updateScript = unstableGitUpdater { };
meta = {
description = "A fan-made multiplayer Super Mario Bros. style deathmatch game";
homepage = "https://github.com/mmatyas/supermariowar";
Expand Down
51 changes: 51 additions & 0 deletions public_gemini/callviz.gmi
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# A toy to visualize recursive function calls
## 2024-06-29
Recently, I did a little project in
=> https://python.org Python
to visualise function calls, especially recursive functions.

It takes the form of a
=> https://python.org Python
decorator applied to the desired functions. The data structure used is fairly basic, a tree with nodes that have a parent and an indefinite number of children. Each node represents a function call, and the nodes also include the arguments passed to the function when it is called and, optionally, a return value.

To generate a visual and have an overview of all the function calls, I used
=> https://graphviz.org/ Graphviz
to manage a graph and save it as a file (DOT, SVG, PNG, etc.).

The decorator also supports memoization, which can also be represented on the final visual.

## How is it used?

These are two clear examples of how the decorator is used.

```python
from callviz.core import callviz, set_output_dir

set_output_dir("out")

@callviz(
_format="png",
memoization=True,
open_file=True,
show_node_result=True,
)
def fib(n: int):
if n < 2:
return n

return fib(n - 2) + fib(n - 1)

@callviz(_format="png", show_link_value=False)
def rev(arr, new):
if arr == []:
return new

return rev(arr[1:], [arr[0]] + new)

fib(5)
rev(list(range(6, 0, -1)), [])
```

## Links

=> https://github.com/theobori/callviz https://github.com/theobori/callviz
Loading

0 comments on commit b4e6d0b

Please sign in to comment.