Skip to content

Commit

Permalink
[mypyc] Updates to dev docs, including debugging segfaults (#18462)
Browse files Browse the repository at this point in the history
Co-authored-by: Valentin Stanciu <250871+svalentin@users.noreply.github.com>
  • Loading branch information
JukkaL and svalentin authored Jan 14, 2025
1 parent ce61d11 commit 075f79a
Showing 1 changed file with 47 additions and 16 deletions.
63 changes: 47 additions & 16 deletions mypyc/doc/dev-intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,9 @@ good error message.

Here are some major things that aren't yet supported in compiled code:

* Many dunder methods (only some work, such as `__init__` and `__eq__`)
* Some dunder methods (most work though)
* Monkey patching compiled functions or classes
* General multiple inheritance (a limited form is supported)
* Named tuple defined using the class-based syntax
* Defining protocols

We are generally happy to accept contributions that implement new Python
features.
Expand All @@ -73,16 +71,16 @@ compiled code. For example, you may want to do interactive testing or
to run benchmarks. This is also handy if you want to inspect the
generated C code (see Inspecting Generated C).

Run `mypyc` to compile a module to a C extension using your
Run `python -m mypyc` to compile a module to a C extension using your
development version of mypyc:

```
$ mypyc program.py
$ python -m mypyc program.py
```

This will generate a C extension for `program` in the current working
directory. For example, on a Linux system the generated file may be
called `program.cpython-37m-x86_64-linux-gnu.so`.
directory. For example, on a macOS system the generated file may be
called `program.cpython-313-darwin.so`.

Since C extensions can't be run as programs, use `python3 -c` to run
the compiled module as a program:
Expand All @@ -95,7 +93,7 @@ Note that `__name__` in `program.py` will now be `program`, not
`__main__`!

You can manually delete the C extension to get back to an interpreted
version (this example works on Linux):
version (this example works on macOS or Linux):

```
$ rm program.*.so
Expand All @@ -114,9 +112,9 @@ extensions) in compiled code.

Mypyc will only make compiled code faster. To see a significant
speedup, you must make sure that most of the time is spent in compiled
code -- and not in libraries, for example.
code, and not in libraries or I/O.

Mypyc has these passes:
Mypyc has these main passes:

* Type check the code using mypy and infer types for variables and
expressions. This produces a mypy AST (defined in `mypy.nodes`) and
Expand Down Expand Up @@ -193,13 +191,13 @@ information. See the test cases in
`mypyc/test-data/irbuild-basic.test` for examples of what the IR looks
like in a pretty-printed form.

## Testing overview
## Testing Overview

Most mypyc test cases are defined in the same format (`.test`) as used
for test cases for mypy. Look at mypy developer documentation for a
general overview of how things work. Test cases live under
`mypyc/test-data/`, and you can run all mypyc tests via `pytest
-q mypyc`. If you don't make changes to code under `mypy/`, it's not
mypyc`. If you don't make changes to code under `mypy/`, it's not
important to regularly run mypy tests during development.

You can use `python runtests.py mypyc-fast` to run a subset of mypyc
Expand Down Expand Up @@ -228,7 +226,7 @@ We also have tests that verify the generate IR

## Type-checking Mypyc

`./runtests.py self` type checks mypy and mypyc. This is pretty slow,
`./runtests.py self` type checks mypy and mypyc. This is a little slow,
however, since it's using an uncompiled mypy.

Installing a released version of mypy using `pip` (which is compiled)
Expand Down Expand Up @@ -311,7 +309,7 @@ number of components at once, insensitive to the particular details of
the IR), but there really is no substitute for running code. You can
also write tests that test the generated IR, however.

### Tests that compile and run code
### Tests That Compile and Run Code

Test cases that compile and run code are located in
`mypyc/test-data/run*.test` and the test runner is in
Expand Down Expand Up @@ -364,15 +362,48 @@ Test cases can also have a `[out]` section, which specifies the
expected contents of stdout the test case should produce. New test
cases should prefer assert statements to `[out]` sections.

### IR tests
### Debuggging Segfaults

If you experience a segfault, it's recommended to use a debugger that supports
C, such as gdb or lldb, to look into the segfault.

If a test case segfaults, you can run tests using the debugger, so
you can inspect the stack:

```
$ pytest mypyc -n0 -s --mypyc-debug=gdb -k <name-of-test>
```

You must use `-n0 -s` to enable interactive input to the debugger.
Instad of `gdb`, you can also try `lldb`.

To get better C stack tracebacks and more assertions in the Python
runtime, you can build Python in debug mode and use that to run tests
or debug outside the test framework.

Here are some hints that may help (for Ubuntu):

```
$ sudo apt install gdb build-essential libncursesw5-dev libssl-dev libgdbm-dev libc6-dev libsqlite3-dev libbz2-dev libffi-dev libgdbm-compat-dev
$ <download Python tarball and extract it>
$ cd Python-3.XX.Y
$ ./configure --with-pydebug
$ make -s -j16
$ ./python -m venv ~/<venv-location>
$ source ~/<venv-location>/bin/activate
$ cd <mypy-repo-dir>
$ pip install -r test-requirements.txt
```

### IR Tests

If the specifics of the generated IR of a change is important
(because, for example, you want to make sure a particular optimization
is triggering), you should add a `mypyc.irbuild` test as well. Test
cases are located in `mypyc/test-data/irbuild-*.test` and the test
driver is in `mypyc.test.test_irbuild`. IR build tests do a direct
comparison of the IR output, so try to make the test as targeted as
possible so as to capture only the important details. (Many of our
possible so as to capture only the important details. (Some of our
existing IR build tests do not follow this advice, unfortunately!)

If you pass the `--update-data` flag to pytest, it will automatically
Expand Down

0 comments on commit 075f79a

Please sign in to comment.