Stackless 3.x segfaults on MacOS

[akruis]

I have no Mac. Therefore I can't debug or test on MacOS. I can use Travis CI to compile Stackless 2.7.15 and 3.6.6. Stackless 3.5.6 fails to compile.

Unfortunately Stackless 3.6.6 does not work. Example::

(testenv) dhcpmue-5:stackless-366-export kruis$ PYTHONPATH=$(pwd)/Lib python Stackless/unittests/test_watchdog.py 
testReceiveOnMain (__main__.TestDeadlock)
(soft) Thest that we get a deadock exception if main tries to block ... ok
testReceiveOnMain_H (__main__.TestDeadlock)
(hard) Thest that we get a deadock exception if main tries to block ... ok
test_error_propagation_when_not_deadlock (__main__.TestDeadlock) ... Segmentation fault: 11

[akruis]

This could be related to the problem reported in pull request #163: "In fact it no longer works for gcc 5.4."
I'll replace the trick and then we will know ...

[ctismer]

Hi Anselm, does it make sense for me to try?
I have more than one Mac machine.

[akruis]

@ctismer Hi Christian, your help is more than welcome. Could you try master-slp too?

[akruis]

@ctismer You can use the instructions for Linux https://github.com/stackless-dev/stackless/wiki/BuildForConda#building-on-linux to reproduce the faulty build on your Mac.

I'm fairly sure that the problem is caused by LTO. You could try the following patch:

diff --git a/Stackless/core/slp_transfer.c b/Stackless/core/slp_transfer.c
index 6564efd1f4..270521f499 100644
--- a/Stackless/core/slp_transfer.c
+++ b/Stackless/core/slp_transfer.c
@@ -117,6 +117,7 @@ slp_transfer(PyCStackObject **cstprev, PyCStackObject *cst,
 {
     PyThreadState *ts = PyThreadState_GET();
     int result;
+    static int (*volatile slp_switch_ptr)(void) = slp_switch;
 
     /* since we change the stack we must assure that the protocol was met */
     STACKLESS_ASSERT();
@@ -148,7 +149,7 @@ slp_transfer(PyCStackObject **cstprev, PyCStackObject *cst,
     _cstprev = cstprev;
     _cst = cst;
     _prev = prev;
-    result = slp_switch();
+    result = slp_switch_ptr();
     SLP_ASSERT_FRAME_IN_TRANSFER(ts);
     if (!result) {
         if (_cst) {

With this code, slp_switch() is called via a pointer. The assembly code within slp_switch assumes, that the respective ABI spec applies to slp_switch(), which is not guaranteed in presence of LTO.

[akruis]

I just wrote a small test program

#include <stdio.h>

void function_a(void)
{
  puts("Hello World!");
}

static void function_b(void)
{
  puts("function_b called");
}

static void function_c(void)
{
  puts("function_c called");
}


int main(int argc, char *argv[])
{
  static void (* volatile b_pointer)(void) = function_b;
  static void (* c_pointer)(void) = function_c;

  function_a();
  b_pointer();
  c_pointer();
  return 0;
}

Then I compiled it using clang 6.0.0 and gcc 6.3.0 with -O3 and looked at the generated assembly code:

• clang inlines function_a() and function_c(),
• gcc inlines function_a()
• gcc with -fwhole-program eliminates function_a()

Obviously the only reliable way to guarantee that the function from the same translation unit gets called without optimisations is to use a volatile function pointer. Therefore I'll apply the patch from my previous post.

[Fohlen]

I can report that compiling stackless on OSX 10.14.2 (Mojave) with stable OSX (Apple LLVM version 10.0.0 (clang-1000.10.44.4)) works successfully. Further I can also report that the multiprocessing_fork test fails with

Fatal Python error: Segmentation fault

Current thread 0x00000001074ec5c0 (most recent call first):
  File "/Users/fohlen/workspace/stackless/Lib/multiprocessing/connection.py", line 581 in __init__
Fatal Python error: Segmentation fault

Current thread 0x00000001074ec5c0 (most recent call first):
  File "/Users/fohlen/workspace/stackless/Lib/multiprocessing/connection.py", line 581 in __init__
Warning -- Dangling processes: {<ForkProcess(QueueManager-462, stopped[SIGSEGV])>}
Warning -- Dangling processes: {<ForkProcess(QueueManager-462, stopped[SIGSEGV])>}
test test_multiprocessing_fork failed -- multiple errors occurred; run in verbose mode for details
0:02:51 load avg: 19.21 [410/415/6] test_asyncio passed (62 sec) -- running: test_lib2to3 (57 sec), test_concurrent_futures (107 sec), test_multiprocessing_spawn (94 sec), test_tools (41 sec)
Executing <Handle <TaskWakeupMethWrapper object at 0x105da5be8>(<Future finis...events.py:375>) created at /Users/fohlen/workspace/stackless/Lib/asyncio/selector_events.py:512> took 0.171 seconds
/Users/fohlen/workspace/stackless/Lib/unittest/mock.py:1865: ResourceWarning: unclosed <socket.socket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 52569), raddr=('127.0.0.1', 52570)>
  setattr(_type, entry, MagicProxy(entry, self))
/Users/fohlen/workspace/stackless/Lib/asyncio/selector_events.py:663: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=12>
  source=self)
/Users/fohlen/workspace/stackless/Lib/selectors.py:192: ResourceWarning: unclosed <socket.socket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 52590), raddr=('127.0.0.1', 52591)>
  raise KeyError("{!r} is not registered".format(fileobj)) from None

Other than this other failed tests seem to be minor naming inconveniences with Python3.

[akruis]

@Fohlen Thank you very much for your feedback. Which version or branch did you compile?

[Fohlen]

@akruis I was building 3629160 (branch master-slp)

[mikalv]

I think I've encountered a similar issue and I've experienced it with both 2.7 and 3.7. At least it seem to trigger right after slp_transfer is called.

I've tested the following code:

import stackless

def SendingSequence(channel, sequence):
    print("sending")
    channel.send_sequence(sequence)

def ReceivingSequence(channel):
    for item in channel:
        print("receiving")
        print(item)

ch = stackless.channel()

task = stackless.tasklet(SendingSequence)(ch, ['a','b','c'])
task2 = stackless.tasklet(ReceivingSequence)(ch)

stackless.run()

And the debug session looks like this:

Process 45443 launched: '/usr/local/stacklesspy27/bin/python2.7' (x86_64)
Python 2.7.15 Stackless 3.1b3 060516 (default, Feb  2 2019, 16:29:32)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import stackless
...
>>> stackless.run()
sending
receiving
a
receiving
b
receiving
c
Process 45443 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x000000010016f74f python2.7`slp_schedule_task_prepared(ts=0x0100000000000008, result=0x0000000800000000, prev=0xcbcbcbcbcbcbcbcb, next=0x00000001004c5410, stackless=1, did_switch=0x00010101000934e9) at scheduling.c:1238
   1235	        transfer = slp_transfer;
   1236
   1237	    transfer_result = transfer(cstprev, next->cstate, prev);
-> 1238	    --ts->st.nesting_level;
   1239	    if (transfer_result >= 0) {
   1240	        PyFrameObject *f = SLP_CLAIM_NEXT_FRAME(ts);
   1241
Target 0: (python2.7) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x000000010016f74f python2.7`slp_schedule_task_prepared(ts=0x0100000000000008, result=0x0000000800000000, prev=0xcbcbcbcbcbcbcbcb, next=0x00000001004c5410, stackless=1, did_switch=0x00010101000934e9) at scheduling.c:1238
    frame #1: 0x00000001004c53e8
(lldb)

I tried to re-produce the same issue in the 3.7 version now, but after much testing in many directions it seems I've provoked another kind of issue which I've pasted bellow. However I recall that the 3.7 python crashed with a "bus error" as well at some point. But at least from a users perspective the same happened with both versions in both cases, once the execution of the code above are done it crashes.

Process 54664 launched: '/usr/local/stacklesspy/bin/python3.7' (x86_64)
Fatal Python error: initfsencoding: unable to load the file system codec
Traceback (most recent call last):
  File "/usr/local/stacklesspy/lib/python3.7/encodings/__init__.py", line 31, in <module>
Process 54664 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x18)
    frame #0: 0x0000000100236b52 python3.7`slp_eval_frame_newstack(f=0x0000000000000000, exc=0, retval=0x0000000000000000) at stacklesseval.c:965
   962
   963 	    retval = cf->ob1;
   964 	    cf->ob1 = NULL;
-> 965 	    if (PyBomb_Check(retval))
   966 	        retval = slp_bomb_explode(retval);
   967 	finally:
   968 	    Py_DECREF(cf);
Target 0: (python3.7) stopped.
(lldb)

[akruis]

@mikalv Thank you for your report. Unfortunately I still can't debug this issue (no Mac), but probably the Stack switching code is not correct for MacOS.
The Mac uses an x86_64 CPU, but we do not have a MacOS specific switching code. See
https://github.com/stackless-dev/stackless/blob/3.7-slp/Include/internal/slp_platformselect.h#L14
Instead slp_platformselect.h includes the Unix amd64 code. This code is based on https://software.intel.com/sites/default/files/article/402129/mpx-linux64-abi.pdf . Some older OS-X documentation (https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/MachOTopics/1-Articles/x86_64_code.html) also references this spec, but I don't know, if this is still current.

For now the fix is to add a && defined(__linux__) in slp_platformselect.h. This way Stackless does not compile any more on MacOS. And then we need an updated definition of slp_switch() for MacOS.
Any help from is highly appreciated.

[mikalv]

@akruis No problem, I'm happy to give it a try to fix this, however I'm not that familiar with assembly and stackless (warning about possible stupid questions upcoming).

MacOS's kernel is a fork&merge of mach3 and the FreeBSD kernel - where most of the BSD api is available for POSIX applications, so in general software that runs on FreeBSD runs fine on MacOS, but sometime that's false.

The document you linked seems correct from what I know. The little I know of assembly would be that MacOS on 64 architectures adopt the System V AMD64 ABI reference, so at least a syscall would use the registers rdi, rsi, rdx, r10, r8 and r9 as arguments just as on linux if I'm correct. I'm currently reading up on stmxcsr and fstcw which was totally new for me.

By the way, check out http://www.darlinghq.org/ - If I recall right they had a kernel module for linux now to simulate the MacOS userspace.

[mikalv]

Ok I think I understand the code now, it store/loads the register states. I did a new build from current master, and seems I provoked it to crash at the startup. However it seems very related.

(lldb) target create "./python"
Current executable set to './python' (x86_64).
(lldb) run
Process 83661 launched: './python' (x86_64)
Assertion failed: ((ts)->st.frame_refcnt > 0), function slp_eval_frame_newstack, file Stackless/core/stacklesseval.c, line 960.
Process 83661 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x00007fff5ebd6b86 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff5ebd6b86 <+10>: jae    0x7fff5ebd6b90            ; <+20>
    0x7fff5ebd6b88 <+12>: movq   %rax, %rdi
    0x7fff5ebd6b8b <+15>: jmp    0x7fff5ebd0e67            ; cerror_nocancel
    0x7fff5ebd6b90 <+20>: retq
Target 0: (python) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff5ebd6b86 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff5ec8cc50 libsystem_pthread.dylib`pthread_kill + 285
    frame #2: 0x00007fff5eb401c9 libsystem_c.dylib`abort + 127
    frame #3: 0x00007fff5eb08868 libsystem_c.dylib`__assert_rtn + 320
    frame #4: 0x000000010031e1c5 libpython3.7dm.dylib`slp_eval_frame_newstack(f=0x0000000101170d48, exc=0, retval=0x0000000000000002) at stacklesseval.c:960
(lldb) f 4
frame #4: 0x000000010031e1c5 libpython3.7dm.dylib`slp_eval_frame_newstack(f=0x0000000101170d48, exc=0, retval=0x0000000000000002) at stacklesseval.c:960
   957 	    cur->cstate = NULL;
   958 	    if (slp_transfer(&cur->cstate, NULL, cur) < 0)
   959 	        goto finally; /* fatal */
-> 960 	    SLP_ASSERT_FRAME_IN_TRANSFER(ts);
   961 	    Py_XDECREF(cur->cstate);
   962
   963 	    retval = cf->ob1;
(lldb) f 5
error: Frame index (5) out of range.
(lldb) quit

[mikalv]

This is the slp_switch function after it's compiled by the default Apple compiler.

        ; Variables:
        ;    var_2C: int32_t, -44
        ;    var_38: int64_t, -56
        ;    var_40: int64_t, -64
        ;    var_44: int32_t, -68
        ;    var_46: int16_t, -70
        ;    var_50: int64_t, -80


                                    _slp_switch:
0000000000235e60 4157                   push       r15                          ; DATA XREF=_slp_transfer.slp_switch_ptr
0000000000235e62 4156                   push       r14
0000000000235e64 4155                   push       r13
0000000000235e66 4154                   push       r12
0000000000235e68 53                     push       rbx
0000000000235e69 4883EC30               sub        rsp, 0x30
0000000000235e6d 9B                     wait
0000000000235e6e D97C2412               fnstcw     word [rsp+0x58+var_46]
0000000000235e72 0FAE5C2414             stmxcsr    dword [rsp+0x58+var_44]
0000000000235e77 488D1D121A2600         lea        rbx, qword [__PyRuntime]     ; __PyRuntime
0000000000235e7e 4889642420             mov        qword [rsp+0x58+var_38], rsp
0000000000235e83 4C8B742420             mov        r14, qword [rsp+0x58+var_38]
0000000000235e88 4C89742420             mov        qword [rsp+0x58+var_38], r14
0000000000235e8d 4883BBE825000000       cmp        qword [rbx+0x25e8], 0x0
0000000000235e95 0F8456000000           je         loc_235ef1

0000000000235e9b 488D05EE192600         lea        rax, qword [__PyRuntime]     ; __PyRuntime
0000000000235ea2 488BB8E8250000         mov        rdi, qword [rax+0x25e8]      ; argument #1 for method _slp_cstack_new
0000000000235ea9 488B742420             mov        rsi, qword [rsp+0x58+var_38] ; argument #2 for method _slp_cstack_new
0000000000235eae 488B90F8250000         mov        rdx, qword [rax+0x25f8]      ; argument #3 for method _slp_cstack_new
0000000000235eb5 E816020000             call       _slp_cstack_new              ; _slp_cstack_new
0000000000235eba 4883F800               cmp        rax, 0x0
0000000000235ebe 0F850D000000           jne        loc_235ed1

0000000000235ec4 C744242CFFFFFFFF       mov        dword [rsp+0x58+var_2C], 0xffffffff
0000000000235ecc E9D0000000             jmp        loc_235fa1

                                    loc_235ed1:
0000000000235ed1 488D05B8192600         lea        rax, qword [__PyRuntime]     ; __PyRuntime, CODE XREF=_slp_switch+94
0000000000235ed8 488B80E8250000         mov        rax, qword [rax+0x25e8]
0000000000235edf 488B38                 mov        rdi, qword [rax]             ; argument #1 for method _slp_cstack_save
0000000000235ee2 E859060000             call       _slp_cstack_save             ; _slp_cstack_save
0000000000235ee7 4889442408             mov        qword [rsp+0x58+var_50], rax
0000000000235eec E927000000             jmp        loc_235f18

                                    loc_235ef1:
0000000000235ef1 488D0598192600         lea        rax, qword [__PyRuntime]     ; __PyRuntime, CODE XREF=_slp_switch+53
0000000000235ef8 488B80F0250000         mov        rax, qword [rax+0x25f0]
0000000000235eff 488B4058               mov        rax, qword [rax+0x58]
0000000000235f03 488B4C2420             mov        rcx, qword [rsp+0x58+var_38]
0000000000235f08 4829C8                 sub        rax, rcx
0000000000235f0b 48C1F803               sar        rax, 0x3
0000000000235f0f 48C1E003               shl        rax, 0x3
0000000000235f13 4889442408             mov        qword [rsp+0x58+var_50], rax

                                    loc_235f18:
0000000000235f18 488D0571192600         lea        rax, qword [__PyRuntime]     ; __PyRuntime, CODE XREF=_slp_switch+140
0000000000235f1f 4883B8F025000000       cmp        qword [rax+0x25f0], 0x0
0000000000235f27 0F850D000000           jne        loc_235f3a

0000000000235f2d C744242C00000000       mov        dword [rsp+0x58+var_2C], 0x0
0000000000235f35 E967000000             jmp        loc_235fa1

                                    loc_235f3a:
0000000000235f3a 488D054F192600         lea        rax, qword [__PyRuntime]     ; __PyRuntime, CODE XREF=_slp_switch+199
0000000000235f41 488B4C2408             mov        rcx, qword [rsp+0x58+var_50]
0000000000235f46 488B80F0250000         mov        rax, qword [rax+0x25f0]
0000000000235f4d 488B4020               mov        rax, qword [rax+0x20]
0000000000235f51 48C1E003               shl        rax, 0x3
0000000000235f55 4829C1                 sub        rcx, rax
0000000000235f58 48894C2418             mov        qword [rsp+0x58+var_40], rcx
0000000000235f5d 488B442418             mov        rax, qword [rsp+0x58+var_40]
0000000000235f62 4801C4                 add        rsp, rax
0000000000235f65 4801C5                 add        rbp, rax
0000000000235f68 488D0521192600         lea        rax, qword [__PyRuntime]     ; __PyRuntime
0000000000235f6f 4883B8F025000000       cmp        qword [rax+0x25f0], 0x0
0000000000235f77 0F8413000000           je         loc_235f90

0000000000235f7d 488D050C192600         lea        rax, qword [__PyRuntime]     ; __PyRuntime
0000000000235f84 488BB8F0250000         mov        rdi, qword [rax+0x25f0]      ; argument #1 for method _slp_cstack_restore
0000000000235f8b E810060000             call       _slp_cstack_restore          ; _slp_cstack_restore

                                    loc_235f90:
0000000000235f90 0FAE542414             ldmxcsr    dword [rsp+0x58+var_44]      ; CODE XREF=_slp_switch+279
0000000000235f95 D96C2412               fldcw      word [rsp+0x58+var_46]
0000000000235f99 C744242C00000000       mov        dword [rsp+0x58+var_2C], 0x0

                                    loc_235fa1:
0000000000235fa1 8B44242C               mov        eax, dword [rsp+0x58+var_2C] ; CODE XREF=_slp_switch+108, _slp_switch+213
0000000000235fa5 4883C430               add        rsp, 0x30
0000000000235fa9 5B                     pop        rbx
0000000000235faa 415C                   pop        r12
0000000000235fac 415D                   pop        r13
0000000000235fae 415E                   pop        r14
0000000000235fb0 415F                   pop        r15
0000000000235fb2 C3                     ret
                        ; endp
0000000000235fb3                        align      64

[akruis]

@mikalv Two suggestions

1. You could change the line
   

   stackless/Include/internal/slp_switch_amd64_unix.h

   Line 84 in a871dd3

   #if 1

   
   to #if 0 and check, if you get a different assembler output.
   In theory, it shouldn't change, but who knows.

2. A debug build is really useful, because it checks many assertions. Use
   configure --with-pydebug && make && make teststackless && make test

And many thanks for the hint to the Darling Project.

[mikalv]

./Include/internal/slp_switch_amd64_unix.h:104:9: error: clobbers must be last on the x87 stack
        "fstcw %0\n\t"

Was the result of your first suggestion with the master branch of this repo. I'm not totally sure how to "fix" that error yet.

I'll report on suggestion two soon.

[mikalv]

This was odd, but good news. Pickle failed the test. However, stackless passed my manual test (the snippet from my first comment). I posted both bellow.

Earlier I've tested these:
./configure --with-openssl=/usr/local/opt/openssl@1.1/ --enable-stacklessfewerregisters --enable-shared --enable-ipv6 --with-pydebug --prefix=/usr/local/stackless
./configure --with-openssl=/usr/local/opt/openssl@1.1/ --enable-shared --enable-ipv6 --with-pydebug --prefix=/usr/local/stackless
./configure --with-openssl=/usr/local/opt/openssl@1.1/ --enable-ipv6 --with-pydebug --prefix=/usr/local/stackless

However, running make clean, and then ./configure --with-pydebug && make && make teststackless && make test like you suggested didn't result in a crashing python. :)

running build_scripts
copying and adjusting /Volumes/localzfs/Workspaces/stackless-python/Tools/scripts/pydoc3 -> build/scripts-3.7
copying and adjusting /Volumes/localzfs/Workspaces/stackless-python/Tools/scripts/idle3 -> build/scripts-3.7
copying and adjusting /Volumes/localzfs/Workspaces/stackless-python/Tools/scripts/2to3 -> build/scripts-3.7
copying and adjusting /Volumes/localzfs/Workspaces/stackless-python/Tools/scripts/pyvenv -> build/scripts-3.7
changing mode of build/scripts-3.7/pydoc3 from 644 to 755
changing mode of build/scripts-3.7/idle3 from 644 to 755
changing mode of build/scripts-3.7/2to3 from 644 to 755
changing mode of build/scripts-3.7/pyvenv from 644 to 755
renaming build/scripts-3.7/pydoc3 to build/scripts-3.7/pydoc3.7
renaming build/scripts-3.7/idle3 to build/scripts-3.7/idle3.7
renaming build/scripts-3.7/2to3 to build/scripts-3.7/2to3-3.7
renaming build/scripts-3.7/pyvenv to build/scripts-3.7/pyvenv-3.7
./python -E -c 'import sys ; from sysconfig import get_platform ; print("%s-%d.%d" % (get_platform(), *sys.version_info[:2]))' >platform
./python  -E ./Stackless/unittests/runAll.py
== CPython 3.7.0a4+ Stackless 3.7 (heads/master-slp:a871dd359e, Feb 7 2019, 01:00:55) [Clang 10.0.0 (clang-1000.11.45.5)]
== Darwin-18.2.0-x86_64-i386-64bit little-endian
== cwd: /Volumes/localzfs/Workspaces/stackless-python/build/test_python_2663
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 3.24 [ 1/17] test_capi
0:00:00 load avg: 3.06 [ 2/17] test_chan_cb
0:00:00 load avg: 3.06 [ 3/17] test_channel
0:00:00 load avg: 3.06 [ 4/17] test_defects
0:00:02 load avg: 3.06 [ 5/17] test_exception
0:00:03 load avg: 3.06 [ 6/17] test_generator
0:00:04 load avg: 3.06 [ 7/17] test_miscell
0:00:08 load avg: 3.13 [ 8/17] test_outside
0:00:08 load avg: 3.13 [ 9/17] test_pickle
test test_pickle failed -- multiple errors occurred; run in verbose mode for details
0:00:24 load avg: 2.88 [10/17/1] test_sched_cb -- test_pickle failed
0:00:24 load avg: 2.88 [11/17/1] test_shutdown
0:00:35 load avg: 2.66 [12/17/1] test_thread
0:00:37 load avg: 2.66 [13/17/1] test_tpflags_have_stackless_call
0:00:38 load avg: 2.66 [14/17/1] test_tracing
0:00:38 load avg: 2.66 [15/17/1] test_tstate
0:00:38 load avg: 2.66 [16/17/1] test_watchdog
0:00:39 load avg: 2.66 [17/17/1] test_weakref
16 tests OK.

1 test failed:
    test_pickle

Total duration: 40 sec
Tests result: FAILURE
make: *** [teststackless] Error 2

Python 3.7.0a4+ Stackless 3.7 (heads/master-slp:a871dd359e, Feb  7 2019, 01:00:55)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import stackless
>>> def SendingSequence(channel, sequence):
...     print("sending")
...     channel.send_sequence(sequence)
...
>>> def ReceivingSequence(channel):
...     for item in channel:
...         print("receiving")
...         print(item)
...
>>> ch = stackless.channel()
>>>
>>> task = stackless.tasklet(SendingSequence)(ch, ['a','b','c'])
>>> task2 = stackless.tasklet(ReceivingSequence)(ch)
>>>
>>> stackless.run()
sending
receiving
a
receiving
b
receiving
c
>>>
>>>
>>>

[mikalv]

Whatever issues 3.7 and 2.7 had, it seems the master-slp branch has resolved it.

[akruis]

clobbers must be last on the x87 stack

I just pushed a fix for this error: d29d047

The assembly code of the C-function int slp_transfer(PyCStackObject **cstprev, PyCStackObject *cst, PyTaskletObject *prev) from Stackless/core/slp_transfer.c would be interesting too. This function calls static int slp_switch(void). If the compiler optimises the call the calling convention does not apply any more. See issue #183.

[akruis]

0:00:08 load avg: 3.13 [ 9/17] test_pickle
test test_pickle failed -- multiple errors occurred; run in verbose mode for details

What's the output of ./python -s -E Stackless/unittests/runAll.py -v test_pickle ?

[akruis]

Whatever issues 3.7 and 2.7 had, it seems the master-slp branch has resolved it.

No, the "Stackless code" in master-slp and 3.7-slp is the same. It is more likely an effect of the disabled optimisation in debug builds.

[mikalv]

I cut the text where everything was OK - bellow is where it started failing:

estTracebackFrameLinkage_P3_H (test_pickle.TestTraceback) ... ok
testTracebackFrameLinkage_P4 (test_pickle.TestTraceback) ... ok
testTracebackFrameLinkage_P4_H (test_pickle.TestTraceback) ... ok

======================================================================
ERROR: testRecursive (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 333, in testRecursive
    self.run_pickled(rectest, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursiveEmbedded (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 377, in testRecursiveEmbedded
    self.run_pickled(rectest, self.verbose, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursiveEmbedded_C1 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 377, in testRecursiveEmbedded
    self.run_pickled(rectest, self.verbose, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursiveEmbedded_C2 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 377, in testRecursiveEmbedded
    self.run_pickled(rectest, self.verbose, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursiveEmbedded_C3 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 377, in testRecursiveEmbedded
    self.run_pickled(rectest, self.verbose, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursiveEmbedded_C4 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 377, in testRecursiveEmbedded
    self.run_pickled(rectest, self.verbose, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursiveEmbedded_P0 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 377, in testRecursiveEmbedded
    self.run_pickled(rectest, self.verbose, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursiveEmbedded_P1 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 377, in testRecursiveEmbedded
    self.run_pickled(rectest, self.verbose, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursiveEmbedded_P2 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 377, in testRecursiveEmbedded
    self.run_pickled(rectest, self.verbose, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursiveEmbedded_P3 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 377, in testRecursiveEmbedded
    self.run_pickled(rectest, self.verbose, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursiveEmbedded_P4 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 377, in testRecursiveEmbedded
    self.run_pickled(rectest, self.verbose, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 374, in rectest
    rectest(verbose, nrec, lev + 1)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursive_C1 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 333, in testRecursive
    self.run_pickled(rectest, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursive_C2 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 333, in testRecursive
    self.run_pickled(rectest, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursive_C3 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 333, in testRecursive
    self.run_pickled(rectest, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursive_C4 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 333, in testRecursive
    self.run_pickled(rectest, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursive_P0 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 333, in testRecursive
    self.run_pickled(rectest, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursive_P1 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 333, in testRecursive
    self.run_pickled(rectest, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursive_P2 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 333, in testRecursive
    self.run_pickled(rectest, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursive_P3 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 333, in testRecursive
    self.run_pickled(rectest, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

======================================================================
ERROR: testRecursive_P4 (test_pickle.TestPickledTasklets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/support.py", line 500, in test
    return method(self)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 333, in testRecursive
    self.run_pickled(rectest, 13)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 302, in run_pickled
    ip.run()
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 36, in accumulate
    rval = (ident, func(*args))
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  File "/Volumes/localzfs/Workspaces/stackless-python/Stackless/unittests/test_pickle.py", line 57, in rectest
    rectest(nrec, lev + 1, lst)
  [Previous line repeated 9 more times]
RuntimeError: cannot execute invalid frame with 'eval_frame_value': frame had a C state that can't be restored.

----------------------------------------------------------------------
Ran 1205 tests in 16.170s

FAILED (errors=20)
test test_pickle failed
test_pickle failed

1 test failed:
    test_pickle

Total duration: 16 sec
Tests result: FAILURE

[akruis]

@mikalv Thank you very much for providing this output. All failures are caused by the same root cause. It could be related to the other Mac failures.

About the root cause of the pickling failures: if Stackless switches from one tasklet to another tasklet it uses one of two different mechanisms. Either hard-switching or soft-switching. Hard switching is always possible. It manipulates the stack (slp_switch). Soft switching is only possible, if the interpreter has not been invoked recursively by an extension function written in C. If soft switching is possible, it it has advantages:

• faster
• it is possible to save and successfully restore inactive tasklets, if their state can be pickled.

The error message "frame had a C state that can't be restored." means, that a pickled tasklet could not be restored, because Stackless used hard switching. But why?