Ubuntu jammy s390x: Test failures #60

junaruga · 2023-09-11T18:14:19Z

I am seeing the following test failures on RubyCI Ubuntu s390x version Jammy (22.04.3 LTS) on the ruby/zlib latest master branch 85637fa with the latest ruby/ruby master branch. Could you take a look at the failures? I think you can log in to the RubyCI Ubuntu s390x server to debug. Thank you for your help.

$ cat /etc/os-release | grep VERSION
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy

$ uname -m
s390x

$ gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ which ruby
/home/jaruga/.local/ruby-05a853c2f2-debug/bin/ruby

$ ruby -v
ruby 3.3.0dev (2023-09-11T15:25:06Z master 05a853c2f2) [s390x-linux]

$ bundle install --standalone

$ bundle list
Gems included by the bundle:
  * power_assert (2.0.3)
  * rake (13.0.6)
  * rake-compiler (1.2.5)
  * test-unit (3.6.1)
  * test-unit-ruby-core (1.0.2)
  * zlib (3.0.0)
Use `bundle info` to print more detailed information about a gem

$ bundle exec rake compile

$ ldd ./lib/zlib.so
  linux-vdso64.so.1 (0x000003ffa007e000)
  libruby.so.3.3 => /home/jaruga/.local/ruby-05a853c2f2-debug/lib/libruby.so.3.3 (0x000003ff9f900000)
  libz.so.1 => /lib/s390x-linux-gnu/libz.so.1 (0x000003ff9f800000)
  libm.so.6 => /lib/s390x-linux-gnu/libm.so.6 (0x000003ff9f700000)
  libc.so.6 => /lib/s390x-linux-gnu/libc.so.6 (0x000003ff9f500000)
  libgmp.so.10 => /lib/s390x-linux-gnu/libgmp.so.10 (0x000003ff9f400000)
  libcrypt.so.1 => /lib/s390x-linux-gnu/libcrypt.so.1 (0x000003ff9f380000)
  /lib/ld64.so.1 (0x000003ffa0000000)

Below is the used Zlib deb package version.

$ dpkg -S /lib/s390x-linux-gnu/libz.so.1
zlib1g:s390x: /lib/s390x-linux-gnu/libz.so.1
$ dpkg -s zlib1g | grep ^Version
Version: 1:1.2.11.dfsg-2ubuntu9.2

$ bundle exec rake test
Loaded suite /home/jaruga/git/ruby/zlib/bundle/ruby/3.3.0+0/gems/rake-13.0.6/lib/rake/rake_test_loader
Started
F
======================================================================================================
Failure: test_deflate_stream(TestZlib)
/home/jaruga/git/ruby/zlib/test/zlib/test_zlib.rb:1411:in `test_deflate_stream'
     1408:         deflated << chunk
     1409:       end
     1410: 
  => 1411:       assert_equal 20016, deflated.length
     1412:     end
     1413: 
     1414:     def test_gzip
<20016> expected but was
<21085>

diff:
? 2 0016
?  1 85 
?  + ??? 
======================================================================================================
P
======================================================================================================
Pending: test_gunzip_no_memory_leak(TestZlib): pended.
/home/jaruga/git/ruby/zlib/bundle/ruby/3.3.0+0/gems/test-unit-ruby-core-1.0.2/lib/core_assertions.rb:192:in `rescue in assert_no_memory_leak'
/home/jaruga/git/ruby/zlib/bundle/ruby/3.3.0+0/gems/test-unit-ruby-core-1.0.2/lib/core_assertions.rb:150:in `assert_no_memory_leak'
/home/jaruga/git/ruby/zlib/test/zlib/test_zlib.rb:1468:in `test_gunzip_no_memory_leak'
     1465:     end
     1466: 
     1467:     def test_gunzip_no_memory_leak
  => 1468:       assert_no_memory_leak(%[-rzlib], "#{<<~"{#"}", "#{<<~'};'}")
     1469:       d = Zlib.gzip("data")
     1470:       {#
     1471:         10_000.times {Zlib.gunzip(d)}
======================================================================================================
F
======================================================================================================
Failure: test_gzip(TestZlib)
/home/jaruga/git/ruby/zlib/test/zlib/test_zlib.rb:1419:in `test_gzip'
     1416:       actual[4, 4] = "\x00\x00\x00\x00" # replace mtime
     1417:       actual[9] = "\xff" # replace OS
     1418:       expected = %w[1f8b08000000000000ff4bcbcf07002165738c03000000].pack("H*")
  => 1419:       assert_equal expected, actual
     1420: 
     1421:       actual = Zlib.gzip("foo".freeze, level: 0)
     1422:       actual[4, 4] = "\x00\x00\x00\x00" # replace mtime
<"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\xFFK\xCB\xCF\a\x00!es\x8C\x03\x00\x00\x00"> expected but was
<"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\xFFJ\xCB\xCF\a\f\x00!es\x8C\x03\x00\x00\x00">

diff:
? �K�� !es�
?           J   
                         
?           ?   +         
======================================================================================================
F
======================================================================================================
Failure: test_deflate_chunked(TestZlibDeflate)
/home/jaruga/git/ruby/zlib/test/zlib/test_zlib.rb:66:in `test_deflate_chunked'
     63: 
     64:       final = z.finish
     65: 
  => 66:       assert_equal 7253, final.length
     67: 
     68:       chunks << final
     69:       all = chunks.join
<7253> expected but was
<8325>

diff:
? 7 253
? 83   
? ?  -
======================================================================================================
F
======================================================================================================
Failure: test_deflate_chunked_break(TestZlibDeflate)
/home/jaruga/git/ruby/zlib/test/zlib/test_zlib.rb:92:in `test_deflate_chunked_break'
     89: 
     90:       final = z.finish
     91: 
  => 92:       assert_equal 3632, final.length
     93: 
     94:       all = chunks.join
     95:       all << final
<3632> expected but was
<4702>

diff:
? 3632
? 470 
? ??? 
======================================================================================================
F
======================================================================================================
Failure: test_unused2(TestZlibGzipReader)
/home/jaruga/git/ruby/zlib/test/zlib/test_zlib.rb:968:in `test_unused2'
     965:       io = Zlib::GzipReader.new zio
     966:       assert_equal('aaaa', io.read)
     967:       unused = io.unused
  => 968:       assert_equal(24, unused.bytesize)
     969:       io.finish
     970: 
     971:       zio.pos -= unused.length
<24> expected but was
<23>

diff:
? 24
?  3
?  ?
======================================================================================================
|
Finished in 6.146844438 seconds.
------------------------------------------------------------------------------------------------------
95 tests, 511 assertions, 5 failures, 0 errors, 1 pendings, 0 omissions, 0 notifications
93.6842% passed
------------------------------------------------------------------------------------------------------
15.46 tests/s, 83.13 assertions/s
rake aborted!
Command failed with status (1)
/home/jaruga/git/ruby/zlib/bundle/ruby/3.3.0+0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/home/jaruga/.local/ruby-05a853c2f2-debug/bin/bundle:25:in `load'
/home/jaruga/.local/ruby-05a853c2f2-debug/bin/bundle:25:in `<main>'
Tasks: TOP => test
(See full trace by running task with --trace)

ruby/ruby

I also can see the failures in ruby/ruby latest master branch 05a853c2f21f60f9e1c544c2d0709f10de453571.

$ make V=1 test-all TESTS="test/zlib/test_zlib.rb -v"
...
  1) Failure:
TestZlib#test_deflate_stream [/home/jaruga/git/ruby/ruby/test/zlib/test_zlib.rb:1411]:
<20016> expected but was
<21085>.

  2) Failure:
TestZlib#test_gzip [/home/jaruga/git/ruby/ruby/test/zlib/test_zlib.rb:1419]:
<"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\xFFK\xCB\xCF\a\x00!es\x8C\x03\x00\x00\x00"> expected but was
<"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\xFFJ\xCB\xCF\a\f\x00!es\x8C\x03\x00\x00\x00">.

  3) Failure:
TestZlibDeflate#test_deflate_chunked [/home/jaruga/git/ruby/ruby/test/zlib/test_zlib.rb:66]:
<7253> expected but was
<8325>.

  4) Failure:
TestZlibDeflate#test_deflate_chunked_break [/home/jaruga/git/ruby/ruby/test/zlib/test_zlib.rb:92]:
<3632> expected but was
<4702>.

  5) Failure:
TestZlibGzipReader#test_unused2 [/home/jaruga/git/ruby/ruby/test/zlib/test_zlib.rb:968]:
<24> expected but was
<23>.

Finished tests in 2.985270s, 31.4879 tests/s, 236.1595 assertions/s.
94 tests, 705 assertions, 5 failures, 0 errors, 0 skips

ruby -v: ruby 3.3.0dev (2023-09-11T15:25:06Z master 05a853c2f2) [s390x-linux]
make: *** [uncommon.mk:914: yes-test-all] Error 5

...

The text was updated successfully, but these errors were encountered:

junaruga · 2023-09-15T16:13:31Z

Reproducer

I created a minimal reproducer below for the test test_deflate_chunked above.

$ cat test/zlib/_test_zlib_test_deflate_chunked.rb
require 'zlib'

z = Zlib::Deflate.new
input = "\x01"
z.deflate(input) do |chunk|
end
final = z.finish
puts "final.length: #{final.length}"

x86_64 Fedora 38 (ok)

On my local machine.

$ rpm -qf /lib64/libz.so.1
zlib-1.2.13-3.fc38.x86_64

$ ruby -I./lib test/zlib/_test_zlib_test_deflate_chunked.rb
final.length: 9

s390x Ubuntu jammy (not ok?)

$ dpkg -S /lib/s390x-linux-gnu/libz.so.1
zlib1g:s390x: /lib/s390x-linux-gnu/libz.so.1
$ dpkg -s zlib1g | grep ^Version
Version: 1:1.2.11.dfsg-2ubuntu9.2

$ ruby -I./lib test/zlib/_test_zlib_test_deflate_chunked.rb
final.length: 10

Debug

Debug 1

$ gdb --args ruby -I./lib test/zlib/_test_zlib_test_deflate_chunked.rb

(gdb) b rb_deflate_deflate

On the x86_64, after executing the do_deflate, the avail_out = 1022, total_out = 2

zlib/ext/zlib/zlib.c

Lines 1796 to 1805 in 85637fa

    
           rb_deflate_deflate(int argc, VALUE *argv, VALUE obj) 
        
           { 
        
               struct zstream *z = get_zstream(obj); 
        
               VALUE src, flush; 
        
               rb_scan_args(argc, argv, "11", &src, &flush); 
        
               do_deflate(z, src, ARG_FLUSH(flush)); 
        
               return zstream_detach_buffer(z); 
        
           }

(gdb) f   
#0  rb_deflate_deflate (argc=1, argv=0x7ffff7440060, obj=140737077406920)
    at ../../../../ext/zlib/zlib.c:1804
1804      return zstream_detach_buffer(z); 
(gdb) p *z
$6 = {flags = 3, buf = 140737077406680, input = 4, mutex = 140737077406880, stream = {
    next_in = 0x7fffe7816c19 "", avail_in = 0, total_in = 1, next_out = 0x5654c2 "", 
    avail_out = 1022, total_out = 2, msg = 0x0, state = 0x4c3880, 
    zalloc = 0x7ffff7e51771 <zlib_mem_alloc>, zfree = 0x7ffff7e5179e <zlib_mem_free>, 
    opaque = 0x7fffe7816cc8, data_type = 2, adler = 131074, reserved = 0}, 
  func = 0x7ffff7e5ecf0 <deflate_funcs>}

On the s390x, after executing the do_deflate, the values are avail_out = 1021, total_out = 3. These are different.

(gdb) f
#0  rb_deflate_deflate (argc=1, argv=0x3fffceff060, obj=4398000606760)
    at ../../../../ext/zlib/zlib.c:1804
1804>-    return zstream_detach_buffer(z);
(gdb) p *z
$8 = {flags = 3, buf = 4398000606520, input = 4, mutex = 4398000606720, stream = {
    next_in = 0x3fffd438d79 "", avail_in = 0, total_in = 1, next_out = 0x2aa001f4c93 "",-
    avail_out = 1021, total_out = 3, msg = 0x0, state = 0x2aa002762a0,-
    zalloc = 0x3fffa885b68 <zlib_mem_alloc>, zfree = 0x3fffa885bc8 <zlib_mem_free>,-
    opaque = 0x3fffd438e28, data_type = 2, adler = 131074, reserved = 0},-
  func = 0x3fffa896d08 <deflate_funcs>}

Debug 2

Then I watched when the z->stream.avail_out is changed.

(gdb) b zstream_run_try
(gdb) r
(gdb) watch z->stream.avail_out
(gdb) c
(gdb) c
...

x86_64

(gdb) c
Continuing.

Hardware watchpoint 3: z->stream.avail_out


Old value = 1024
New value = 1022
flush_pending (strm=strm@entry=0x6cf850) at /usr/src/debug/zlib-1.2.13-3.fc38.x86_64/deflate.c:824
824         s->pending      -= len;

(gdb) bt
#0  flush_pending (strm=strm@entry=0x6cf850)
    at /usr/src/debug/zlib-1.2.13-3.fc38.x86_64/deflate.c:824
#1  0x00007ffff7f9959a in deflate (strm=0x6cf850, flush=0)
    at /usr/src/debug/zlib-1.2.13-3.fc38.x86_64/deflate.c:922
#2  0x00007ffff7e525f0 in zstream_run_func (ptr=0x7fffffffc610) at ../../../../ext/zlib/zlib.c:1024
#3  0x00007ffff7aa1684 in rb_nogvl (func=0x7ffff7e52590 <zstream_run_func>, data1=0x7fffffffc610, 
    ubf=0x7ffff7e5274c <zstream_unblock_func>, data2=0x7fffffffc610, flags=2) at thread.c:1552
#4  0x00007ffff7e52890 in zstream_run_try (value_arg=140737488340496)
    at ../../../../ext/zlib/zlib.c:1115
...

(gdb) f
#0  flush_pending (strm=strm@entry=0x6cf850)
    at /usr/src/debug/zlib-1.2.13-3.fc38.x86_64/deflate.c:824
824	    s->pending      -= len;

(gdb) l
819	    zmemcpy(strm->next_out, s->pending_out, len);
820	    strm->next_out  += len;
821	    s->pending_out  += len;
822	    strm->total_out += len;
823	    strm->avail_out -= len;
824	    s->pending      -= len;
825	    if (s->pending == 0) {
826	        s->pending_out = s->pending_buf;
827	    }
828	}

(gdb) c
Continuing.

Watchpoint 3 deleted because the program has left the block in
which its expression is valid.
0x00007ffff78ff3f1 in rb_ensure (b_proc=0x7ffff7e5276a <zstream_run_try>, data1=140737488340496, 
    e_proc=0x7ffff7e52a37 <zstream_run_ensure>, data2=140737488340496) at eval.c:1009
1009	        result = (*b_proc) (data1);

s390x

(gdb) c
Continuing.

Hardware watchpoint 2: z->stream.avail_out

Old value = 0
New value = 1024
zstream_expand_buffer_into (z=0x2aa00272e20, size=1024) at ../../../../ext/zlib/zlib.c:714
714		rb_obj_hide(z->buf);
(gdb) c
Continuing.

Hardware watchpoint 2: z->stream.avail_out

Old value = 1024
New value = 1022
0x000003fffd60930e in ?? () from /lib/s390x-linux-gnu/libz.so.1
(gdb) c
Continuing.

Hardware watchpoint 2: z->stream.avail_out

Old value = 1022
New value = 1021
0x000003fffd60b7da in deflate () from /lib/s390x-linux-gnu/libz.so.1

(gdb) bt
#0  0x000003fffd60b7da in deflate () from /lib/s390x-linux-gnu/libz.so.1
#1  0x000003fffa88712e in zstream_run_func (ptr=0x3ffffffd7a0) at ../../../../ext/zlib/zlib.c:1024
#2  0x000003fffdc38202 in rb_nogvl (func=0x3fffa8870b0 <zstream_run_func>, data1=0x3ffffffd7a0, 
    ubf=0x3fffa887320 <zstream_unblock_func>, data2=0x3ffffffd7a0, flags=2) at thread.c:1552
#3  0x000003fffa8874e4 in zstream_run_try (value_arg=4398046500768)
    at ../../../../ext/zlib/zlib.c:1115
#4  0x000003fffda2066c in rb_ensure (b_proc=0x3fffa887368 <zstream_run_try>, data1=4398046500768, 
    e_proc=0x3fffa887738 <zstream_run_ensure>, data2=4398046500768) at eval.c:1009
...

(gdb) f
#0  0x000003fffd60b7da in deflate () from /lib/s390x-linux-gnu/libz.so.1

(gdb) l
709		/* I uses rb_str_new here not rb_str_buf_new because
710		   rb_str_buf_new makes a zero-length string. */
711		z->buf = rb_str_buf_new(size);
712		z->stream.next_out = (Bytef*)RSTRING_PTR(z->buf);
713		z->stream.avail_out = MAX_UINT(size);
714		rb_obj_hide(z->buf);
715	    }
716	    else if (z->stream.avail_out != size) {
717		rb_str_modify_expand(z->buf, size);
718		z->stream.next_out = (Bytef*)RSTRING_END(z->buf);

Now I want to install a debug symbol of the zlib package to print the lines in the zlib library in Ubuntu jammy. Do you know how to do it?

junaruga · 2023-09-15T21:05:07Z

Now I want to install a debug symbol of the zlib package to print the lines in the zlib library in Ubuntu jammy. Do you know how to do it?

All right. Now I was able to install the zlib package's debug symbol package on the s390x server.

I executed the page's "Getting -dbgsym.ddeb packages" section.
https://wiki.ubuntu.com/Debug%20Symbol%20Packages

I was not able to install the zlib1g-dbg package.

$ sudo apt-get install zlib1g-dbg
...
E: Package 'zlib1g-dbg' has no installation candidate

But I was able to install the zlib1g-dbgsym package.

$ sudo apt-get install zlib1g-dbgsym
...
The following NEW packages will be installed:
  zlib1g-dbgsym
0 upgraded, 1 newly installed, 0 to remove and 22 not upgraded.
...

The zlib debug symbol pacakge's version is same with the zlib package.

$ dpkg -s zlib1g-dbgsym | grep ^Version
Version: 1:1.2.11.dfsg-2ubuntu9.2

$ dpkg -s zlib1g | grep ^Version
Version: 1:1.2.11.dfsg-2ubuntu9.2

junaruga · 2023-09-15T21:53:50Z

First, here are the information about the Ubuntu zlib and upsgream zlib.

Ubuntu jammy zlib1g (zlib) package
- https://packages.ubuntu.com/jammy-updates/zlib1g
  
  Download Source Package zlib:
zlib upstream
- Official web: https://www.zlib.net/
- Source code: https://github.com/madler/zlib

On the s390x server, I debugged again. I was able to see the code line number now.

$ gdb --args ruby -I./lib test/zlib/_test_zlib_test_deflate_chunked.rb
Reading symbols from ruby...
(gdb) b zstream_run_try
Function "zstream_run_try" not defined.
Breakpoint 1 (zstream_run_try) pending.
(gdb) r
Starting program: /home/jaruga/.local/ruby-05a853c2f2-debug/bin/ruby -I./lib test/zlib/_test_zlib_test_deflate_chunked.rb
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1".

Breakpoint 1, zstream_run_try (value_arg=4398046500768) at ../../../../ext/zlib/zlib.c:1082
1082	    struct zstream_run_args *args = (struct zstream_run_args *)value_arg;
(gdb) watch z->stream.avail_out
Hardware watchpoint 2: z->stream.avail_out
(gdb) c
Continuing.

Hardware watchpoint 2: z->stream.avail_out

Old value = 0
New value = 1024
zstream_expand_buffer_into (z=0x2aa00272e20, size=1024) at ../../../../ext/zlib/zlib.c:714
714		rb_obj_hide(z->buf);
(gdb) c
Continuing.

Hardware watchpoint 2: z->stream.avail_out

Old value = 1024
New value = 1022
0x000003fffd60930e in flush_pending (strm=strm@entry=0x2aa00272e40) at deflate.c:806
806	deflate.c: No such file or directory.
(gdb) c
Continuing.

Hardware watchpoint 2: z->stream.avail_out

Old value = 1022
New value = 1021
0x000003fffd60b7da in dfltcc_cmpr (strm=0x2aa00272e40) at contrib/s390/dfltcc.c:307
307	contrib/s390/dfltcc.c: No such file or directory.

Hmm, the debug information is still not enough. I wanted to see the actual codes with lines.

(gdb) l
302	in contrib/s390/dfltcc.c

Here is the backtrace.

(gdb) bt
#0  0x000003fffd60b7da in dfltcc_cmpr (strm=0x2aa00272e40) at contrib/s390/dfltcc.c:307
#1  dfltcc_deflate (result=<synthetic pointer>, flush=<optimized out>, strm=0x2aa00272e40)
    at contrib/s390/dfltcc.c:475
#2  deflate (flush=<optimized out>, strm=0x2aa00272e40) at deflate.c:1062
#3  deflate (strm=0x2aa00272e40, flush=<optimized out>) at deflate.c:825
#4  0x000003fffa88712e in zstream_run_func (ptr=0x3ffffffd7a0) at ../../../../ext/zlib/zlib.c:1024
#5  0x000003fffdc38202 in rb_nogvl (func=0x3fffa8870b0 <zstream_run_func>, data1=0x3ffffffd7a0, 
    ubf=0x3fffa887320 <zstream_unblock_func>, data2=0x3ffffffd7a0, flags=2) at thread.c:1552
#6  0x000003fffa8874e4 in zstream_run_try (value_arg=4398046500768)
    at ../../../../ext/zlib/zlib.c:1115
...

But I couldn't find contrib/s390/dfltcc.c printed on the backtrace above, seeing the zlib 1.2.11's upstream source below.

https://github.com/madler/zlib/tree/v1.2.11/contrib

$ git remote -v
origin	https://github.com/madler/zlib.git (fetch)
origin	https://github.com/madler/zlib.git (push)


$ git branch
* (HEAD detached at v1.2.11)
  develop

$ pwd
/home/jaruga/git/zlib

$ grep -r dfltcc.c
  => Empty result

I downloaded the Ubuntu jammy zlib package's patch files.

https://packages.ubuntu.com/jammy-updates/zlib1g

$ wget http://archive.ubuntu.com/ubuntu/pool/main/z/zlib/zlib_1.2.11.dfsg-2ubuntu9.2.debian.tar.xz

$ tar xvf zlib_1.2.11.dfsg-2ubuntu9.2.debian.tar.xz

$ cd debian/

$ grep -r dfltcc.c * | wc -l
49

It seems the patches/410.patch is the patch file creating the contrib/s390/dfltcc.c file.

$ cat patches/410.patch
From 992a7afc3edfa511dff0650d1c545b11bf64e655 Mon Sep 17 00:00:00 2001
From: Ilya Leoshkevich <iii@linux.ibm.com>
Date: Wed, 18 Jul 2018 13:14:07 +0200
Subject: [PATCH] Add support for IBM Z hardware-accelerated deflate
...
 contrib/s390/dfltcc.c         | 996 ++++++++++++++++++++++++++++++++++
...

Hmm, I feel it's harder to debug the Ubuntu package than Debian package. Note Debian zlib package link is here. It seems Ubuntu doesn't have the open opened Git repository to track the package's patch files change. And I am family with Fedora Linux packaging things.

I found one zlib package s390x issue ticket affecting Ubuntu 22.04 (jammy) in the list of the zlib Ubuntu package bugs.

junaruga · 2023-09-15T22:08:07Z

It seems that following ticket is related to DFLTCC, reported from s390x machine (= "z15/LinuxONE III" on the ticket).
https://bugs.launchpad.net/ubuntu/+source/zlib/+bug/1990379

And the PR madler/zlib#410 is reported on the ticket. The PR is adding the contrib/s390/dfltcc.c file newly.

junaruga · 2023-09-15T22:25:32Z

I plan to run the ruby/zlib unit tests and the minimal reproducer (#60 (comment)), and debug on GDB on Ubuntu focal (the failures didn't happen on Ubuntu focal native s390x server), and newer versions Ubuntu and Fedora rawhide s390x containers on QEMU, and check the difference of the behaviors.

iii-i · 2023-09-25T08:18:10Z

Hi, madler/zlib#410 author here.

Things like

assert_equal 20016, deflated.length

are very likely to break with the DFLTCC hardware accelerator, since it produces a different (but still valid) compressed byte stream than the software implementation. I would suggest to either verify the validity of compressed streams differently (e.g., try to decompress them) or disable DFLTCC by setting the DFLTCC=0 environment variable.

junaruga · 2023-09-25T13:16:43Z

Hi @iii-i, thanks for your help providing the info!

I ran the reproducing script that I mentioned at #60 (comment) and the ruby/zlib's unit tests again with DFLTCC=0 in Ubuntu jammy s390x environment above. And I see the result of the reproducer is same with the x86_64 case, and the unit tests passed.

$ uname -m
s390x

$ dpkg -s zlib1g | grep ^Version
Version: 1:1.2.11.dfsg-2ubuntu9.2

$ DFLTCC=0 ruby -I./lib test/zlib/_test_zlib_test_deflate_chunked.rb
final.length: 9

$ DFLTCC=0 bundle exec rake test
...
------------------------------------------------------------------------------------------------------
95 tests, 518 assertions, 0 failures, 0 errors, 1 pendings, 0 omissions, 0 notifications
98.9474% passed
------------------------------------------------------------------------------------------------------
17.45 tests/s, 95.17 assertions/s
ruby01|s390x$ echo $?
0

And I can see the logic below in the madler/zlib#410 - contrib/s390/dfltcc.c

env = secure_getenv("DFLTCC");
env_dfltcc_disabled = env && !strcmp(env, "0");

Comparing the 410.patch between Ubuntu focal and jammy, the 410.patch is the same between the versions. However, the build options are different in the rules file, The --dfltcc and --crc32-vx are added in rules filel in Ubuntu jammy.

$ diff -u ubuntu_focal/zlib1g/debian/rules ubuntu_jammy/zlib1g/debian/rules
--- ubuntu_focal/zlib1g/debian/rules	2020-08-20 01:52:59.000000000 +0200
+++ ubuntu_jammy/zlib1g/debian/rules	2021-08-12 05:28:03.000000000 +0200
@@ -21,6 +21,9 @@
 LDFLAGS = `dpkg-buildflags --get LDFLAGS`
 EXTRA_MAKE =

+CONFIGURE_COMMON=--shared --prefix=/usr
+CONFIGURE_HOST=--libdir=\$${prefix}/lib/$(DEB_HOST_MULTIARCH)
+
 # binutils doesn't supply the prefixed version normally like GCC does so
 # we can't just unconditionally use DEB_HOST_GNU_TYPE-ar
 ifeq ($(DEB_HOST_GNU_TYPE),$(DEB_BUILD_GNU_TYPE))
@@ -46,8 +49,9 @@
 # s390x fails at compatibility.
 ifneq (,$(findstring $(DEB_HOST_ARCH), s390x))
 m32=-m31
-CFLAGS += -DDFLTCC
-EXTRA_MAKE += OBJA=dfltcc.o PIC_OBJA=dfltcc.lo
+CFLAGS += -DDFLTCC_LEVEL_MASK=0x7e
+CONFIGURE_COMMON += --dfltcc
+CONFIGURE_HOST += --crc32-vx
 else
 m32=-m32
 endif
@@ -95,7 +99,7 @@

 	if [ ! -f Makefile.stash ]; then cp Makefile Makefile.stash ; fi

-	AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc" CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/lib/$(DEB_HOST_MULTIARCH)
+	AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc" CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" uname=GNU ./configure $(CONFIGURE_COMMON) $(CONFIGURE_HOST)

 	touch $@

@@ -106,7 +110,7 @@
 	cp -r $(COPYLIST) debian/64
 	cd debian/64 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(m64)" \
 	    CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \
-	    uname=GNU ./configure --shared --prefix=/usr  --libdir=\$${prefix}/usr/lib64
+	    uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/lib64
 	touch $@

 configure32-stamp: configure
@@ -116,7 +120,7 @@
 	cp -r $(COPYLIST) debian/32
 	cd debian/32 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(m32)" \
 	    CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \
-	    uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/lib32
+	    uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/lib32
 	touch $@

 configuren32-stamp: configure
@@ -126,7 +130,7 @@
 	cp -r $(COPYLIST) debian/n32
 	cd debian/n32 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(mn32)" \
 	    CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \
-	    uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/lib32
+	    uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/lib32
 	touch $@

 configurex32-stamp: configure
@@ -136,7 +140,7 @@
 	cp -r $(COPYLIST) debian/x32
 	cd debian/x32 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(mx32)" \
 	    CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \
-	    uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/libx32
+	    uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/libx32
 	touch $@

 build: build-stamp $(EXTRA_BUILD)

I would suggest to either verify the validity of compressed streams differently (e.g., try to decompress them) or disable DFLTCC by setting the DFLTCC=0 environment variable.

Thanks for your suggestion. Right now it seems to me that it's a hard to verify the each test failing with enabled DFLTCC feature, I will send a PR to add a logic to run the ruby/zlib s390x case with DFLTCC=0.

junaruga · 2023-09-25T17:51:45Z

I sent the PR #63.

junaruga mentioned this issue Sep 11, 2023

.travis.yml: Upgrade the used Ubuntu s390x to 22.04 (Jammy). ruby/ruby#8401

Merged

junaruga mentioned this issue Sep 25, 2023

Workaround: Fix test failures on Ubuntu jammy s390x. #63

Merged

junaruga closed this as completed in #63 Sep 25, 2023

junaruga mentioned this issue Sep 26, 2023

Ubuntu jammy s390x: Test failures ruby/spec#1083

Closed

eregon mentioned this issue Sep 28, 2023

Workaround: Fix library zlib test failures on Ubuntu jammy s390x. ruby/spec#1084

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ubuntu jammy s390x: Test failures #60

Ubuntu jammy s390x: Test failures #60

junaruga commented Sep 11, 2023 •

edited

Loading

junaruga commented Sep 15, 2023 •

edited

Loading

junaruga commented Sep 15, 2023

junaruga commented Sep 15, 2023 •

edited

Loading

junaruga commented Sep 15, 2023 •

edited

Loading

junaruga commented Sep 15, 2023 •

edited

Loading

iii-i commented Sep 25, 2023

junaruga commented Sep 25, 2023 •

edited

Loading

junaruga commented Sep 25, 2023

Ubuntu jammy s390x: Test failures #60

Ubuntu jammy s390x: Test failures #60

Comments

junaruga commented Sep 11, 2023 • edited Loading

ruby/ruby

junaruga commented Sep 15, 2023 • edited Loading

Reproducer

x86_64 Fedora 38 (ok)

s390x Ubuntu jammy (not ok?)

Debug

Debug 1

Debug 2

x86_64

s390x

junaruga commented Sep 15, 2023

junaruga commented Sep 15, 2023 • edited Loading

junaruga commented Sep 15, 2023 • edited Loading

junaruga commented Sep 15, 2023 • edited Loading

iii-i commented Sep 25, 2023

junaruga commented Sep 25, 2023 • edited Loading

junaruga commented Sep 25, 2023

junaruga commented Sep 11, 2023 •

edited

Loading

junaruga commented Sep 15, 2023 •

edited

Loading

junaruga commented Sep 15, 2023 •

edited

Loading

junaruga commented Sep 15, 2023 •

edited

Loading

junaruga commented Sep 15, 2023 •

edited

Loading

junaruga commented Sep 25, 2023 •

edited

Loading