Zetasql Building #288

aceforeverd · 2021-04-09T10:27:52Z

aceforeverd
Apr 9, 2021
Maintainer

Discussing zetasql building issues

My building status here: building status
at https://github.com/aceforeverd/zetasql/tree/dev

Build

mac build will fail for sure
centos 7 with devtoolset is able to compile but fails a few tests, same in ubuntu:18.04
linux with a bit newer toolchain is able to pass compile and test, including debian:buster, ubuntu:20.04 and archlinux

since ZetaSQL build every deps from source except gcc & glibc, I'm wondering if gcc/glibc cause those fails.

os	gcc	glibc	build	test
centos:7	7.3.1	2.17	pass	fail
centos:7	8.3.1	2.17	pass	fail
centos:7	9.3.1	2.17	pass	fail
centos:8	8.3.1	2.28	pass	pass
ubuntu:bionic	8.4.0	2.27	pass	fail
ubuntu:focal	9.3.0	2.31	pass	pass
Debian:buster	8.31	2.28	pass	pass
ArchLinux	10.2.0	2.33	pass	pass

glibc 2.27/2.28 seems the gap, but I have no idea how to test between different glibc version while keep every else the same

Problems and Solutions

protobuf 3.6.1 fail to compile on centos7 with devtoolset
compile seems can be fixed with Linking error when building Tensorflow 2.1 tensorflow/tensorflow#35867 (comment)

imotai · 2021-04-09T11:23:47Z

imotai
Apr 9, 2021

Google那边的研发公司内部都用ubuntu吧，我有点奇怪就是，他们用了什么东西，对操作系统依赖这么强

1 reply

jingchen2222 Apr 9, 2021

proto, icu, grpc,etc
有一些模块其实我们可能不需要。zetadql是可以提供service的所以用了grpc 。我们都是直接用库的

jingchen2222 · 2021-04-09T21:02:25Z

jingchen2222
Apr 9, 2021

My building status here: building status
at https://github.com/jingchen2222/zetasql/tree/dev

https://github.com/aceforeverd/zetasql/actions shows building errors of macos:

external/upb/upb/upb.h:39:3: error: '_Bool' is a C99 extension [-Werror,-Wc99-extensions]

We can fix those errors by updating grpc version to 1.25.0. It can address issue: https://github.com/protocolbuffers/upb/issues/212

Here are related issues:

jingchen2222/zetasql@cf05607#diff-a70ff1813d701c83652081d186aba748e180da906b69ec3b86e8d6383530d5e2

So far, -Wc99-extensions error has been solved. But we still have other errors triggered from abseil-cpp,
https://github.com/jingchen2222/zetasql/runs/2309337552?check_suite_focus=true

error executing command external/local_config_cc/cc_wrapper.sh -lc++ -fobjc-link-runtime -Wl,-S -shared -o bazel-out/darwin-fastbuild/bin/external/com_google_absl/absl/synchronization/libsynchronization.so ... (remaining 13 argument(s) skipped)

bazel build "..." --features=-supports_dynamic_linker seems solved the error above but it will trigger another error:

ERROR: com.google.devtools.build.lib.actions.MutableActionGraph$ActionConflictException: for zetasql/local_service/liblocal_service_jni.so, previous action: action 'Reporting failed target //zetasql/local_service:local_service_jni located at /Users/runner/work/zetasql/zetasql/zetasql/local_service/BUILD:242:1', attempted action: action 'Linking zetasql/local_service/liblocal_service_jni.so'
INFO: Elapsed time: 104.660s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (227 packages loaded, 11220 targets configured)
FAILED: Build did NOT complete successfully (227 packages loaded, 11220 targets configured)

5 replies

jingchen2222 Apr 10, 2021

Actually, we do not need zetasql java sdk and local service. Maybe we can build a simple zetasql

bazel build "//zetasql/analyzer:all" --features=-supports_dynamic_linker

Build Ok.
https://github.com/jingchen2222/zetasql/runs/2310296693?check_suite_focus=true

aceforeverd Apr 10, 2021
Maintainer Author

Sounds cool, I'll test this.

aceforeverd Apr 12, 2021
Maintainer Author

https://github.com/aceforeverd/zetasql/runs/2313112786?check_suite_focus=true
centos passed, ubuntu:bionic failed again 🐶

Anyway is zetasql:analyzer really have 5 tests only ?

jingchen2222 Apr 12, 2021

Yep, there is only 5 test in analyser/...
And macOS only fail 1 test. https://github.com/jingchen2222/zetasql/runs/2310493910?check_suite_focus=true

We should run tests under parser/... . It contains most of the tests.

jingchen2222 Apr 12, 2021

BTW, I can't build zetasql/analyser successfully on my local masOS.

jingchen2222 · 2021-04-14T00:27:51Z

jingchen2222
Apr 14, 2021

Build zetasql/... except local_service and jdk.
Pls try it on centos xxx and ubuntu xxx.
It works on my macOS. But still, fail many tests.

bazel build --features=-supports_dynamic_linker -- zetasql/... -zetasql/jdk:all -zetasql/local_service:all

@aceforeverd
https://github.com/jingchen2222/zetasql/pull/3

We can bazel test and ignore parser and reference_impl module

bazel test --test_summary=detailed --features=-supports_dynamic_linker -- zetasql/... -zetasql/jdk:all -zetasql/local_service:all -zetasql/reference_impl:all -zetasql/parser:all

Result:

INFO: Elapsed time: 14.092s, Critical Path: 13.92s
INFO: 4 processes: 1 internal, 3 darwin-sandbox.
INFO: Build completed successfully, 4 total actions
Test cases: finished with 9192 passing and 0 failing out of 9192 test cases

Executed 1 out of 128 tests: 128 tests pass.
INFO: Build completed successfully, 4 total actions

Now that we have passes 9000+ cases by skipping some module tests. Then we have these things to do:

Build zetasql (excluding jdk and local_service) on Ubuntu and Cento @aceforeverd
Figure out why tests fail in mac or other Linux os (if they fail)
Do survery on zetasql ASTNode, ASTExpression and related API
Figure out how to use zetasql compile results (e.g. libs and headers). @aceforeverd

2 replies

aceforeverd Apr 14, 2021
Maintainer Author

update:
centos7 and ubuntu:bionic build as same
https://github.com/aceforeverd/zetasql/runs/2341651817?check_suite_focus=true
ubuntu, devtoolset-7 and devtoolset-8/9 suffers different errors.

aceforeverd Apr 14, 2021
Maintainer Author

there are discussions how to use bazel build artifacts
https://groups.google.com/g/bazel-discuss/c/LP9tyCws_cg
bazelbuild/bazel#1920
bazelbuild/bazel#5200

jingchen2222 · 2021-04-15T01:24:59Z

jingchen2222
Apr 15, 2021

We fix zetasql to make parser and reference_impl tests work.
Now. MacOS can pass most of the cases except some timestamp_test and function cast case.

https://github.com/jingchen2222/zetasql/runs/2348401484?check_suite_focus=true

0 replies

aceforeverd · 2021-04-19T10:17:49Z

aceforeverd
Apr 19, 2021
Maintainer Author

update 0419:
I create a cmake based repo zetasql-sample, use zetasql (bazel based) as a dependency.
First test binary compiled & tested in my local environment. 🗿

3 replies

aceforeverd Apr 19, 2021
Maintainer Author

another target like resolver_test will fail:

[100%] Linking CXX executable resolver_test
[100%] Built target resolved_ast_test
/usr/bin/ld: /home/ace/Git/zetasql-sample/lib/libzetasql.a(sample_catalog.pic.o): in function `zetasql::TemplatedSQLTVF::TemplatedSQLTVF(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, zetasql::FunctionSignature const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, zetasql::ParseResumeLocation const&)':
sample_catalog.cc:(.text._ZN7zetasql15TemplatedSQLTVFC2ERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EERKNS_17FunctionSignatureESB_RKNS_19ParseResumeLocationE[_ZN7zetasql15TemplatedSQLTVFC5ERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EERKNS_17FunctionSignatureESB_RKNS_19ParseResumeLocationE]+0x37): undefined reference to `vtable for zetasql::TemplatedSQLTVF'
collect2: error: ld returned 1 exit status

concerned whether it is correct to merge all zetasql archive into single one

imotai Apr 24, 2021

I think it's easy to use if merging all zetasql archive into single one

aceforeverd Apr 25, 2021
Maintainer Author

yep, after learning some link rules from https://eli.thegreenplace.net/2013/07/09/library-order-in-static-linking, I think a united archive should be fine as long as no file name is duplicated.

aceforeverd · 2021-04-25T11:41:07Z

aceforeverd
Apr 25, 2021
Maintainer Author

zetasql as cmake dependency

There is a sample zetasql-sample

zetasql is bazel-based project and have its dependencies in the mean time. We'll use zetasql in a cmake proeject. Generally C-based project only need two things:

include headers
compiled shared library (.so) or static library (.a)

We can reorganize zetasql's build output (at $project/bazel-bin) to suffice those condition:

find and install compiled .so/.a into same directory (without directory hierarchy)
- for static library, using tools like ar or libtool to merge into single static archive
find every header file and install to a include directory
after telling cmake how to find zetasql's lib and include headers, it should link.

Exceptions
bazel's cc_library should produce both .a and .so by default. But a lib called template_sql_tvf do not follow this, it is built with likestatic=True in bazel's cc_library. Look like a implementation limit.

Transitive dependence

zetasql's thirdparty dependencies, should merge into HybridSQL's standard dependency list. Some may have breaking upgrade:

protobuf: form 2.x.x -> 3.x.x

Compiler and Options

zetasql is built with c++1z(c++17), since c++17 is stabilized in gcc 8 (not sure which specific version but 8.3 works fine), we should upgrade to gcc 8 or later. And use -std=c++1z as well because zetasql's include file use c++17 features.

checkout this for more: https://stackoverflow.com/questions/46746878/is-it-safe-to-link-c17-c14-and-c11-objects

0 replies

jingchen2222 · 2021-04-26T05:04:31Z

jingchen2222
Apr 26, 2021

Build Summary

Build and Test zetasql parser

Linux

Build

bazel build -- //... -zetasql/jdk/... -zetasql/local_service/... -java/... -javatests/...

Test

bazel test --test_summary=detailed  //zetasql/parser/...

Linux system compatibility can be shown as below table:

os	gcc	glibc	build	test
centos:7	7.3.1	2.17	pass	pass
centos:7	8.3.1	2.17	pass	pass
centos:7	9.3.1	2.17	pass	pass
centos:8	8.3.1	2.28	pass	pass
ubuntu:bionic	8.4.0	2.27	pass	pass
ubuntu:focal	9.3.0	2.31	pass	pass
Debian:buster	8.31	2.28	pass	pass
ArchLinux	10.2.0	2.33	pass	pass

Mac

bazel build --features=-supports_dynamic_linker -- //... -zetasql/jdk/... -zetasql/local_service/... -java/... -javatests/...

Test

bazel test --test_summary=detailed --features=-supports_dynamic_linker //zetasql/parser/...

Mac system compatibility can be shown as below table:

os	Clang	build	test
Mac:10.15	Apple clang version 12.0.0 (clang-1200.0.32.21)	pass	Pass

Build and Test Problems

absl

executing command external/local_config_cc/cc_wrapper.sh -lc++ -fobjc-link-runtime -Wl,-S -shared -o bazel-out/darwin-fastbuild/bin/external/com_google_absl/absl/synchronization/libsynchronization.so ... (remaining 13 argument(s) skipped)

abseil/abseil-cpp#848
abseil/abseil-cpp#848 (comment)
Workaround by using --features=-supports_dynamic_linker

grpc

external/upb/upb/upb.h:39:3: error: '_Bool' is a C99 extension [-Werror,-Wc99-extensions]

See workaround here jingchen2222/zetasql@cf05607#diff-a70ff1813d701c83652081d186aba748e180da906b69ec3b86e8d6383530d5e2

bazel test //zetasql/parser/...

FAIL: //zetasql/parser:parser_create_table_test (see /private/var/tmp/_bazel_runner/407c56d6e0400a81316224b3e40aaea4/execroot/com_google_zetasql/bazel-out/darwin-fastbuild/testlogs/zetasql/parser/parser_create_table_test/test.log)
FAIL: //zetasql/parser:parser_standalone_type_test (see /private/var/tmp/_bazel_runner/407c56d6e0400a81316224b3e40aaea4/execroot/com_google_zetasql/bazel-out/darwin-fastbuild/testlogs/zetasql/parser/parser_standalone_type_test/test.log)

See workaround here:
https://github.com/jingchen2222/zetasql/pull/3/files#diff-7ebc691e7f3f59ec4bc1719cb502fda627d8dcb1c4bbd4e2bc62370bcacaf6f1
zetasql/parser/builddefs.bzl

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Zetasql Building #288

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 0 comments 18 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Zetasql Building #288

aceforeverd Apr 9, 2021 Maintainer

Build

Problems and Solutions

Replies: 0 comments · 18 replies

aceforeverd Apr 10, 2021 Maintainer Author

aceforeverd Apr 12, 2021 Maintainer Author

aceforeverd Apr 14, 2021 Maintainer Author

aceforeverd Apr 14, 2021 Maintainer Author

aceforeverd Apr 19, 2021 Maintainer Author

aceforeverd Apr 19, 2021 Maintainer Author

aceforeverd Apr 25, 2021 Maintainer Author

aceforeverd Apr 25, 2021 Maintainer Author

zetasql as cmake dependency

Transitive dependence

Compiler and Options

Build Summary

Build and Test zetasql parser

Linux

Mac

Build and Test Problems

absl

grpc

bazel test //zetasql/parser/...

aceforeverd
Apr 9, 2021
Maintainer

Replies: 0 comments 18 replies

aceforeverd Apr 10, 2021
Maintainer Author

aceforeverd Apr 12, 2021
Maintainer Author

aceforeverd Apr 14, 2021
Maintainer Author

aceforeverd Apr 14, 2021
Maintainer Author

aceforeverd
Apr 19, 2021
Maintainer Author

aceforeverd Apr 19, 2021
Maintainer Author

aceforeverd Apr 25, 2021
Maintainer Author

aceforeverd
Apr 25, 2021
Maintainer Author