Feature/devcontainer #731

pnoltes · 2024-02-19T09:39:44Z

This PR introduces a Development Container setup.

Development Container:

A development container (or dev container for short) allows you to use a container as a full-featured development environment. It can be used to run an application, to separate tools, libraries, or runtimes needed for working with a codebase, and to aid in continuous integration and testing. Dev containers can be run locally or remotely, in a private or public cloud, in a variety of supporting tools and editors.

Source https://containers.dev

I think the development container setup is nicer that our current containers dir, because the development container is based on a specification and is currently already supported in Clion and VSCode.

I tried building and debugging individual tests using a dev container in CLion and VSCode and both worked for me.
IMO the support for dev containers work better in VSCode. Also - if I am correct - VSCode has now a better CMake support and combined with the C++ TestMate plugin I think I will starting using VSCode more.

pnoltes · 2024-02-19T09:41:20Z

@rlenferink Could you have a look and see if the devcontainer setup works for you?

If you agree I would like to remove the container dir and gitpod badge in favor for the devcontainer setup.

.devcontainer/devcontainer.json

Co-authored-by: Gertjan Rolink <32330940+Deedss@users.noreply.github.com>

codecov-commenter · 2024-06-09T21:36:29Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.26%. Comparing base (758aa79) to head (f9c22aa).
Report is 8 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #731      +/-   ##
==========================================
+ Coverage   90.25%   90.26%   +0.01%     
==========================================
  Files         226      226              
  Lines       26320    26320              
==========================================
+ Hits        23755    23759       +4     
+ Misses       2565     2561       -4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Also update the dev container based on remote development testing with CLion.

pnoltes · 2024-06-23T18:53:45Z

I updated the devcontainer setup to use Ubuntu 24.04. For now, this is different from the Ubuntu CI. When the GitHub Ubuntu runners for 24.04 are no longer in beta, I would like to update our Ubuntu CI to 24.04 as well.

I also moved all the container setup to the .devcontainer directory.

conanfile.py

PengZheng · 2024-06-24T09:01:40Z

There is one remaining test failure:

26/43 Test #26: run_unit_test_discovery_zeroconf .................***Failed    0.01 sec
/workspace/build/bundles/remote_services/discovery_zeroconf/gtest/unit_test_discovery_zeroconfd: /lib/x86_64-linux-gnu/libssl.so.3: version `OPENSSL_3.2.0' not found (required by /root/.conan2/p/b/libcu5cd8ac887cea2/p/lib/libcurl.so.4)

When the tests are run under sudo, the system openssl rather than the one in Conan cache will be used.
We don't need sudo any more in container, so I drop it.

After dropping sudo, run_unit_test_discovery_zeroconf will hang. Please have a look, @xuzhenbao

# Conflicts: # libs/framework/gtest/src/CelixLauncherTestSuite.cc # libs/framework/src/celix_launcher.c

PengZheng · 2024-06-24T13:06:07Z

Besides the sudo issue, there seems to be shaky tests:
https://github.com/apache/celix/actions/runs/9644957822/job/26598173196?pr=731

[ RUN      ] DiscoveryZeroconfWatcherTestSuite.BrowseServicesFailed1
[2024-06-24T12:13:10] [   info] [celix_framework] [framework_start:483] Celix framework started
[2024-06-24T12:13:10] [  trace] [celix_framework] [framework_start:484] Celix framework started with uuid f82db217-e714-4955-8bf9-94a775f5fa59
[2024-06-24T12:13:10] [  debug] [celix_framework] Framework started event received -> registering framework.ready condition service
[2024-06-24T12:13:10] [  error] [DiscoveryZeroconf] Watcher: Failed to create watched services map.
/workspace/bundles/remote_services/discovery_zeroconf/gtest/src/DiscoveryZeroconfWatcherTestSuite.cc:706: Failure
Value of: timeOut
  Actual: true
Expected: false
[2024-06-24T12:13:40] [  trace] [celix_framework] [celix_framework_shutdownAsync:1157] Start shutdown thread for framework f82db217-e714-4955-8bf9-94a775f5fa59
[2024-06-24T12:13:40] [  trace] [celix_framework] [celix_bundleContext_cleanup:116] Cleaning up bundle context `celix_framework` (id=0)
[2024-06-24T12:13:40] [  trace] [celix_framework] [framework_shutdown:1110] Celix framework shutting down
[2024-06-24T12:13:40] [  trace] [celix_framework] [celix_framework_stopAndJoinEventQueue:1095] Stop and joining event loop thread for framework f82db217-e714-4955-8bf9-94a775f5fa59
[2024-06-24T12:13:40] [  debug] [celix_framework] [celix_framework_stopAndJoinEventQueue:1104] Joined event loop thread for framework f82db217-e714-4955-8bf9-94a775f5fa59
[  FAILED  ] DiscoveryZeroconfWatcherTestSuite.BrowseServicesFailed1 (30004 ms)

pnoltes · 2024-06-24T17:09:05Z

bundles/remote_services/discovery_zeroconf/gtest/src/DiscoveryZeroconfAnnouncerTestSuite.cc

@@ -53,11 +53,11 @@ static int GetLoopBackIfIndex(void);
 class DiscoveryZeroconfAnnouncerTestSuite : public ::testing::Test {
 public:
    static void SetUpTestCase() {
-        (void)system(MDNSD);
+        (void)system("sudo " MDNSD);


👍

If we need to start processes from C more often (e.g. etcd, msnd, etc) maybe we should also think about running integration test with help of something like behave or avocado. I think it is even possible to bootstrap behave/avocado from cmake/ctest using a add_test

I'll find some time to optimize the tests in another new PR.
I have tried to add FIXTURES in cmake yesterday, but due to some synchronization issues in the current unit tests, and I can't resolve them in a short time. so I'm going to start subprocess from C in this PR.

pnoltes · 2024-06-24T17:11:40Z

conanfile.py

@@ -340,6 +339,7 @@ def requirements(self):
            # https://github.com/conan-io/conan-center-index/pull/16254
            self.requires("mdnsresponder/1310.140.1")
        # 'libzip/1.10.1' requires 'zlib/1.2.13' while 'libcurl/7.64.1' requires 'zlib/1.2.12'
+        self.requires("openssl/[>=3.2.0]", override=True)


pnoltes · 2024-06-27T20:32:48Z

I think this PR is ready for review again. @rlenferink and/or @Deedss If able could you have a look at this PR?

Deedss

Overall LGTM, builts on Fedora 40 with SELinux enabled.

@pnoltes Please note that SELinux requires the presence of "securityOpt":["label=disable"] in the devcontainer.json to build. Otherwise SELinux blocks the execution of the conan scripts.

rlenferink · 2024-07-03T14:49:26Z

@EyeDevelop do you feel interested (with your VScode knowsledge) to review this PR?

@pnoltes I’ll review it somewhere this week.

rlenferink · 2024-07-11T18:49:56Z

@pnoltes I just tested this with Fedora + podman + CLion and at least for me this doesn't seem to work.

The steps I executed:

[rlenferink@fedora .devcontainer]$ git checkout feature/devcontainer
[rlenferink@fedora .devcontainer]$ cd .devcontainer
[rlenferink@fedora .devcontainer]$ ./build-devcontainer-image.sh
<snip>
[rlenferink@fedora .devcontainer]$ ./run-devcontainer.sh 
Do you want to mount the .gnupg directory to the container (as an overlayfs)? (yes/no): n
Do you want to mount the .gitconfig file to the container (as read-only)? (yes/no): no
Do you want to forward the SSH agent to the container? (yes/no): no
Starting container 'celixdev' with command: sudo /usr/sbin/sshd -D -e -p 2233
5a9d8ba9ffb813252bd6e77e67e97ad9da07ce364687e1e9a0e09ffbf8cc49dc

Do you want to build Celix dependencies with Conan? (yes/no): no
Done. You can connect with ssh using 'ssh -p 2233 celixdev@localhost' and password 'celixdev'

Then, using CLion to connect via SSH to the container succeeds, but editing files is impossible, since the files are marked as read-only.

Entering the container the error is indeed correct, since the rootless celixdev user is not allowed to write to the mounted volume.

[rlenferink@fedora .devcontainer]$ podman exec -it celixdev bash
celixdev@fedora:~/workspace$ touch test
touch: cannot touch 'test': Permission denied
celixdev@fedora:~/workspace$ ls -al
total 240
drwxr-xr-x. 1 rlenferink users       532 Jul 11 18:33 .
drwxr-x---. 1 celixdev   celixdev     66 Jul 11 18:43 ..
-rw-r--r--. 1 rlenferink users      2182 Jan  7  2024 .asf.yaml
-rw-r--r--. 1 rlenferink users      1616 Jan  7  2024 .clang-format
drwxr-xr-x. 1 rlenferink users       448 Jul 11 18:33 .devcontainer
drwxr-xr-x. 1 rlenferink users       226 Jul 11 18:33 .git
drwxr-xr-x. 1 rlenferink users        18 Jul 17  2022 .github
-rw-r--r--. 1 rlenferink users       949 Jan  7  2024 .gitignore
-rw-r--r--. 1 rlenferink users       988 Jan  7  2024 .gitpod.yml
drwxr-xr-x. 1 rlenferink users       146 Jan  7  2024 .idea
-rw-r--r--. 1 rlenferink users       534 Mar  7  2021 BUILDING
-rw-r--r--. 1 rlenferink users     22338 Jul 11 18:33 CHANGES.md
-rw-r--r--. 1 rlenferink users      9720 Jul 11 18:33 CMakeLists.txt
-rw-r--r--. 1 rlenferink users    107822 Apr 27  2023 Doxyfile
-rw-r--r--. 1 rlenferink users     11273 Mar  7  2021 KEYS
-rw-r--r--. 1 rlenferink users     18766 Jan  7  2024 LICENSE
-rw-r--r--. 1 rlenferink users       169 Jul 11 18:33 NOTICE
-rw-r--r--. 1 rlenferink users     10241 Jul 11 18:33 README.md
drwxr-xr-x. 1 rlenferink users       278 Jan  7  2024 build
drwxr-xr-x. 1 rlenferink users       206 Jul 11 18:33 bundles
drwxr-xr-x. 1 rlenferink users       250 Jul 11 18:33 cmake
drwxr-xr-x. 1 rlenferink users       196 Jan  7  2024 cmake-build-debug
drwxr-xr-x. 1 rlenferink users       196 Jan  7  2024 cmake-build-relwithdebinfo
-rw-r--r--. 1 rlenferink users       845 Jul 11 18:33 codecov.yml
-rw-r--r--. 1 rlenferink users     17174 Jul 11 18:33 conanfile.py
drwxr-xr-x. 1 rlenferink users        28 Apr 27  2023 doap
drwxr-xr-x. 1 rlenferink users       424 Jul 11 18:33 documents
drwxr-xr-x. 1 rlenferink users       134 Sep 23  2023 examples
drwxr-xr-x. 1 rlenferink users       164 Jan  7  2024 libs
drwxr-xr-x. 1 rlenferink users        60 Jul 17  2022 misc
-rw-r--r--. 1 rlenferink users      1665 Jan  7  2024 rat-excludes.txt

Am I doing something wrong here?

pnoltes · 2024-07-22T11:06:18Z

@pnoltes I just tested this with Fedora + podman + CLion and at least for me this doesn't seem to work.

The steps I executed:

[rlenferink@fedora .devcontainer]$ git checkout feature/devcontainer
[rlenferink@fedora .devcontainer]$ cd .devcontainer
[rlenferink@fedora .devcontainer]$ ./build-devcontainer-image.sh
<snip>
[rlenferink@fedora .devcontainer]$ ./run-devcontainer.sh 
Do you want to mount the .gnupg directory to the container (as an overlayfs)? (yes/no): n
Do you want to mount the .gitconfig file to the container (as read-only)? (yes/no): no
Do you want to forward the SSH agent to the container? (yes/no): no
Starting container 'celixdev' with command: sudo /usr/sbin/sshd -D -e -p 2233
5a9d8ba9ffb813252bd6e77e67e97ad9da07ce364687e1e9a0e09ffbf8cc49dc

Do you want to build Celix dependencies with Conan? (yes/no): no
Done. You can connect with ssh using 'ssh -p 2233 celixdev@localhost' and password 'celixdev'

Then, using CLion to connect via SSH to the container succeeds, but editing files is impossible, since the files are marked as read-only.

Entering the container the error is indeed correct, since the rootless celixdev user is not allowed to write to the mounted volume.

[rlenferink@fedora .devcontainer]$ podman exec -it celixdev bash
celixdev@fedora:~/workspace$ touch test
touch: cannot touch 'test': Permission denied
celixdev@fedora:~/workspace$ ls -al
total 240
drwxr-xr-x. 1 rlenferink users       532 Jul 11 18:33 .
drwxr-x---. 1 celixdev   celixdev     66 Jul 11 18:43 ..
-rw-r--r--. 1 rlenferink users      2182 Jan  7  2024 .asf.yaml
-rw-r--r--. 1 rlenferink users      1616 Jan  7  2024 .clang-format
drwxr-xr-x. 1 rlenferink users       448 Jul 11 18:33 .devcontainer
drwxr-xr-x. 1 rlenferink users       226 Jul 11 18:33 .git
drwxr-xr-x. 1 rlenferink users        18 Jul 17  2022 .github
-rw-r--r--. 1 rlenferink users       949 Jan  7  2024 .gitignore
-rw-r--r--. 1 rlenferink users       988 Jan  7  2024 .gitpod.yml
drwxr-xr-x. 1 rlenferink users       146 Jan  7  2024 .idea
-rw-r--r--. 1 rlenferink users       534 Mar  7  2021 BUILDING
-rw-r--r--. 1 rlenferink users     22338 Jul 11 18:33 CHANGES.md
-rw-r--r--. 1 rlenferink users      9720 Jul 11 18:33 CMakeLists.txt
-rw-r--r--. 1 rlenferink users    107822 Apr 27  2023 Doxyfile
-rw-r--r--. 1 rlenferink users     11273 Mar  7  2021 KEYS
-rw-r--r--. 1 rlenferink users     18766 Jan  7  2024 LICENSE
-rw-r--r--. 1 rlenferink users       169 Jul 11 18:33 NOTICE
-rw-r--r--. 1 rlenferink users     10241 Jul 11 18:33 README.md
drwxr-xr-x. 1 rlenferink users       278 Jan  7  2024 build
drwxr-xr-x. 1 rlenferink users       206 Jul 11 18:33 bundles
drwxr-xr-x. 1 rlenferink users       250 Jul 11 18:33 cmake
drwxr-xr-x. 1 rlenferink users       196 Jan  7  2024 cmake-build-debug
drwxr-xr-x. 1 rlenferink users       196 Jan  7  2024 cmake-build-relwithdebinfo
-rw-r--r--. 1 rlenferink users       845 Jul 11 18:33 codecov.yml
-rw-r--r--. 1 rlenferink users     17174 Jul 11 18:33 conanfile.py
drwxr-xr-x. 1 rlenferink users        28 Apr 27  2023 doap
drwxr-xr-x. 1 rlenferink users       424 Jul 11 18:33 documents
drwxr-xr-x. 1 rlenferink users       134 Sep 23  2023 examples
drwxr-xr-x. 1 rlenferink users       164 Jan  7  2024 libs
drwxr-xr-x. 1 rlenferink users        60 Jul 17  2022 misc
-rw-r--r--. 1 rlenferink users      1665 Jan  7  2024 rat-excludes.txt

Am I doing something wrong here?

I don't see anything wrong in your approach. The docker/podman run commands uses the --userns=keep-id argument, which should correctly map the celixdev user to the host user.

I'll retest this, but it may take some time before I can get to it.

pnoltes added 3 commits February 18, 2024 20:47

Add initial setup for devcontainer

eb38591

Fix trailing commas in devcontainer.json

74450e1

Add devcontainer badge in README.md

9310ddc

pnoltes requested a review from rlenferink February 19, 2024 09:39

Deedss reviewed Apr 9, 2024

View reviewed changes

.devcontainer/devcontainer.json Outdated Show resolved Hide resolved

pnoltes and others added 3 commits April 16, 2024 18:42

Update .devcontainer/devcontainer.json

dc8afe8

Co-authored-by: Gertjan Rolink <32330940+Deedss@users.noreply.github.com>

Merge branch 'refs/heads/master' into feature/devcontainer

b15249f

Refactor devcontainer setup for building with conan

6f45fb6

pnoltes added 16 commits June 10, 2024 20:20

Refactor Containerfile to build and dev with non root user

3ceb3c5

Update devcontainer setup for gpg

70922c9

Improve run container scripting and add initial readme

247744a

Combine container build ci with devcontainer setup

d28280e

Fix apt container ci build configuration

9dea26d

Update docker build command

8bc3f59

Move celixdev user to dev images

8e5ff23

Split container install, build & test in multiple steps

de8b49a

Add conan.lock file to lock dependencies when using conan

ada1c88

Add conan cache action for ci container build using conan.lock

8f678b5

Revert introduction conan.lock

e5a0640

Update Containerfile and conan cache setup

d2779cc

Add additional cache mechanisms for container ci builds

aa947f4

Remove disallowed docker cache action

fa845b9

Update devcontainer to ubuntu 24.04 and fix openssl issue

e69a3e2

Replace usage of tags for github action to hashes

6606e77

Also update the dev container based on remote development testing with CLion.

pnoltes commented Jun 23, 2024

View reviewed changes

conanfile.py Outdated Show resolved Hide resolved

PengZheng added 2 commits June 24, 2024 13:44

Avoid the introduction of openssl dependency.

d1ba2bf

Remove openssl 1.1.1 override for Conan build on Ubuntu.

d20f2b4

PengZheng added 7 commits June 24, 2024 14:48

Fix wrong conanrun.sh path.

a779cbf

Merge branch 'master' into feature/devcontainer

d4d41ce

Fix ctest warning.

1e77207

Fix ctest dynamic linking error.

14398f4

Fix conan option format.

52001cb

Update to macOS-12.

50e1193

Enable address sanitizer in test_package.

d3b1e15

PengZheng added 2 commits June 24, 2024 17:55

Remove sudo in run_unit_test_discovery_zeroconf.

95a34d8

Fix test_package error.

00dc17a

PengZheng assigned xuzhenbao Jun 24, 2024

Merge branch 'master' into feature/devcontainer

2206733

# Conflicts: # libs/framework/gtest/src/CelixLauncherTestSuite.cc # libs/framework/src/celix_launcher.c

xuzhenbao added 2 commits June 24, 2024 23:24

Use sudo to start mdnsd in discovery_zeroconf unit test

ae2243a

Resolve unit test failed for discovery_zeroconf

02a82bf

pnoltes commented Jun 24, 2024

View reviewed changes

PengZheng unassigned xuzhenbao Jun 25, 2024

pnoltes requested a review from Deedss June 27, 2024 20:31

Deedss approved these changes Jun 28, 2024

View reviewed changes

Add securityOpts config for devcontainer.json

f9c22aa

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/devcontainer #731

Feature/devcontainer #731

pnoltes commented Feb 19, 2024

pnoltes commented Feb 19, 2024

codecov-commenter commented Jun 9, 2024 •

edited

Loading

pnoltes commented Jun 23, 2024

PengZheng commented Jun 24, 2024 •

edited

Loading

PengZheng commented Jun 24, 2024 •

edited

Loading

pnoltes Jun 24, 2024

xuzhenbao Jun 25, 2024

pnoltes Jun 24, 2024

pnoltes commented Jun 27, 2024 •

edited

Loading

Deedss left a comment •

edited

Loading

rlenferink commented Jul 3, 2024

rlenferink commented Jul 11, 2024

pnoltes commented Jul 22, 2024

Feature/devcontainer #731

Are you sure you want to change the base?

Feature/devcontainer #731

Conversation

pnoltes commented Feb 19, 2024

pnoltes commented Feb 19, 2024

codecov-commenter commented Jun 9, 2024 • edited Loading

Codecov Report

pnoltes commented Jun 23, 2024

PengZheng commented Jun 24, 2024 • edited Loading

PengZheng commented Jun 24, 2024 • edited Loading

pnoltes Jun 24, 2024

Choose a reason for hiding this comment

xuzhenbao Jun 25, 2024

Choose a reason for hiding this comment

pnoltes Jun 24, 2024

Choose a reason for hiding this comment

pnoltes commented Jun 27, 2024 • edited Loading

Deedss left a comment • edited Loading

Choose a reason for hiding this comment

rlenferink commented Jul 3, 2024

rlenferink commented Jul 11, 2024

pnoltes commented Jul 22, 2024

codecov-commenter commented Jun 9, 2024 •

edited

Loading

PengZheng commented Jun 24, 2024 •

edited

Loading

PengZheng commented Jun 24, 2024 •

edited

Loading

pnoltes commented Jun 27, 2024 •

edited

Loading

Deedss left a comment •

edited

Loading