docker, core, editoast: add mode single-worker for all infra #9166

bougue-pe · 2024-10-03T08:09:53Z

Also:

editoast: remove http core client
cleanup unavailable envvar OSRD_BACKEND_URL

🔍 please review by commit + one more developper should test it (integration + e2e + regular use)
This work may probably be improved (doc, script, etc.) any suggestion is welcome.

Fix: #8599

codecov-commenter · 2024-10-03T08:12:10Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 4.34783% with 22 lines in your changes missing coverage. Please review.

Project coverage is 39.12%. Comparing base (8a66782) to head (7f4c167).
Report is 7 commits behind head on dev.

Files with missing lines	Patch %	Lines
...re/src/main/java/fr/sncf/osrd/cli/WorkerCommand.kt	0.00%	12 Missing ⚠️
editoast/src/core/mq_client.rs	0.00%	7 Missing ⚠️
editoast/src/client/mod.rs	0.00%	2 Missing ⚠️
editoast/src/core/mod.rs	0.00%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff              @@
##                dev    #9166      +/-   ##
============================================
- Coverage     39.15%   39.12%   -0.03%     
  Complexity     2245     2245              
============================================
  Files          1290     1289       -1     
  Lines         97316    97269      -47     
  Branches       3280     3280              
============================================
- Hits          38100    38061      -39     
+ Misses        57277    57269       -8     
  Partials       1939     1939

Flag	Coverage Δ
core	`74.92% <0.00%> (-0.02%)`	⬇️
editoast	`72.46% <9.09%> (-0.02%)`	⬇️
front	`10.33% <ø> (ø)`
gateway	`2.50% <ø> (ø)`
osrdyne	`3.52% <ø> (ø)`
railjson_generator	`87.49% <ø> (ø)`
tests	`86.71% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

flomonster · 2024-10-03T08:44:33Z

I did not reviewed it yet but I have a small question.

This features seems useful for debuging purpose (see #8599). I believe we're talking of core? If so I don't see why we should handle this feature using docker compose.

eckter

LGTM for core with one minor comment.

Not tested yet, I'll probably come back to it later.

core/src/main/java/fr/sncf/osrd/cli/WorkerCommand.kt

bougue-pe · 2024-10-03T08:57:08Z

I did not reviewed it yet but I have a small question.

This features seems useful for debuging purpose (see #8599). I believe we're talking of core? If so I don't see why we should handle this feature using docker compose.

Compose file here is more of a helper/documentation to plug all together, as editoast also needs some specific param.
And I tried to put the params/envvar at the best place (IMO 😅) to enable mutualization when reusing core in another compose.

younesschrifi

LGTM ! Didn't tested it yet. I'm waiting for your rebase to do so and then I'll approve the PR👍🏽

woshilapin

Looks good although, as discussed, it will make us not really use the full complexity of osrdyne on local, might not help to detect possible bugs in it. That said, all for making the developer experience smoother.

⚠️ not tested since I still didn't find the time to handle osrdyne that depends on a Docker daemon that I don't have (using podman, daemon less).

docker/docker-compose.host.yml

editoast/src/core/mod.rs

leovalais

Lgtm, not tested

ElysaSrc · 2024-10-07T06:54:54Z

I do not have anything to say about the PR per se, but I'm curious on why we would want to make this change the default behavior ?

What's the rationale behind this ?

bougue-pe · 2024-10-07T13:13:09Z

@ElysaSrc there is a bit more info in the related issue #8599 but you may already have had a look and I'm not sure about your question.

The main purpose is to ease debug of core (mainly without having to spawn a new core and track the id of the infra which can be painful, like for integration tests).
The solution provided is OK for the business logic and hijacks part of the ops/architecture part, so it is limited in that regard (but a bit less than switching back to the web api).

Then the rationale of changing the noopdyne compose file is to consider that using dyne as noop is (almost?) only for that use case, so we might as well add the "changes" to editoast and core conf on the way to ease/document that case.
So the use case (which I didn't test completly, as I realized few minutes ago) would be:

compose up the whole stack
down the component(s) that I want to debug/replace locally
spawn locally those components
<do actual test/use>

Hope that it answers your question, and maybe it's worth adding a bit of doc and a bit of description (will try to after more test, but any suggestion is welcome).

ElysaSrc · 2024-10-07T13:16:16Z

@ElysaSrc there is a bit more info in the related issue #8599 but you may already have had a look and I'm not sure about your question.

The main purpose is to ease debug of core (mainly without having to spawn a new core and track the id of the infra which can be painful, like for integration tests).
The solution provided is OK for the business logic and hijacks part of the ops/architecture part, so it is limited in that regard (but a bit less than switching back to the web api).

Then the rationale of changing the noopdyne compose file is to consider that using dyne as noop is (almost?) only for that use case, so we might as well add the "changes" to editoast and core conf on the way to ease/document that case.
So the use case (which I didn't test completly, as I realized few minutes ago) would be:

compose up the whole stack

down the component(s) that I want to debug/replace locally

spawn locally those components

<do actual test/use>

Hope that it answers your question, and maybe it's worth adding a bit of doc and a bit of description (will try to after more test, but any suggestion is welcome).

Thanks for the detailed response !

Erashin

Tested for core debug purposes. Works as intended. Good job :)

Khoyo

Great work!

Should we also rename noopdyne as single-worker? It wasn't a great name, and I mostly named it that way because it wasn't a single-worker mode ^^'

core/src/main/java/fr/sncf/osrd/cli/WorkerCommand.kt

docker-compose.yml

bougue-pe · 2024-10-11T19:39:35Z

Renamed in 2329db4 & d60f674:

osrd-compose.sh to host-compose.sh
docker-compose.noopdyne.yml to docker-compose.single-worker.yml

Also added a dedicated single-worker-compose.sh script to wrap this up with a bit more doc.

flomonster

LGTM (README change)

Erashin

Gj again :) The scale option is a huge plus :)

Also cleanup unavailable envvar OSRD_BACKEND_URL Signed-off-by: Pierre-Etienne Bougué <bougue.pe@proton.me>

Signed-off-by: Pierre-Etienne Bougué <bougue.pe@proton.me>

target ~1 min 30 s for each deps and 30 s for editoast and front Signed-off-by: Pierre-Etienne Bougué <bougue.pe@proton.me>

emersion

front/ changes LGTM (not tested)

In general huge +1 on the idea, although I haven't looked at all at the impl :P

bougue-pe requested review from Khoyo, woshilapin, leovalais, clarani and ElysaSrc October 3, 2024 08:09

bougue-pe requested review from a team as code owners October 3, 2024 08:09

github-actions bot added area:core Work on Core Service area:editoast Work on Editoast Service labels Oct 3, 2024

bougue-pe requested a review from Erashin October 3, 2024 08:10

eckter reviewed Oct 3, 2024

View reviewed changes

core/src/main/java/fr/sncf/osrd/cli/WorkerCommand.kt Outdated Show resolved Hide resolved

younesschrifi reviewed Oct 3, 2024

View reviewed changes

woshilapin approved these changes Oct 3, 2024

View reviewed changes

docker/docker-compose.host.yml Show resolved Hide resolved

editoast/src/core/mod.rs Outdated Show resolved Hide resolved

bougue-pe force-pushed the peb/osrdyne/one_core_for_all branch from c23a2d7 to e7b8258 Compare October 3, 2024 13:46

bougue-pe requested a review from younesschrifi October 3, 2024 14:02

leovalais approved these changes Oct 3, 2024

View reviewed changes

ElysaSrc approved these changes Oct 7, 2024

View reviewed changes

Erashin approved these changes Oct 9, 2024

View reviewed changes

bougue-pe force-pushed the peb/osrdyne/one_core_for_all branch from 302e4ef to 56cee29 Compare October 11, 2024 07:03

Khoyo reviewed Oct 11, 2024

View reviewed changes

core/src/main/java/fr/sncf/osrd/cli/WorkerCommand.kt Outdated Show resolved Hide resolved

docker-compose.yml Outdated Show resolved Hide resolved

bougue-pe force-pushed the peb/osrdyne/one_core_for_all branch from 56cee29 to d60f674 Compare October 11, 2024 19:32

bougue-pe requested a review from a team as a code owner October 11, 2024 19:32

github-actions bot added the area:front Work on Standard OSRD Interface modules label Oct 11, 2024

bougue-pe requested a review from Khoyo October 11, 2024 19:40

flomonster approved these changes Oct 14, 2024

View reviewed changes

Khoyo approved these changes Oct 15, 2024

View reviewed changes

Erashin approved these changes Oct 16, 2024

View reviewed changes

bougue-pe added 2 commits October 16, 2024 10:55

docker, core, editoast: add mode single-worker for all infra

9fdcf17

Also cleanup unavailable envvar OSRD_BACKEND_URL Signed-off-by: Pierre-Etienne Bougué <bougue.pe@proton.me>

editoast: remove http core client

14eb514

Signed-off-by: Pierre-Etienne Bougué <bougue.pe@proton.me>

bougue-pe force-pushed the peb/osrdyne/one_core_for_all branch from d60f674 to 14eb514 Compare October 16, 2024 08:58

docker: increase up delay for healthchecks

7f4c167

target ~1 min 30 s for each deps and 30 s for editoast and front Signed-off-by: Pierre-Etienne Bougué <bougue.pe@proton.me>

bougue-pe force-pushed the peb/osrdyne/one_core_for_all branch from d7c749a to 7f4c167 Compare October 16, 2024 14:22

emersion approved these changes Oct 16, 2024

View reviewed changes

clarani approved these changes Oct 17, 2024

View reviewed changes

bougue-pe added this pull request to the merge queue Oct 17, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 17, 2024

bougue-pe added this pull request to the merge queue Oct 17, 2024

bougue-pe removed this pull request from the merge queue due to a manual request Oct 17, 2024

bougue-pe added this pull request to the merge queue Oct 17, 2024

Merged via the queue into dev with commit a2dc87f Oct 17, 2024
24 checks passed

bougue-pe deleted the peb/osrdyne/one_core_for_all branch October 17, 2024 09:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker, core, editoast: add mode single-worker for all infra #9166

docker, core, editoast: add mode single-worker for all infra #9166

bougue-pe commented Oct 3, 2024 •

edited

Loading

codecov-commenter commented Oct 3, 2024 •

edited

Loading

flomonster commented Oct 3, 2024

eckter left a comment

bougue-pe commented Oct 3, 2024

younesschrifi left a comment •

edited

Loading

woshilapin left a comment •

edited

Loading

leovalais left a comment

ElysaSrc commented Oct 7, 2024

bougue-pe commented Oct 7, 2024 •

edited

Loading

ElysaSrc commented Oct 7, 2024

Erashin left a comment •

edited

Loading

Khoyo left a comment •

edited

Loading

bougue-pe commented Oct 11, 2024 •

edited

Loading

flomonster left a comment

Erashin left a comment

emersion left a comment

docker, core, editoast: add mode single-worker for all infra #9166

docker, core, editoast: add mode single-worker for all infra #9166

Conversation

bougue-pe commented Oct 3, 2024 • edited Loading

codecov-commenter commented Oct 3, 2024 • edited Loading

Codecov Report

flomonster commented Oct 3, 2024

eckter left a comment

Choose a reason for hiding this comment

bougue-pe commented Oct 3, 2024

younesschrifi left a comment • edited Loading

Choose a reason for hiding this comment

woshilapin left a comment • edited Loading

Choose a reason for hiding this comment

leovalais left a comment

Choose a reason for hiding this comment

ElysaSrc commented Oct 7, 2024

bougue-pe commented Oct 7, 2024 • edited Loading

ElysaSrc commented Oct 7, 2024

Erashin left a comment • edited Loading

Choose a reason for hiding this comment

Khoyo left a comment • edited Loading

Choose a reason for hiding this comment

bougue-pe commented Oct 11, 2024 • edited Loading

flomonster left a comment

Choose a reason for hiding this comment

Erashin left a comment

Choose a reason for hiding this comment

emersion left a comment

Choose a reason for hiding this comment

bougue-pe commented Oct 3, 2024 •

edited

Loading

codecov-commenter commented Oct 3, 2024 •

edited

Loading

younesschrifi left a comment •

edited

Loading

woshilapin left a comment •

edited

Loading

bougue-pe commented Oct 7, 2024 •

edited

Loading

Erashin left a comment •

edited

Loading

Khoyo left a comment •

edited

Loading

bougue-pe commented Oct 11, 2024 •

edited

Loading