Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tasks: fail to submit workflow start operation #197

Open
diegodelemos opened this issue Jun 30, 2020 · 0 comments
Open

tasks: fail to submit workflow start operation #197

diegodelemos opened this issue Jun 30, 2020 · 0 comments
Labels
type/bug Something isn't working

Comments

@diegodelemos
Copy link
Member

diegodelemos commented Jun 30, 2020 •
edited
Loading

Currently, if there are dangling images in the Kubernetes node(s), the check_predefined_conditions call will fail when we call list_nodes because of kubernetes-client/python#895. The exact reason of this is described in kubernetes-client/python#895 (comment), there has been a fix attempt but it hasn't been tackled yet.

The traceback when this happens:

$ kubectl logs reana-server-xxxxx-yyy scheduler --previous
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/models/v1_container_image.py", line 75, in names  
    raise ValueError("Invalid value for `names`, must not be `None`")  # noqa: E501           
ValueError: Invalid value for `names`, must not be `None`

Note the usage of --previous, because the process exits you won't see any logs if you do kubectl logs reana-server-xxxxx-yyy as you will see the logs of the new pod.

Since we cannot control this, we should be more resilient and avoid blocking the whole system for this reason.
@diegodelemos diegodelemos added the type/bug Something isn't working label Jun 30, 2020
@diegodelemos diegodelemos changed the title tasks: fail to start workflow dangling images are present on the Kubernetes nodes tasks: fail to submit workflow start operation Jun 30, 2020
mvidalgarcia added a commit to mvidalgarcia/reana-commons that referenced this issue Jun 9, 2021
@mvidalgarcia
k8s: add monkeypatch to solve list_node() error
4be4c92 
To remove once Kubernetes releases the new version fixing it
reanahub#197
kubernetes-client/python#895
kubernetes/kubernetes#102159
mvidalgarcia added a commit to mvidalgarcia/reana-commons that referenced this issue Jun 9, 2021
@mvidalgarcia
k8s: add monkeypatch to solve list_node() error
5cb55aa 
To remove once Kubernetes releases the new version fixing it
reanahub#197
kubernetes-client/python#895
kubernetes/kubernetes#102159
@mvidalgarcia mvidalgarcia mentioned this issue Oct 18, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@diegodelemos