Prometheus failing to start.

[radicalgeek]

I have just deployed a fresh swarm using the docker CE cloudformation template, followed by Traefic (https://dockerswarm.rocks/traefik/) then swarmprom. I have followed the guides exactly, and have the latest version of this repository.

All the containers come up with the exception of prometheus which seems to have some issues around locking the /prometheus directory. Here is the output from the container.

12:00:16
level=info ts=2019-04-01T12:00:16.485249182Z caller=main.go:244 msg="Starting Prometheus" version="(version=2.5.0, branch=HEAD, revision=67dc912ac8b24f94a1fc478f352d25179c94ab9b)"

12:00:16
level=info ts=2019-04-01T12:00:16.485431331Z caller=main.go:245 build_context="(go=go1.11.1, user=root@578ab108d0b9, date=20181106-11:40:44)"

12:00:16
level=info ts=2019-04-01T12:00:16.485492975Z caller=main.go:246 host_details="(Linux 4.9.81-moby #1 SMP Thu Mar 22 18:46:05 UTC 2018 x86_64 16f80b7a80df (none))"

12:00:16
level=info ts=2019-04-01T12:00:16.485611133Z caller=main.go:247 fd_limits="(soft=1048576, hard=1048576)"

12:00:16
level=info ts=2019-04-01T12:00:16.485669096Z caller=main.go:248 vm_limits="(soft=unlimited, hard=unlimited)"

12:00:16
level=info ts=2019-04-01T12:00:16.486263519Z caller=main.go:562 msg="Starting TSDB ..."

12:00:16
level=info ts=2019-04-01T12:00:16.488751312Z caller=main.go:431 msg="Stopping scrape discovery manager..."

12:00:16
level=info ts=2019-04-01T12:00:16.48884943Z caller=main.go:445 msg="Stopping notify discovery manager..."

12:00:16
level=info ts=2019-04-01T12:00:16.488903369Z caller=main.go:467 msg="Stopping scrape manager..."

12:00:16
level=info ts=2019-04-01T12:00:16.488974076Z caller=main.go:441 msg="Notify discovery manager stopped"

12:00:16
level=info ts=2019-04-01T12:00:16.490750149Z caller=web.go:399 component=web msg="Start listening for connections" address=0.0.0.0:9090

12:00:16
level=info ts=2019-04-01T12:00:16.508059116Z caller=main.go:427 msg="Scrape discovery manager stopped"

12:00:16
level=info ts=2019-04-01T12:00:16.513500817Z caller=manager.go:657 component="rule manager" msg="Stopping rule manager..."

12:00:16
level=info ts=2019-04-01T12:00:16.513604072Z caller=manager.go:663 component="rule manager" msg="Rule manager stopped"

12:00:16
level=info ts=2019-04-01T12:00:16.513669156Z caller=notifier.go:512 component=notifier msg="Stopping notification manager..."

12:00:16
level=info ts=2019-04-01T12:00:16.513742797Z caller=main.go:616 msg="Notifier manager stopped"

12:00:16
level=info ts=2019-04-01T12:00:16.517522311Z caller=main.go:461 msg="Scrape manager stopped"

12:00:16
level=error ts=2019-04-01T12:00:16.523471029Z caller=main.go:625 err="opening storage failed: lock DB directory: open /prometheus/lock: permission denied"

[grostim]

I have the same error.
I suspect that the file permissions are not set correctly on the volumes binded to /prometheus.
Which user id shoudl i give full authorization to this folder ?

[Thyrst]

Prometheus is using nobody user. This should solve your problem: chown nobody:nogroup /prometheus/