log rotation config is not working

[amitpatel-airlinq]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

The following command is execute
/usr/bin/etcd --config-file=/opt/Airlinq/etcd/etcd.conf.yml --enable-log-rotation=true --log-rotation-config-json="/opt/Airlinq/etcd/etcd-log-rotation.json" --log-outputs="/opt/Airlinq/etcd/logs/etcd.log"

getting the below error
{"level":"info","ts":"2024-09-13T20:51:36.757862Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["/usr/bin/etcd","--config-file=/opt/Airlinq/etcd/etcd.conf.yml","--enable-log-rotation=true","--log-rotation-config-json=/opt/Airlinq/etcd/etcd-log-rotation.json","--log-outputs=/opt/Airlinq/etcd/logs/etcd.log"]}
{"level":"warn","ts":"2024-09-13T20:51:36.758116Z","caller":"etcdmain/etcd.go:75","msg":"failed to verify flags","error":"error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go struct field configYAML.enable-log-rotation of type bool"}

What is the correct flags for log rotation to startup etcd.

What did you expect to happen?

The expected result is that the log file should be created in a required dir and it will be rotating as per the below values

{"maxsize": 1024, "maxage": 1, "maxbackups": 14, "localtime": true, "compress": true}

How can we reproduce it (as minimally and precisely as possible)?

use etcd 5.8.9 and execute the below command

/usr/bin/etcd --enable-log-rotation=true --log-rotation-config-json=/opt/Airlinq/etcd/etcd-log-rotation.json

Anything else we need to know?

entered the in etcd.conf.yml the following parameter it does not work

Etcd version (please run commands below)

$ etcd --version
[aqadmin@MIVMAMXETCD01 etcd]$ etcd --version
etcd Version: 3.5.9
Git SHA: bdbbde998
Go Version: go1.19.9
Go OS/Arch: linux/amd64

$ etcdctl version
[aqadmin@MIVMAMXETCD01 etcd]$ etcdctl version
etcdctl version: 3.5.9
API version: 3.5

</details>


### Etcd configuration (command line flags or environment variables)

<details>

# paste your configuration here
[aqadmin@MIVMAMXETCD01 etcd]$ cat etcd.conf.yml 
name: 'MIVMAMXETCD01'
data-dir: /opt/Airlinq/etcd/data
listen-peer-urls: http://10.121.69.160:2380
listen-client-urls: http://10.121.69.160:2379,http://localhost:2379
initial-advertise-peer-urls: http://10.121.69.160:2380
initial-cluster: 'MIVMAMXETCD01=http://MIVMAMXETCD01:2380,MIVMAMXETCD02=http://MIVMAMXETCD02:2380,MIVMAMXETCD03=http://MIVMAMXETCD03:2380'
initial-cluster-state: 'new'
initial-cluster-token: 'etcd-cluster'
advertise-client-urls: http://10.121.69.160:2379
enable-v2: 'true'
enable-log-rotation: 'yes'
log-rotation-config-json: {"maxsize": 1024, "maxage": 1, "maxbackups": 14, "localtime": true, "compress": true}


# JSON File
[aqadmin@MIVMAMXETCD01 etcd]$ cat etcd-log-rotation.json 
{
  "maxsize": 1024,
  "maxage": 1,
  "maxbackups": 14,
  "localtime": true,
  "compress": true
}
</details>


### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

<details>

```console
$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

{"level":"info","ts":"2024-09-13T20:51:36.757862Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["/usr/bin/etcd","--config-file=/opt/Airlinq/etcd/etcd.conf.yml","--enable-log-rotation=true","--log-rotation-config-json=/opt/Airlinq/etcd/etcd-log-rotation.json","--log-outputs=/opt/Airlinq/etcd/logs/etcd.log"]}
{"level":"warn","ts":"2024-09-13T20:51:36.758116Z","caller":"etcdmain/etcd.go:75","msg":"failed to verify flags","error":"error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go struct field configYAML.enable-log-rotation of type bool"}

[jmhbnz]

Hi @amitpatel-airlinq - As covered in the etcd logging documentation https://etcd.io/docs/v3.5/op-guide/configuration/#logging, have you tried passing the json configuration in line?

i.e.:

--log-rotation-config-json '{"maxsize": 100, "maxage": 0, "maxbackups": 0, "localtime": false, "compress": false}'

[jbml]

First of all, as stated in https://etcd.io/docs/v3.5/op-guide/configuration/#logging:

If you provide a configuration file all command-line flags and environment variables are ignored.

Which means that in the way you start your etcd server:
sh > /usr/bin/etcd --config-file=/opt/Airlinq/etcd/etcd.conf.yml --enable-log-rotation=true --log-rotation-config-json="/opt/Airlinq/etcd/etcd-log-rotation.json" --log-outputs="/opt/Airlinq/etcd/logs/etcd.log"

all command line options are ignored, and the command boils down to:
sh > /usr/bin/etcd --config-file=/opt/Airlinq/etcd/etcd.conf.yml

And when you execute that, you'll see that, as expected, you get the same error you reported:
error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go struct field configYAML.enable-log-rotation of type bool

The error points to the value of enable-log-rotation being invalid, which makes sense considering the configuration file contains:

enable-log-rotation: 'yes'

Change that to

enable-log-rotation: true

and you'll see that the next error is:
cannot unmarshal object into Go struct field configYAML.log-rotation-config-json of type string

This is due to the lack of quotes in your configuration file for the log-rotation-config-json parameter, so the next thing you want to modifiy, is as follows (notice the single quotes):

log-rotation-config-json: '{"maxsize": 1024, "maxage": 1, "maxbackups": 14, "localtime": true, "compress": true}'

The next error from there is:
--log-outputs requires a single file path when --log-rotate-config-json is defined

That makes sense, because as said at the start of this comment, all the command line arguments are ignored, including your --log-outputs. So the next change in your configuration file, is to add (notice the brackets, as this is a list parameter):

log-outputs: ['/opt/Airlinq/etcd/logs/etcd.log']

In summary, you should not mix command line and configuration file options, and only define your startup parameters in the configuration file. The configuration file should look like this:

name: 'MIVMAMXETCD01'
[...]
enable-log-rotation: true
log-rotation-config-json: '{"maxsize": 1024, "maxage": 1, "maxbackups": 14, "localtime": true, "compress": true}'
log-outputs: ['/opt/Airlinq/etcd/logs/etcd.log']

I hope this helps.

[amitpatel-airlinq]

I have tried this configuration from the config file and so far I have not got any error, etcd has started. I am waiting for the rotation to work.

ETCD_ENABLE_LOG_ROTATION="true"
ETCD_LOG_ROTATION_CONFIG_JASON="{"maxsize": 1024, "maxage": 1, "maxbackups": 14, "localtime": true, "compress": true}"

Further more it would help with the error where the "enable-log-rotation: " is not with the correct value then instead of error a warn mentioning "ignoring dependant parameters will be ignored" will really help.

[amitpatel-airlinq]

This has worked with the following entries. where the log files is created in the specified dir and the log did rotate on restart of the service.

ETCD_LOG_OUTPUTS="stderr,/opt/Airlinq/logs/etcd/etcd.log"
ETCD_ENABLE_LOG_ROTATION="true"
ETCD_LOG_ROTATION_CONFIG_JASON="{"maxsize": 1024, "maxage": 1, "maxbackups": 14, "localtime": true, "compress": true}"