Rights issue Radicale with Modoboa rights file

[LauraRozier]

When I try to manually log in to the Radicale server using their own web interface thing I seem to get a 401. (I fixed the issue with imap, btw, you now need to use a separate plugin for Radicale).
I also seem to be unable to create new agendas from within the Modoboa webclient (Using a mail-enabled user) I keep getting http/500 there, but probably a separate issue.

The log that I get when trying to access the calendars (A bit redacted, only changed the domain/tld):

[7f084d881780] INFO: PROPFIND request for '/' received from '217.100.199.170, 10.10.200.10' (forwarded by 127.0.0.1) using 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0'
[7f084d881780] DEBUG: Request headers:
{'CONTENT_LENGTH': '127',
 'CONTENT_TYPE': 'text/plain;charset=UTF-8',
 'HTTP_ACCEPT': '*/*',
 'HTTP_ACCEPT_ENCODING': 'gzip, deflate, br',
 'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.5',
 'HTTP_AUTHORIZATION': 'Basic **masked**',
 'HTTP_CONNECTION': 'close',
 'HTTP_COOKIE': '**masked**',
 'HTTP_HOST': 'localhost:5232',
 'HTTP_REFERER': 'https://mail.domain.tld/radicale/.web/',
 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) '
                    'Gecko/20100101 Firefox/61.0',
 'HTTP_X_FORWARDED_FOR': '217.100.199.170, 10.10.200.10',
 'HTTP_X_FORWARDED_HOST': 'mail.domain.tld',
 'HTTP_X_FORWARDED_PROTO': 'https',
 'HTTP_X_FORWARDED_SERVER': 'mail02.domain.tld',
 'HTTP_X_SCRIPT_NAME': '/radicale',
 'PATH_INFO': '/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REQUEST_METHOD': 'PROPFIND',
 'REQUEST_URI': '/',
 'SCRIPT_NAME': '',
 'SERVER_NAME': 'Mail02',
 'SERVER_PORT': '5232',
 'SERVER_PROTOCOL': 'HTTP/1.0',
 'UWSGI_APPID': 'localhost:5232|',
 'uwsgi.core': 0,
 'uwsgi.node': b'Mail02',
 'uwsgi.version': b'2.0.14-debian',
 'wsgi.errors': <_io.TextIOWrapper name=2 mode='w' encoding='UTF-8'>,
 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
 'wsgi.input': <uwsgi._Input object at 0x7f084722cc90>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 0)}
 
[7f084d881780] DEBUG: Script name overwritten by client: '/radicale'
[7f084d881780] DEBUG: Sanitized script name: '/radicale'
[7f084d881780] DEBUG: Sanitized path: '/'
[7f084d881780] INFO: Successful login: 'postmaster@domain.tld'

[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'postmaster@domain.tld' doesn't match 'admin':'.*' from section 'sa-admin-acr'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'postmaster@domain.tld' doesn't match 'postmaster@domain1.tld':'domain1.tld/user/.*' from section 'da-postmaster@domain1.tld-to-domain1.tld-acr'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'postmaster@domain.tld' doesn't match 'postmaster@domain2.tld':'domain2.tld/user/.*' from section 'da-postmaster@domain2.tld-to-domain2.tld-acr'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'postmaster@domain.tld' doesn't match 'postmaster@domain.tld':'domain.tld/user/.*' from section 'da-postmaster@domain.tld-to-domain.tld-acr'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'postmaster@domain.tld' doesn't match '^(.+)@(.+)$':'{1}/.+$' from section 'domain-shared-calendars'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'postmaster@domain.tld' matches '.+':'postmaster\\@domain\\.tld(/.*)?' from section 'owners-access'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'' doesn't match 'admin':'.*' from section 'sa-admin-acr'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'' doesn't match 'postmaster@domain1.tld':'domain1.tld/user/.*' from section 'da-postmaster@domain1.tld-to-domain1.tld-acr'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'' doesn't match 'postmaster@domain2.tld':'domain2.tld/user/.*' from section 'da-postmaster@domain2.tld-to-domain2.tld-acr'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'' doesn't match 'postmaster@domain.tld':'domain.tld/user/.*' from section 'da-postmaster@domain.tld-to-domain.tld-acr'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'' doesn't match '^(.+)@(.+)$':'{1}/.+$' from section 'domain-shared-calendars'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'' doesn't match '.+':'postmaster\\@domain\\.tld(/.*)?' from section 'owners-access'
[7f084d881780] INFO: Rights: 'postmaster@domain.tld':'' doesn't match any section

[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'' doesn't match 'admin':'.*' from section 'sa-admin-acr'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'' doesn't match 'postmaster@domain1.tld':'domain1.tld/user/.*' from section 'da-postmaster@domain1.tld-to-domain1.tld-acr'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'' doesn't match 'postmaster@domain2.tld':'domain2.tld/user/.*' from section 'da-postmaster@domain2.tld-to-domain2.tld-acr'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'' doesn't match 'postmaster@domain.tld':'domain.tld/user/.*' from section 'da-postmaster@domain.tld-to-domain.tld-acr'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'' doesn't match '^(.+)@(.+)$':'{1}/.+$' from section 'domain-shared-calendars'
[7f084d881780] DEBUG: Rule 'postmaster@domain.tld':'' doesn't match '.+':'postmaster\\@domain\\.tld(/.*)?' from section 'owners-access'
[7f084d881780] INFO: Rights: 'postmaster@domain.tld':'' doesn't match any section

[7f084d881780] INFO: Access to '/' denied for 'postmaster@domain.tld'
[7f084d881780] DEBUG: Response content:
Access to the requested resource forbidden.
[7f084d881780] INFO: PROPFIND response status for '/' in 4.573 seconds: 403 Forbidden
localhost:5232 [pid: 6868|app: 0|req: 7/15] 127.0.0.1 () {52 vars in 1075 bytes} [Mon Jul 16 17:46:55 2018] PROPFIND / => generated 61 bytes in 4572 msecs (HTTP/1.0 403) 3 headers in 111 bytes (1 switches on core 0)

[tonioo]

@thibmo Have you checked rights file content?

[LauraRozier]

Yep, I'll post it here, too as reference.
I generated the rights file via the modoboa command python manage.py generate_rights --force

The config file

root@Mail02:~# cat /etc/radicale/config
[auth]
# Authentication method
# Value: None | htpasswd | radicale_imap | remote_user | http_x_remote_user
type = radicale_imap

# Radicale_IMAP Configuration
imap_host = mail.domain.tld:143
imap_secure = True

[rights]
# Rights backend
# Value: None | authenticated | owner_only | owner_write | from_file | custom
type = from_file

# Custom rights handler
#custom_handler =

# File for rights management from_file
file = /etc/modoboa_radicale/rights

[logging]
debug = True

The rights file

root@Mail02:~# cat /etc/modoboa_radicale/rights
# Rights management file for Radicale
# This file was generated by Modoboa on 2018-07-17 09:32:02.315145
# DO NOT EDIT MANUALLY!

[sa-admin-acr]
user = admin
collection = .*
permission = rw

[da-postmaster@domain1.tld-to-domain1.tld-acr]
user = postmaster@domain1.tld
collection = domain1.tld/user/.*
permission = rw

[da-postmaster@domain2.tld-to-domain2.tld-acr]
user = postmaster@domain2.tld
collection = domain2.tld/user/.*
permission = rw

[da-postmaster@domain.tld-to-domain.tld-acr]
user = postmaster@domain.tld
collection = domain.tld/user/.*
permission = rw

# Access rule to domain shared calendars
[domain-shared-calendars]
user = ^(.+)@(.+)$
collection = {1}/.+$
permission = rw

# Read/Write permission for calendar owners
[owners-access]
user = .+
collection = %(login)s(/.*)?
permission = rw

The collection dir

root@Mail02:~# ls -la /var/lib/radicale/collections/
total 12
drwxrwxrwx 3 modoboa www-data 4096 Jul 16 17:18 .
drwxrwx--- 3 modoboa www-data 4096 Jul 16 17:18 ..
drwxrwxrwx 8 modoboa www-data 4096 Jul 16 17:32 collection-root
-rw-rw-rw- 1 modoboa www-data    0 Jul 16 17:44 .Radicale.lock
root@Mail02:~# ls -la /var/lib/radicale/collections/collection-root
total 32
drwxrwxrwx 8 modoboa www-data 4096 Jul 16 17:32 .
drwxrwxrwx 3 modoboa www-data 4096 Jul 16 17:18 ..
drwxrwxrwx 2 modoboa www-data 4096 Jul 16 17:31 abuse@domain.tld
drwxrwxrwx 2 modoboa www-data 4096 Jul 16 17:31 contact@domain.tld
drwxrwxrwx 2 modoboa www-data 4096 Jul 16 17:32 another_contact@domain.tld
drwxrwxrwx 2 modoboa www-data 4096 Jul 16 17:31 postmaster@domain1.tld
drwxrwxrwx 2 modoboa www-data 4096 Jul 16 17:31 postmaster@domain2.tld
drwxrwxrwx 2 modoboa www-data 4096 Jul 16 17:18 postmaster@domain.tld

[tonioo]

Which version of Radicale are you using?

[LauraRozier]

2.1.9

[tonioo]

And what calendar client do you use? Are you trying to access an owned calendar, or a shared one?

[LauraRozier]

I was using the Radicale's own webapp. With the Modoboa extention/webmail I only get the 500 errors.

[tonioo]

@thibmo How did you install the radicale plugin? Manually?

[LauraRozier]

Sorry for the late reply.

I installed it manually via:

sudo -u modoboa -i
bash
source env/bin/activate
cd instance/
pip install modoboa-radicale
python manage.py migrate
python manage.py collectstatic
python manage.py check --deploy

edit: Tried with the installer now, too. Same issue with modoboa giving a http 500 error when trying to create a calendar. (There also seem to be no default ones?)

[tonioo]

@thibmo The 500 error at creation might be due a bad certificate (ie self signed one or invalid certification chain)

[LauraRozier]

@tonioo Any way to retrieve this from a log?
I am using a wildcard certificate for the mailservers and webserver. I do have multiple domains, though.
I'll check and see if I can find an issue.

EDIT:
Enabled debugging and got this issue back:
ImportError at /api/v1/user-calendars/
cannot import name ical

Title	Value
Request Method:	POST
Request URL:	https://mail.server.tld/api/v1/user-calendars/
Django Version:	1.11.13
Exception Type:	ImportError
Exception Value:	cannot import name ical
Exception Location:	/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_radicale/backends/caldav_.py in <module>, line 7
Python Executable:	/usr/bin/uwsgi-core
Python Version:	2.7.13
Python Path:	['.', '', '/srv/modoboa/env/lib/python2.7', '/srv/modoboa/env/lib/python2.7/plat-x86_64-linux-gnu', '/srv/modoboa/env/lib/python2.7/lib-tk', '/srv/modoboa/env/lib/python2.7/lib-old', '/srv/modoboa/env/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/srv/modoboa/env/local/lib/python2.7/site-packages', '/srv/modoboa/env/lib/python2.7/site-packages']
Server time:	Sun, 19 Aug 2018 15:07:39 +0200

[LauraRozier]

Right, so I found the issue here.
Caldav release 0.5.0 doesn't have ical.py, this was added after the release of 0.5.0: 0af268c9c40415e9c464d994ca35fe925f461baf

Edit: Also had to execute the followin gin the virt env: pip install icalendar
Edit 2: Now I get gateway timeouts.. not sure why this is happening.

[LauraRozier]

Perhaps it'd be easier if I provide you with some temp credentials, then you can have a look at how things are on my server and what the real issue is here.
(It's a Debian container, btw)

[tonioo]

@thibmo You need to install the fork we made of caldav. Look here: https://github.com/modoboa/caldav.

[LauraRozier]

Thanks for the reply.
Pulled your fork but still have the issue.

[tonioo]

@thibmo How did you install it?

[LauraRozier]

The current one (30 days ago from today): Using the modoboa installer.
Then I followed the steps of the recent comments. I downloaded your caldav fork via wget for each (raw) file.

[tonioo]

To make sure the installation is correct, you can execute the following command (with the virtualenv loaded):

pip install -e git+https://github.com/modoboa/caldav#egg=caldav

[tonioo]

And reload uwsgi of course.

[mirtouf]

Is it possible to test with a virtualenv based on python 3 ?
I used to have the same issue with python 2.7 but not the same version of modoboa and modoboa-radicale though.

[LauraRozier]

Sorry for the long wait.
Just installed the egg, fixed a TLS version issue (Seems I needed to switch it from PROTOCOL_TLSv1_2 to PROTOCOL_TLS.
I restarted supervisord and uwsgi but still get the 500 error.
I can create calendars but I can't do anything else with them.

When inspecting /srv/radicale/collections/collection-root I do see directories but no content, also .Radicale.lock is dated to Aug 19.
If I can do anything to test or if you need something delivered, do tell me. I'm glad to help.

[mirtouf]

Are you using uwsgi for spawning radicale ?

[LauraRozier]

I used to, but this didn't work, so I switched to supervisord (Which the installer setup, iirc)

[mirtouf]

Would you mind copying the configuration for supervisor you are using ?

[LauraRozier]

Sureo

root@mail02:~# cat /etc/supervisor/conf.d/radicale.conf
# This file was automatically installed on 2018-07-21T21:08:16.272886
[program:radicale]
autostart=true
autorestart=true
command=/srv/radicale/env/bin/radicale -C /etc/radicale/config
directory=/srv/radicale
redirect_stderr=true
user=radicale
numprocs=1

[mirtouf]

I can reproduce this behaviour.

[LauraRozier]

What setup are you using that fixed the issue?
I'm guessing Python 3 virt env based on your earlier reply..

[mirtouf]

It helps, give me a moment I am trying to figure out where it fails.

[mirtouf]

OK, here the steps to make it work:

• use a python3 venv
• install modoboa-radicale, radicale (and its imap plugin) and caldav as described by @tonioo
• make radicale spawn with the method of your choice
• delete all past calendars through modoboa interface
• recreate calendars and events, it should work.
  By the way, using a reverse proxy for accessing radicale web interface does not work in my setup.

[LauraRozier]

Could you perhaps share your steps?
Perhaps I did it all a bit too simplistic, but what I did: (Note, with this I still get the 500 after removing and recreating the calendars)

root@mail02:~# sudo -u modoboa -i
$ /bin/bash
modoboa@mail02:~$ source env/bin/activate
(env) modoboa@mail02:~$ cd instance/
(env) modoboa@mail02:~/instance$ pip3 install modoboa-radicale
(env) modoboa@mail02:~/instance$ pip3 install -e git+https://github.com/modoboa/caldav#egg=caldav
(env) modoboa@mail02:~/instance$ python manage.py migrate
(env) modoboa@mail02:~/instance$ python manage.py collectstatic
(env) modoboa@mail02:~/instance$ python manage.py check --deploy
root@mail02:~# service supervisor stop
root@mail02:~# service supervisor start
root@mail02:~# service uwsgi restart

When doing a ps aunxf I get:

// Snipped to only show relevant parts
Ss   22:38   0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
S    22:38   0:00  \_ /srv/radicale/env/bin/python3 /srv/radicale/env/bin/radicale -C /etc/radicale/config
S    22:38   0:00 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/automx_instance.ini --daemonize /var/log/uwsgi/app/automx_instance.log
S    22:38   0:00  \_ /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/automx_instance.ini --daemonize /var/log/uwsgi/app/automx_instance.log
S    22:38   0:00  \_ /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/automx_instance.ini --daemonize /var/log/uwsgi/app/automx_instance.log
S    22:38   0:00 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/modoboa_instance.ini --daemonize /var/log/uwsgi/app/modoboa_instance.log
S    22:38   0:01  \_ /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/modoboa_instance.ini --daemonize /var/log/uwsgi/app/modoboa_instance.log
S    22:38   0:01  \_ /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/modoboa_instance.ini --daemonize /var/log/uwsgi/app/modoboa_instance.log

[mirtouf]

Can you confirm you installed imap plugin for radicale ?

[LauraRozier]

I just double checked and saw it was missing, so I did the following:

root@mail02:~# sudo -u radicale -i
radicale@mail02:~$ pwd
/srv/radicale
radicale@mail02:~$ source env/bin/activate
(env) radicale@mail02:~$ pip3 install radicale-imap
root@mail02:~# service supervisor stop
root@mail02:~# service supervisor start
root@mail02:~# service uwsgi restart

Checked again, same issue when deleting, then recreating. I noticed that when creating it also ignores the given color.

[tonioo]

@thibmo And the error is still the same? Make sure you installed all the required Radicale plugins. You can check what the installer does: https://github.com/modoboa/modoboa-installer/blob/master/modoboa_installer/scripts/radicale.py#L30.