Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sliding Sync: Update filters to be robust against remote invite rooms #17450

Merged
merged 51 commits into from
Jul 30, 2024
Merged
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
3a6eec1
Make `filters.is_encrypted` robust to remote invites
MadLittleMods Jul 15, 2024
7f72d80
Try share work
MadLittleMods Jul 15, 2024
18ec437
More robust
MadLittleMods Jul 15, 2024
a67157e
Fix lints
MadLittleMods Jul 15, 2024
ffb61cf
Return bool from db
MadLittleMods Jul 15, 2024
9b77c97
Add tests
MadLittleMods Jul 16, 2024
d609205
Standardize
MadLittleMods Jul 16, 2024
8ca8ae5
Add TODO for more tests
MadLittleMods Jul 16, 2024
4f50054
Add changelog
MadLittleMods Jul 16, 2024
7ae2425
Invalidate caches
MadLittleMods Jul 16, 2024
b35a2da
Use `ROOM_UNKNOWN_SENTINEL`
MadLittleMods Jul 16, 2024
4bb34d3
Iterate on sentinels
MadLittleMods Jul 16, 2024
a07ee22
Simply sentinel for now
MadLittleMods Jul 16, 2024
2c9ec61
Merge branch 'develop' into madlittlemods/robust-remote-invite-filters
MadLittleMods Jul 16, 2024
27a97a0
fetch -> get
MadLittleMods Jul 16, 2024
517bfc4
Update comments
MadLittleMods Jul 16, 2024
d22bae4
Fix typo
MadLittleMods Jul 16, 2024
8624fb0
Better docstring
MadLittleMods Jul 16, 2024
a77b70e
Use constants
MadLittleMods Jul 16, 2024
df5093b
Prefer not implementing if not used
MadLittleMods Jul 18, 2024
7c33c83
Separate to its own function (`_bulk_get_stripped_state_for_rooms` ->…
MadLittleMods Jul 18, 2024
474b480
Use `StateMap`
MadLittleMods Jul 18, 2024
82fa037
Fix function name in assertion message
MadLittleMods Jul 18, 2024
6f05bb3
Add tests for when the server leaves the room
MadLittleMods Jul 18, 2024
d4c4de1
Ideally, not just current state
MadLittleMods Jul 18, 2024
0ace82d
Add `test_filter_encrypted_after_we_left`
MadLittleMods Jul 18, 2024
8c97eaa
Try get server left rooms (not working)
MadLittleMods Jul 19, 2024
8dd5a47
Implement `is_local_host_in_rooms(...)`
MadLittleMods Jul 19, 2024
5267b03
Start of join room_stats_current
MadLittleMods Jul 19, 2024
329bf09
Working `room_stats_state` joined by `room_stats_current.local_users_…
MadLittleMods Jul 19, 2024
d7dcbfd
Add corresponding tests for `filter.room_types`
MadLittleMods Jul 19, 2024
60ec4e3
Fix some lints
MadLittleMods Jul 19, 2024
affee22
Fix lints
MadLittleMods Jul 19, 2024
8ee1708
Explain why we join tables
MadLittleMods Jul 19, 2024
fe2d84b
Use `StrippedStateEvent` type in tests
MadLittleMods Jul 19, 2024
4ae3079
Prefer f-string
MadLittleMods Jul 22, 2024
05bbcb0
Add `_bulk_get_partial_current_state_content_for_rooms_from_sync_room…
MadLittleMods Jul 23, 2024
376d900
Simplify and share
MadLittleMods Jul 23, 2024
28a36e8
Remove unused methods
MadLittleMods Jul 23, 2024
ac7ca67
Maybe current state in the future
MadLittleMods Jul 23, 2024
15e157e
Fix cache busting
MadLittleMods Jul 23, 2024
19968e1
Fix lints
MadLittleMods Jul 23, 2024
8f5e4bf
Shorten name
MadLittleMods Jul 23, 2024
3c144b3
Reconsider all stripped events
MadLittleMods Jul 23, 2024
f3b4c9f
Merge branch 'develop' into madlittlemods/robust-remote-invite-filters
MadLittleMods Jul 23, 2024
1732b02
Prefer cache invalidation helper
MadLittleMods Jul 23, 2024
20902e6
Merge branch 'develop' into madlittlemods/robust-remote-invite-filters
MadLittleMods Jul 25, 2024
e9b76cc
Go back to dedicated `room_stats_state` lookups
MadLittleMods Jul 25, 2024
484007e
Update cache busting
MadLittleMods Jul 25, 2024
7acb1ad
Prefer `assert_never`
MadLittleMods Jul 30, 2024
45ed4ce
Merge branch 'develop' into madlittlemods/robust-remote-invite-filters
MadLittleMods Jul 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Make filters.is_encrypted robust to remote invites
MadLittleMods committed Jul 15, 2024
commit 3a6eec1cc88c3ee5a202f053ad00e52417175aa9
3 changes: 3 additions & 0 deletions synapse/api/constants.py
Original file line number Diff line number Diff line change
@@ -237,6 +237,9 @@ class EventContentFields:
# an unspecced field added to to-device messages to identify them uniquely-ish
TO_DEVICE_MSGID: Final = "org.matrix.msgid"

# `m.room.encryption`` algorithm field
ENCRYPTION_ALGORITHM: Final = "algorithm"


class EventUnsignedContentFields:
"""Fields found inside the 'unsigned' data on events"""
76 changes: 58 additions & 18 deletions synapse/handlers/sliding_sync.py
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@
import attr
from immutabledict import immutabledict

from synapse.api.constants import AccountDataTypes, Direction, EventTypes, Membership
from synapse.api.constants import AccountDataTypes, Direction, EventTypes, EventContentFields, Membership
from synapse.events import EventBase
from synapse.events.utils import strip_event
from synapse.handlers.relations import BundledAggregations
@@ -1109,23 +1109,63 @@ async def filter_rooms(

# Filter for encrypted rooms
if filters.is_encrypted is not None:
# Make a copy so we don't run into an error: `Set changed size during
# iteration`, when we filter out and remove items
for room_id in filtered_room_id_set.copy():
state_at_to_token = await self.storage_controllers.state.get_state_at(
room_id,
to_token,
state_filter=StateFilter.from_types(
[(EventTypes.RoomEncryption, "")]
),
# Partially-stated rooms should have all state events except for the
# membership events so we don't need to wait because we only care
# about retrieving the `EventTypes.RoomEncryption` state event here.
# Plus we don't want to block the whole sync waiting for this one
# room.
await_full_state=False,
)
is_encrypted = state_at_to_token.get((EventTypes.RoomEncryption, ""))
room_to_is_encrypted = await self.store.bulk_get_room_is_encrypted(
filtered_room_id_set
)
room_ids_with_results = set(room_to_is_encrypted.keys())

# We might not have room state for remote invite/knocks if we are the first
# person on our server to see the room. The best we can do is look in the
# optional stripped state from the invite/knock event.
room_ids_without_results = filtered_room_id_set.difference(
room_ids_with_results
)
invite_or_knock_event_ids: List[str] = []
for room_id in room_ids_without_results:
event_id = sync_room_map[room_id].event_id
# If this is an invite/knock then there should be an event_id
assert event_id is not None

if sync_room_map[room_id].membership in (
Membership.INVITE,
Membership.KNOCK,
):
invite_or_knock_event_ids.append(event_id)

invite_or_knock_events = await self.store.get_events(
invite_or_knock_event_ids
)
for invite_or_knock_event in invite_or_knock_events.values():
room_id = invite_or_knock_event.room_id
membership = sync_room_map[room_id].membership

stripped_state = []
if membership == Membership.INVITE:
stripped_state.extend(
invite_or_knock_event.unsigned.get("invite_room_state", [])
)
elif membership == Membership.KNOCK:
stripped_state.extend(
invite_or_knock_event.unsigned.get("knock_room_state", [])
)

for event in stripped_state:
if event["type"] == EventTypes.RoomEncryption:
is_encrypted = (
event["content"].get(
EventContentFields.ENCRYPTION_ALGORITHM
)
is not None
)
room_to_is_encrypted[room_id] = is_encrypted
break

for room_id in filtered_room_id_set:
is_encrypted = room_to_is_encrypted.get(room_id)

# Just remove rooms if we can't determine their encryption status
if is_encrypted is None:
filtered_room_id_set.remove(room_id)

# If we're looking for encrypted rooms, filter out rooms that are not
# encrypted and vice versa
46 changes: 44 additions & 2 deletions synapse/storage/databases/main/state.py
Original file line number Diff line number Diff line change
@@ -325,8 +325,9 @@ async def get_room_type(self, room_id: str) -> Optional[str]:
async def bulk_get_room_type(
self, room_ids: Set[str]
) -> Mapping[str, Optional[str]]:
"""Bulk fetch room types for the given rooms, the server must be in all
the rooms given.
"""Bulk fetch room types for the given rooms.

Will only return results if the server is in the room given.
"""

rows = await self.db_pool.simple_select_many_batch(
@@ -348,6 +349,47 @@ async def bulk_get_room_type(

return results

@cachedList(cached_method_name="get_room_is_encrypted", list_name="room_ids")
async def bulk_get_room_is_encrypted(
self, room_ids: Set[str]
) -> Mapping[str, Optional[str]]:
"""Bulk fetch whether the given rooms are encrypted.

Will only return results if the server is in the room given.
"""

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have a shortcut to check the get_partial_current_state_ids cache? Worth it?

rows = await self.db_pool.simple_select_many_batch(
table="room_stats_state",
column="room_id",
iterable=room_ids,
retcols=("room_id", "room_type"),
desc="bulk_get_room_type",
)

# If we haven't updated `room_stats_state` with the room yet, query the state
# directly. This should happen only rarely so we don't mind if we do this in a
# loop.
results = dict(rows)
for room_id in room_ids - results.keys():
state_map = await self.get_partial_filtered_current_state_ids(
room_id,
state_filter=StateFilter.from_types([(EventTypes.RoomEncryption, "")]),
)
encryption_event = state_map.get((EventTypes.RoomEncryption, ""))

is_encrypted = False
if encryption_event is not None:
is_encrypted = (
encryption_event.content.get(
EventContentFields.ENCRYPTION_ALGORITHM
)
is not None
)

results[room_id] = is_encrypted

return results

@cached(max_entries=100000, iterable=True)
async def get_partial_current_state_ids(self, room_id: str) -> StateMap[str]:
"""Get the current state event ids for a room based on the