The SQLite implementation for the
EventCache::find_event_with_relations() the relation type list isn't
inserted using SQL placeholders.
The relation types are inserted manually using a format!() call. The
usage of the format!() call can lead to SQL injection if a
RelationType::Custom variant is used which contains SQL expressions.
This patch modifies the, query logic which retrieves the related events,
to use two separate queries which use SQL placeholders to insert all
the dynamic variables.
Security-Impact: Moderate
CVE: CVE-2025-53549
GitHub-Advisory: GHSA-275g-g844-73jh
This patch adds the new `RoomListService::new_with_share_pos`
constructor. It decides whether the `share_pos` feature of sliding sync
should be enabled or not.
`SyncServiceBuilder` gains a new `with_share_pos` method to configure
the way the `RoomListService` is built.
The FFI bindings are updated accordingly.
This patch updates the `assert_update` macro in the send queue test
suites to test that `SendQueue::subscribe` provides the same data that
`RoomSendQueue::subscribe`. It ensures `SendQueueUpdate` won't miss
an update.
To get updates from the `SendQueue`, one needs to use
`RoomSendQueue::subscribe`, it emits `RoomSendQueueUpdate`s. However, if
one wants to receive updates for all rooms managed by the `SendQueue`,
instead of subscribing to all `RoomSendQueue` individually, this
patch introduces a new `SendQueue::subscribe` method, which emits
`SendQueueUpdate`s. It's basically a wrapper around
`RoomSendQueueUpdate` by adding an `OwnedRoomId`.
This patch renames variants of `RoomEventCacheGenericUpdate`:
- `TimelineUpdated` becomes `UpdateTimeline`,
- `Cleared` becomes `Clear`.
It matches the rest of the codebase where we use _verb_ + _subject_.
This reorders methods so that they're grouped in "dual" pairs
(encode/decode, serialize/deserialize). Also adds a doc comment to
`encode_key`, as I've wondered in the past what it did.
This patch updates `RoomListService::subscribe_to_rooms` to call
`LatestEvents::listen_to_room` automatically. This method becomes async,
which propagates to a couple of callers.
The idea is that when one is interested by a specific room, a
subscription will be applied. This is an opportunity to also “activate”
the computation of the `LatestEvent` for this specific room, so that the
user doesn't have to do that manually (except if room subscription is
never used).
This patch adds a new log field, named `request_duration` in
`HttpClient::send_request`. It helps to know how much time the request
took to be executed.
This is a bugfix for the `compat-lax-room-create-deser` and
`compat-lax-room-topic-deser` features, which didn't work with events
in the wild.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch removes an optimisation that was considering
`LatestEventValue::None` as a value that should no overwrite the
previous value if any. This is a mistake: if the event cache is cleared,
it won't be possible to calculate a new value, and the previous value
must be overwritten.
This patch adds new variants to `LatestEventValue` that represent
all the possible suitable latest events. It's heavily inspired by
`matrix_sdk_base::latest_event::PossibleLatestEvent`. It uses the
new `find_and_map` method to find a suitable `TimelineEvent` and to
map it into a `LatestEventValue`. This method is heavily inspired by
`matrix_sdk_base::latest_value::is_suitable_for_latest_event`.
To be able to provide the `power_levels` to `find_and_map`, a
`WeakClient` is required by `LatestEvents::new`. It flows up to a
`WeakRoom` in `RoomLatestEvents`, used to create or to update the
`LatestEventValue`.
A particular care is applied to re-compute the `power_levels` only when
necessary and once **per room** (and not per latest event value).
Fetching the `power_levels` requires an access to the storage, it's
not anodyne.
This is more precise than using the event timeline item kind: indeed,
when the event for which we want to add a reaction is a local echo that
has been sent, it is still marked as a local echo, but the send queue
will not know about it, and thus will not be able to add the reaction
immediately, leading to a silent failure.
The `TimelineItemHandle` was made to help with this kind of situation:
in this case, it will return `TimelineItemHandle::Remote`, even though
the item is a local echo that has been sent; that way we can use the
event id, and correctly send a (remote) reaciton event immediately.
The following commit includes a regression test.
Using this type will be mandatory in the next breaking release of Ruma,
that will gain support for recognizing unstable features. Besides, it
allows to cache the supported versions and features in a single
CachedValue, which makes more sense than separately because we always
get both at the same time in production.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch fixes the markup in the `README.md` file. Links were broken because of the mix of HTML and Markdown. Plus, the HTML markup was kind of incorrect. Plus, too much `<br />`, most of them were “collapsed” automatically.
Signed-off-by: Ivan Enderlin <ivan@mnt.io>
Usually tests that don't construct it via MockMatrixServer don't care
about the homeserver URL, so this allows to provide a default URL for
them.
Also we don't force to allocate a string when the inner API actually
uses a borrowed string.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
It seems that some `m.room.topic` events in the wild have the wrong
format for the `m.topic` field. Since this field was stabilized recently
in Ruma, deserializing these events now fails. We added a feature to
ignore this field if its deserialization fails.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
The `EventContent` trait is gone in Ruma so this will ease the upgrade
to the next breaking release.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
With the UI crate now sending threaded read receipts we need to start considering them when computing unread counts. As a first step before the participation model, threaded messages will be ignored when computing room unread counts.
This patch moves away from always sending unthreaded read receipts to checking the timeline's focus and `hide_threaded_events` associated values to see whether `ReceiptThread::Main` or `ReceiptThread::Thread` should be used.
It doesn't make sense to name a thing "Matrix" when it comes from the
core SDK crate.
Also adjust a now slightly-incorrect doc comment for `get_notification`.
These are not included in Element's main privacy policy, and may
constitute PII (because the homeserver may include the name of some
user). We keep the information as separate log lines, so that
rageshakes/manual reports still include those.
The metadata loading shouldn't cause an error to be displayed, when the
linked chunk is empty; this can happen for new rooms we've never
visited.
Spotted while investigating some failures in a rageshake.
This patch introduces a new method named
`RoomEventCache::rfind_event_in_memory_by` to look for an in-memory
event, starting from the most recent event, by applying a predicate
closure on each event.
This patch renames the `RoomEventCache::event_with_relations` method to
`find_event_with_relations` for the sake of clarity, also to match the
other method names, and to match the Rust standard library namings.
The swift bindings aren't getting tested (they don't run) in CI anymore,
so building with the reldbg profile (that's a workaround to make it run
and not crash in production) doesn't provide more value than building in
debug mode, while taking much longer to build.
Let's use the default dev profile for this; we have to specify it
manually, because the default for the xtask command is to use the
`reldbg` profile otherwise.
This requires a fix for the dev profile, that consists in being able to
set the iOS deployment target, and set it to a high value in CI
settings. Production builds *don't* have to set it, though.
These two methods could be only made private now, since they're only
used in `finish_back_pagination` (and in tests). But they only call
inside other methods of the underlying linked chunk without any extra
value, so they can be inlined instead. This reduces the public API and
removes tests for other APIs that were tested some place else.
This patch fixes `EventCache::for_room` to return an error when the room
isn't found. Additionally, it also returns an error when the `Client`
(actually the `WeakClient`) is dropped.
These errors were “hidden” behind the room version logic because of
the fallback. The code has been rewritten in a way it makes it harder to
do it again.
This patch is the first part of the new `LatestEvents` API. It contains
the “framework”, i.e. the structure, tasks, logic to make it work, but
no latest events are computed for the moment.
The Latest Events API provides a lazy, reactive and efficient way to
compute the latest event for a room or a thread.
The latest event represents the last displayable and relevant event
a room or a thread has been received. It is usually displayed in a
_summary_, e.g. below the room title in a room list.
The entry point is `LatestEvents`. It is preferable to get a reference
to it from `Client::latest_events`, which already plugs everything to
build it. `LatestEvents` is using the `EventCache` and the `SendQueue`
to respectively get known remote events (i.e. synced from the server),
or local events (i.e. ones being sent).
\## Laziness
`LatestEvents` is lazy, it means that, despites `LatestEvents`
is listening to all `EventCache` or `SendQueue` updates, it will
only do something if one is expected to get the latest event for a
particular room or a particular thread. Concretely, it means that until
`LatestEvents::listen_to_room` is called for a particular room, no
latest event will ever be computed for that room (and similarly with
`LatestEvents::listen_to_thread`).
If one is no longer interested to get the latest event for a
particular room or thread, the `LatestEvents::forget_room` and
`LatestEvents::forget_thread` methods must be used.
\## Reactive
`LatestEvents` is designed to be reactive. Use
`LatestEvents::listen_and_subscribe_to_room` (same for thread) to get
a `Subscriber`.
This patch moves the `JoinHandleExt` trait and the
`AbortOnDrop` type from `matrix_sdk::sliding_sync::utils` into
`matrix_sdk_common::executor`. This is going to be useful for other
modules.
This patch updates `test_sync_indicator` since the `SyncIndicator`
is shown for the `SettingUp` state. Also, this patch makes this
test non-flaky entirely by changing the `sync::mpsc::channel` to a
`sync::Barrier`.
Since `RoomListService` uses a persistent `pos` for sliding sync, the
`SyncIndicator` no longer shows its face except if the sliding sync
session doesn't exist or has expired.
This patch changes that by extending the `Show` from `Init` to
`SettingUp`.
When we upgrade Ruma, the `EventContent` bound on `EventBuilder` will be
changed to `StaticEventContent`, which is not implemented by the
`Any*EventContent` enums.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
One must now specify the target linked chunk; both callers would filter
out items based on this after the fact, which is a bit overkill since
most items would thus be filtered out on the sinking end.
The one hardship is that lazy-loading updates must NOT affect the order
tracker, otherwise its internal state will be incorrect (disynchronized
from the store) and thus return incorrect values upon shrink/lazy-load.
In this specific case, some updates must be ignored, the same way we do
it for the store using `let _ = store_updates().take()` in a few places.
The author considered that a right place where to flush the pending
updates was at the same time we flushed the updates-as-vector-diffs,
since they would be observable at the same time.
This is a new data structure that will help figuring out a local,
absolute ordering for events in the current linked chunk. It's designed
to work even if the linked chunk is being lazily loaded, and it provides
a few high-level primitives that make it possible to work nicely with
the event cache.
In the next patch, we're going to introduce another user of
`UpdatesToVectorDiff` which doesn't require accumulating the
`VectorDiff` updates; so as to make it optional, let's generalize the
algorithm with a trait, that carries the same semantics.
No changes in functionality.
It doesn't belong there anymore, because it's not used within the event
cache itself. It's used in the timeline, and I think it's nice to keep
it available for external users as well, so it now lives in its own
`matrix-sdk/src/paginators` module. Next step is to move the thread
loader in there as well.
It avoids using different versions in several places for consistency. It
also allows to be able to change it in a single place when needed.
This also bumps the fallback to v11 everywhere, since it is the default
version for new rooms since Matrix 1.14 and it has the sanest redaction
rules.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Process:
- set style_edition to 2024 in .rustfmt.toml
- run `cargo fmt`
- undo .rustfmt.toml change
- run `cargo fmt` again
- manually rewrap some strings
The `/auth_metadata` endpoint has been supported by Synapse for 6 months
now so there shouldn't be any homeserver that still depend exclusively
on it. This endpoint is also part of Matrix 1.15.
Support for this endpoint has been removed from Ruma so this is
necessary before an upgrade of the dependency.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
It turns out that downstream clients can and do call
`Client::join_room_by_id()` rather than `Room::join`, so we need to do
the room key history import in the lower-level method.
---------
Signed-off-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
The previous code would parse the entire bundled event, only to look at
its type indirectly. We can do better than this, and only look at the
type instead. This brings a few benefits:
- it is faster, as we don't have to deserialize the entire event
- while the spec seems to indicate that the latest thread event has a
`room_id`, it seems that, under some circumstances, it does not, as
indicated by some rageshakes. As such, not parsing as `AnyMessageLike`
(which mandates a room id) makes it more robust to the absence of a
room id in there, marking more events as latest events.
The latest mobile designs for Element Call have it displaying the room name in an "app bar". So the Element Call widget will soon start requesting the capability to learn the room name, and the SDK should auto-approve this capability.
This prevents us from having to add type constraints on the dynamic type
and instead only having to specify the type constraint at the call site.
Signed-off-by: Michael Goldenberg <m@mgoldenberg.net>
This can be used to invalidate the persisted state on disk related to
the sliding sync positions. It's useful to do so when clearing up all
the caches, since the sliding sync `pos`itions are stored in the crypto
store (to benefit from the cross-process lock).
The join rule state event can be missing from a room state. In this
case, it's an API footgun to return a default value; instead, we should
return none and let the caller decide what to do with missing
information.
`RoomEventCache::subscribe` is nice to subscribe to every update
happening inside a room in the event cache. However, the returned
`RoomEventCacheSubscriber` has side-effects when dropped (see
auto-shrink to save memory space). In some situation, this is pretty
annoying. for example, if one wants to listen to multiple room updates,
all the room event cache subscribers must be kept in memory, thus
breaking the side-effects. This isn't always the desired output. In
addition, listening to multiple channels/subscribers at the same is
quite complex, as it implies non-trivial async runtime efforts or
complex future types.
To solve this problem, this patch introduces a new
`EventCache::subscribe_to_room_generic_updates` method, which returns a
single `Receiver<RoomEventCacheGenericUpdate>`.
First off, it hides the details of `RoomEventCacheUpdate` (returned by
`RoomEventCacheSubscriber`), which might be desired, but particularly
lighter because events aren't part of the payload.
Second, one no longer needs to subscribe to all rooms. Only one channel
can be listened to get updates for all rooms. It reduces the complexity
on the caller side, plus `Receiver<RoomEventCacheGenericUpdate>` doesn't
have any side-effect.
This patch tests this feature in 4 situations:
1. when a room is created/loaded empty,
2. when a room is loaded and is not empty because data exists in the storage,
3. when a room receives data from the sync,
4. when a room receives data from the pagination.
<!-- description of the changes in this PR -->
Features to configure UserAgent, Proxy, Disabling SSL, and additional
certificates are not available on Wasm platforms. We remove these
configuration options from the FFI layer, while preserving them on
non-Wasm platforms.
- [ ] Public API changes documented in changelogs (optional)
<!-- Sign-off, if not part of the commits -->
<!-- See CONTRIBUTING.md if you don't know what this is -->
Signed-off-by: Daniel Salinas
---------
Co-authored-by: Daniel Salinas <danielsalinas@Daniels-MacBook-Pro-2.local>
Previously, we held a TrustRequirement inside Client, and had to wrap it
in a DecryptionSettings manually a couple of times.
It feels more right to hold on to a DecryptionSettings directly. If
there were ever some additional settings, most likely we would need them
all here.
If I haven't screwed it up, this should not affect behaviour in any way.
The pagination status was only mapped onto the "global" state, that is, the state of the event cache pagination.
Now, consider the following case, where two timeline instances are live:
- the first one could have backpaginated all the items back to the start of the room
- the second one is created later. Because of the initial value for the skip count, it will only return a subset of event items (~20).
However, listening to the pagination status for the second timeline would incorrectly state that the timeline was entirely paginated (because it returned the "global" pagination status). As such, an observer might think that there are no more items in the timeline, while a subsequent pagination would adjust the skip count and return more items.
This fixes it by combining the global pagination state with the local timeline state (aka the skip count value). If the skip count is positive (meaning, we could set it to 0 later and thus returning more events), we pretend we haven't reached the start of the timeline. This way, an observer can call pagination later, which may adjust the skip count and "return" more items.
`RoomPowerLevels` already holds an inner version of the power levels and has access to everything needed to compute the user to level map.
This way the async fetching only happens once and the mapping only on request.
Currently, clients have to make async requests to `Room::get_power_levels` from multiple places throughout the app in order to correctly configure the various UI components and again, on randomly decided events, to keep them up to date.
This patch starts publishing the power levels directly on the `RoomInfo` and allows them to be handled (and updated!) through the normal `subscribe_to_room_info_updates` mechanism.
This patch removes a name ambiguity around _listener_ vs. _subscriber_.
Both terms are used to talk about `RoomEventCacheListener`. We usually
use the term _subscriber_ for the type being returned by a `subscribe`
method. The code refers to this sometimes as listener, sometimes
as subscriber, sometimes both in the same sentence, which can be very
confusing! This patch solves this by using the _subscriber_ term only.
They are only semantically different if a macro of the same name exists,
in which case the foo::{self} import only imports foo-the-module, not
foo-the-macro.
We have a working implementation of the additional forms of QR Code login
of MSC4108. This commit moves the existing code into its own file
to make future updates easier to follow.
Remove the MediaFileHandle concept from the matrix-sdk-ffi crate on Wasm
platforms. File handles are not supported in the browser.
Signed-off-by: Daniel Salinas
The uniffi tool for generating JS/Wasm bindings utilizes rust as its
intermediate language.
As a result, the 'target' uniffi annotated library needs to be marked as
a 'lib' so that the generated rust code can utilize it to generate the
Wasm create + typescript bindings.
Signed-off-by: Daniel Salinas
This patch expands on the already existent `RoomPowerLevels` record (which it renames to `RoomPowerLevelsValues` to work around uniffi not exposing public fields) and nests them inside a new exported object that also provides methods for retrieving the actions that an user can take, methods moved from the room object.
This reduces the amount of async calls the clients need to make, simplifies the API and groups the code better together.
The system of platform targets was already quite messy, and becoming
even worse as we start preparing for Wasm support. Switch to features
instead to make this easier to work with.
UInt::MAX.try_into().unwrap() was causing errors on Wasm platforms, due
to the result being unrepresentable.
This `unwrap_or` was also always being calculated regardless, so I think
using `usize::MAX` is preferable on all platforms.
Signed-off-by: Daniel Salinas <zzorba@users.noreply.github.com>
Some of these Box<dyn Error + Send + Sync> are okay, but a few are
problematic.
Confirmed this still compiles fine on the fully working wasm tree.
Signed-off-by: Daniel Salinas
The main thing left now are the store traits, unsure how to deal with
those. `dynosaur` + `trait_variant` are kind of the modern replacement
for `async_trait`, but (a) `trait_variant` seemed to generate invalid
code here when I tried it and (b) even with that fixed I think the error
type erasure is going to present some extra problems. Maybe it's fine to
just keep the current solution for the store traits for now.
Signed-off-by: Jonas Platte <jplatte+matrix@posteo.de>
This patch switches from tarpaulin to llvm-cov for our coverage reports.
llvm-cov can use cargo-nextest to run the tests which means that we can
tolerate flaky tests for coverage just like do for the rest of our CI
run.
We can also start using JUnit reports to track flaky tests.
Proper support for receiving to-device messages for widgets.
If the widget is in an e2ee room, clear to-device traffic will be excluded. Also filter out internal to-device messages that widgets should not be aware off.
On this date (heh), the date divider reports include the following
information:
- initial timeline items in a shortened format,
- operations to apply,
- final timeline items in the same shortened format,
- errors
The shortened format includes:
- either, for events: "[event id]: server timestamp"
- or for date divider: "--- date divider timestamp"
As such, they don't include any personal information.
The initial timeline items state and set of operations to apply
constitutes a fully enclosed test case, so it's nice to report it to
Sentry, so we can reuse it almost as is (we'd only need to randomize the
event IDs) and fix it in a subsequent commit.
This patch simplifies the selected read receipt rendering by the fact
that we can simply fetch the selected timeline item instead of the event
ID and then do a search for the selected item.
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
Signed-off-by: Damir Jelić <poljar@termina.org.uk>
This patch changes the strategy of `check_room_upgrades`.
Instead of checking all state events in `StateChanges` and
throwing an error in case of an invalid state, this patch updates
`state_events::sync::dispatch` to filter out invalid state events
(specifically `m.room.create` and `m.room.tombstone`), and log the
error. That way, the sync doesn't stop and the app can continue working
smoothly.
So `check_room_upgrades` is splited into two functions:
`is_create_event_valid` and `is_tombstone_event_valid`. It's no longer
necessary to detect mergers or splitters because those are _soft
errors_, however loops are still critical errors (hence the fact
the state events aren't stored nor saved nor applied, and that a log
is emitted).
Note: `check_room_upgrades` has been reverted in a previous commit,
that's why it doesn't appear in this diff.
This patch adds optional thread root event id parameters to the drafting functions exposed on the room level
allowing unfinished messages to be managed for the main room as well as any inner thread.
Internally it uses the room id or a tuple of the room id and the thread as keys for the various backing stores.
It doesn't make sense to do it after a back-pagination, since a
back-pagination does require a previous-batch token in the first place,
meaning that if we did paginate, then we did wait for a previous-batch
token beforehand.
Having decrypted an event with a given megolm session, we need to check that
the owner of that session actually matches the sender of an event, otherwise
there is a danger of the sender being spoofed to make it look like it was sent
by another user.
Security-Impact: High
CVE: CVE-2025-48937
GitHub-Advisory: GHSA-x958-rvg6-956w
Internal Sentry reports tell us that enabling the busy_timeout seems to
have *increased* the number of "database is busy" errors, instead of
lowering those. As a result, we're going to disable the pragmas in all
the places where we enabled it before, and observe how the number of
"database is busy" errors evolves.
This correctly handles the reply fallback behavior:
- the behavior isn't changed for a live timeline,
- when a timeline is thread-focused, we will extract the `replied_to`
field if and only if the thread relation is *not* marked as behaving in
a fallback manner.
This makes it possible to distinguish actual in-thread replies.
In a "soon" future, threads have their own linked chunk. All our code
has been written with the fact that a linked chunk belong to *a room* in
mind, so it needs some biggish update. Fortunately, most of the changes
are mechanical, so they should be rather easy to review.
Part of #4869, namely #5122.
This patch renames `check_tombstone` to `check_room_upgrades`.
Then it rewrites it **entirely** to remove all false-positives and
false-negatives. More importantly, the room versions are no longer
involved: they can't be compared or ordered, they must be treated as
opaque values.
This new version of `check_room_upgrades` does a first path to check
predecessor-successor consistency. Then it does a second version to
detect loops. This new algorithm is robust to absent `m.room.create`
events. Making them mandatory is left to another patch.
More tests are added, especially to ensure that `m.room.create` cannot
be overwritten, and to ensure loops or inconsistent predecessors and
successors are correctly detected.
And add getters and setters. It makes it clear who are the external
readers/writers of the push actions, and it makes it impossible to
create a `TimelineEvent` out of thin air (since it now has a private
field).
There's been many segfaults happening in tests, while running the test
coverage for this specific crate. An issue has been opened on
cargo-tarpaulin's repository:
https://github.com/xd009642/tarpaulin/issues/1749
Until this is fixed or worked around, we'll disable coverage testing for
this specific crate.
This module only builds on non-wasm with the patched async-compat from
the workspace Cargo.toml's patch section, and it is only used by the ffi
crate. It is currently breaking the use of the SDK as a git dependency,
and would prevent the publishing of matrix-sdk-common (unless using
--no-verify, but then that would just break all users of the newly
published crates.io version).
This bug was introduced in
https://github.com/matrix-org/matrix-rust-sdk/pull/5089.
Signed-off-by: Jonas Platte <jplatte+matrix@posteo.de>
This helps in the case we want to observe the membership state changes - or some other info - for a room that's still not known so we can't just use `Client::get_room` to fetch it.
Addendum to https://github.com/matrix-org/matrix-rust-sdk/pull/5125 to
allow sending galleries from the FFI crate. This is the final PR for
galleries (apart from possible enhancements to report the upload status
in https://github.com/matrix-org/matrix-rust-sdk/pull/5008).
This is somewhat lengthy again, apologies. Most of the changes are just
wrappers and type mapping, however. So I was hoping that it's relatively
straightforward to review in bulk. Happy to try and elaborate on the
changes or break them up into smaller commits if that helps, however.
---------
Signed-off-by: Johannes Marbach <n0-0ne+github@mailbox.org>
This commit systematically replaces Send+Sync trait bounds throughout
the matrix-rust-sdk codebase to enable Wasm compatibility while
maintaining thread safety on native targets.
Key changes:
- Use SendOutsideWasm/SyncOutsideWasm traits instead of Send/Sync for
trait bounds
- Apply conditional compilation for error types and trait objects
- Update FFI trait definitions to use Wasm-compatible bounds
- Fix event handler and type alias definitions for cross-platform
compatibility
- Maintain existing functionality while enabling WebAssembly target
support
The SendOutsideWasm/SyncOutsideWasm traits are empty on Wasm (allowing
all types) and alias to Send/Sync on native targets, ensuring zero-cost
abstraction.
Files updated:
- All FFI bindings (30+ trait definitions)
- Core SDK error types and type aliases
- Event handler infrastructure
- Store and crypto abstractions
- UI service filters and sorters
- Timeline and authentication modules
<!-- description of the changes in this PR -->
- [ ] Public API changes documented in changelogs (optional)
<!-- Sign-off, if not part of the commits -->
<!-- See CONTRIBUTING.md if you don't know what this is -->
Signed-off-by: Daniel Salinas
---------
Signed-off-by: Daniel Salinas <zzorba@users.noreply.github.com>
Co-authored-by: Daniel Salinas <danielsalinas@Daniels-MacBook-Pro-2.local>
Co-authored-by: Daniel Salinas <danielsalinas@daniels-mbp-2.myfiosgateway.com>
The `ProcessedToDeviceEvent::Decrypted` variant now also have an
`EncryptionInfo` field.
The enum variant changed from `Decrypted(Raw<AnyToDeviceEvent>)` to `Decrypted {
raw: Raw<AnyToDeviceEvent>, encryption_info: EncryptionInfo) }`
Mechanical move from tokio::spawn to matrix_sdk_common::executor::spawn
that has support for Wasm platforms. On non-Wasm, this shim defaults to
tokio::spawn.
The actual code useful in `with_events_mut` and used in that function
was to propagate the changes to the store and propagating diffs to
observers. It often striked me as hacky to use this method to do that,
so instead I'm proposing here to inline the useful bits. That way,
`with_events_mut` is now clearly called only in two cases: after a sync,
or after a successful network back-pagination.
This patch adds the `check_tombstone` response processor to
detect invalid state with room upgrades. One can check the new
`InconsistentTombstonedRooms` error to learn more about the detected
patterns.
If the vec of actions is defined (Some) but empty, then it means we
could determine there are no push actions. It is a slight optimization
to not serialize them, in this case; otherwise, this can result in many
serialized empty action vector, in the event cache's persistent storage.
When the user does an `/invite`, if the target user doesn't start with
an `@`, try expanding it as a user on the local server.
This makes it much easier when repeatedly testing inviting!
Rather than always writing the logs to `/tmp`, write them to the session
directory. The session directory defaults to `/tmp` so by default this
will do the same as before, but if you override the session path on the
commandline, the logs will get stored alongside the stores and caches.
This is particularly useful when running two instances of multiverse,
and you want them to put their logs in different places.
The .boxed() method requires a Send trait that makes it
unusable on Wasm platforms. This re-exports the existing StreamExt,
while providing a new extension for the wasm family of targets.
This is an implementation of an update to MSC2762 (https://github.com/matrix-org/matrix-spec-proposals/pull/4237). It changes the widget driver to split state updates out into an `update_state` action and use the `send_event` action exclusively for forwarding timeline events. Accordingly, `read_events` now always reads from /messages, never the state store.
It's >100 bytes large and often optional, so it makes sense to put it
on the heap to reduce the size of structs with such optional fields,
and the stack size of functions with such optional parameters.
It's also cloned in a couple places in the UI crate, so it probably
makes sense to just always refcount it.
This started as a clippy suggestion to box PendingEdit inside
AggregationKind::Edit.
Part of #3716
I did notice that the `sender` and `member` methods have some overlap in
what values get set for the "sender" and "state key", and I tried to
make sure that in my changes to use `EventFactory` the original event
configuration is being replicated, so please do double-check me on that
note in particular.
Signed-off-by: Yousef Moazzam <yousefmoazzam@hotmail.co.uk>
The test was ignored since the functionality was only implemented for
the memory and SQLite store. This caused a bug in the SQLite
implementation to go unnoticed.
Let's just disable it for WASM since this is the only place where we
didn't yet implement the necessary methods.
The status code is usually returned by Cloudflare to indicate an unknown server error, so we should cancel the upload and let the user retry if they want to.
This patch removes `RoomTombstoneInfo` and replaces it by
`SuccessorRoom`. This patch renames `RoomInfo::tombstone` to
`RoomInfo::success_room`.
This patch also implements `Room::successor_room()` and
`Room::predecessor_room()`, and adds documentation.
This allows applications to decide what they'd like to do if there isn't a value present. It allows the application to decide what the default should be.
This patch adds the new `deduplicate_versions`. This new filter will
filter out room versions that are outdated. Only the “active” versions
are kept.
A room version is considered active if and only if:
* the room is joined and has no successor,
* the room is joined and has a successor room that is invited or knocked,
* the room is left, invited, banned or knocked.
All other rooms are filtered out.
This patch extracts a bit of code from `dispatch` in a new response
processor: `dispatch_room_member`. The idea is to clarify the content of
`dispatch` itself.
This patch moves the insertion of `new_user_ids` in
`updated_members_in_room` after the use of `new_user_ids` by
`update_or_set_if_room_is_newly_encrypted`. By moving it after, it saves
the need to clone it entirely, thus saving memory allocations.
This patch renames `dispatch_and_get_new_users` to `dispatch`. This
response processor no longer returns the new user IDs, but they are
written in a mutable reference argument. This argument is
typed by a new private trait: `NewUsers`. This trait is implemented on
`BTreeSet<OwnedUserId>` and on `()`. It helps to avoid allocations when
the new users are not needed.
This reproduces the issue originally described, where it wasn't possible
to run a /members query in an event handler, because the sync lock was
taken at this point.
This is a dangerous operation, and requires that the sync service must
be stopped while we're touching this. Since this is an internal method,
that shouldn't be used in most clients anyways (as it usually papers
over actual issues happening elsewhere, on the server for instance), I
kept it simple with a scary doc comment explaining the preconditions,
but we could assert them at runtime, with a little bit more effort.
It's just a room, now! Since there were some users of the
`latest_event()` that returned an `EventTimelineItem`, I kept this
method as a method in the Room trait extension that's in the UI crate,
so it's still convenient to use.
This patch updates `Room::leave` to not early return when the server
returns a 403 error code on `/leave`. Indeed, if the user doesn't have
the permissions to leave a room, it's impossible for they to leave it.
Let's consider it's fine to ignore this particular error and continue
the process of leaving the room.
This patch adds the `predecessor` and `no_predecessor` methods on
`EventBuilder<RoomCreateEventContent>`. This is helpful to configure the
`predecessor` field.
This patch fixes `EventFactory::create` where the `m.room.create` wasn't
created as a state-event (the `state_key` field was missing).
Also, it uses the `creator_user_id` in the `sender` field if no sender
was given.
First off, this patch renames `Room::tombstone` to
`Room::tombstone_content` (to be consistent with other methods, such as
`Room::create_content`).
Second, this patch adds the `Room::successor_room` and
`Room::predecessor_room` methods, along with the `SuccessorRoom` and
`PredecessorRoom` types. This naming more or less comes from the Matrix
specification:
- the term _predecessor_ is part of the specification,
- the term _successor_ isn't present _per se_, the words _replacement
room_ are used instead, but I prefer _successor_ as it brings a nice
symmetry with _predecessor_.
The widget Driver should be able to send and receive to-device events.
This is useful for element call encryption keys.
This PR focusses on the widget driver and machine logic. To
send/communicate the events from the widget to the driver.
It skips any encryption logic. Some of the encryption logic will be part
of crypto crate and the code in the widget driver crate should be kept
minimal once the crypto crate is ready.
---------
Co-authored-by: Valere <bill.carson@valrsoft.com>
This behaviour was added only at the `RoomPreview::leave` method, but since we're slowly moving away from it we should move the forget action to the `Room::leave` method instead
On Android, we should use rustls instead of native-tls; this requires
unsetting the default features of the `sentry` crate, and specifying
them by hand instead.
For consistency, I've done the same for the non-android sentry
dependency.
This patch moves the `Room::get_member_hints` method inside the newly
created `display_name` module. That way it is isolated from the rest of
the codebase.
This patch moves the `Room::has_active_room_call` and
`Room::active_room_call_participants` methods into the new `call`
module. This patch also moves the associated tests in this new module.
This patch moves the `Room::*knock*` methods into the new `knock`
module.
The idea is to group API by “theme” to get smaller modules and more
organised code.
This patch moves the `Room::state` method, along with the ``RoomState`
and `RoomStateFilter` types into the new `state` module. This patch also
moves the tests in this new module.
The idea is to group API by “theme” to get smaller modules and more
organised code.
This patch merges the `rooms/normal.rs` file into `rooms/mod.rs`.
The name `normal` is present for historical reasons that no longer stand
today. This is no needed anymore. Let's simplify the modules.
This patch moves everything related to the `Room` latest event API
inside the new `latest_event` module. This patch also moves the tests.
The idea is to get a smaller `rooms::normal` module, and to clarify the
code by grouping `Room` APIs by “theme”.
This patch moves the Room::is_favourite` and `::is_low_priority`, along
with the `RoomNotableTags` into the new `tags` module. This patch also
moves the tests in this new module.
The idea is to group API by “theme” to get smaller modules and more
organised code.
This patch moves the `Room::encryption_state()` and
`Room::encryption_settings()` methods, and the sibling types, into a new
`encryption` module. This patch also moves the tests in this new module.
The idea is to group `Room` API by “theme” in modules, to clarify the
code.
This patch moves the `Room::display_name()` method, and all the
siblings, into a new `display_name` module. It includes types like
`RoomDisplayName`, `UpdatedRoomDisplayName`, `RoomSummary` etc. This
patch also moves the tests in this new module. The idea is to group
`Room` API by “theme” in modules, to clarify the code and to make the
big `rooms::normal` module smaller.
This patch moves the `Room::subscribe_info`, `::clone_info` and
`::set_room_info` methods from the `rooms::normal` module to the
`rooms::room_info` module. This patch also moves the tests related to
room info inside `room_info`.
This patch explains why having `m.room.create` is necessary. It's not
only about the room previews, but also about the room versions, and thus
the tombstoned rooms.
This patch renames the `RoomInfo::version` field to
`data_format_version` to avoid all possible confusion with the
`room_version` (from `m.room.create`).
If the replied-to event is an aggregation, the
`RepliedToEvent::try_from_timeline_event` will now return `Ok(None)`,
and the caller may handle this as they please.
In the FFI layer, this will be filled with an error message indicating
that the event is unsupported.
As per the plan defined in #4718:
```
the room_list_service::room::RoomInner shouldn't make use of its inner timeline;
it's only used in a direct getter, or to compute the latest room event, but it's not working
as intended, since local echoes aren't properly displayed in the room list.
This non-working feature can be removed, in favor of #4112
```
Updates the unread flag or the room in `Timeline::send_single_receipt()`
and `Timeline::send_multiple_receipts()` if the room is marked as unread
and the receipts are unthreaded.
Updates it also in `Timeline::mark_as_read()`, even if there is no
latest event ID.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Updates the unread flag or the room in `Room::send_single_receipt()` and
`Room::send_multiple_receipts()` if the room is marked as unread and the
receipts are unthreaded.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This makes it possible to handle reactions/redactions/edits/etc. on
non-live timelines. As a result, the pinned and focused timelines will
now get live reactions/redactions and so on. This makes it possible to
also have the thread timelines handle those live events, although it's
unclear how it will pane out in the end, when the event cache is also
involved.
This can be done by splitting the handling of the msgtype/mentions from
the handling of the `relates_to` field, requiring a few API changes here
and there.
We only need the edit_json if we're about to save the edit aggregation.
Likewise, if there's no current event id (i.e. the event being handled
is a local echo), then we don't need to even try to extract anything
from the bundle information.
The poll events's `related_to` field is a `RelationWithoutReplacement`,
while the two others are `Relation<C>`, where `C` is the event content
type (in case it was replacement). As a matter of fact, we try
converting the `Relation<C>` into a `RelationWithoutReplacement` (which
unfortunately requires cloning, which is wasteful if the relation was a
replacement indeed), and then we can use a single function to extract
the reply information and thread root info, for all three.
There are two reasons for this.
Firstly. we've already done a bunch of work to map `SenderData` into a
`VerificationState`, and the decision tree from `VerificationState` to
allow/reject is simpler than going from `SenderData`, even if we have to
fudge it a bit to get the "legacy" flag. (Note that it allows us to get
rid of an `unreachable!` panic.)
Secondly, `VerficationState` represents the state of an *event*, whereas
`SenderData` is about the session as a whole. A session can be fine,
whilst events (claiming to be) encrypted with it can be suspect. What we
want here is to check a specific message. Currently, this doesn't make
any functional difference, but conceptually it's cleaner to check the
`VerificationState`.
Note that there are a bunch of tests for this method in
`matrix-sdk-crypto/src/machine/tests/decryption_verification_state.rs`,
called `test_decryption_trust_requirement`.
This patch removes the `SlidingSync::rooms` field. A cascade of removal
happens, and many part of the code is simplified. The most notable is
`FrozenSlidingSync`.
This patch
- Updates Ruma to use the improved MediaPreviewConfig event type that
also supports a `Default` for the content type
- Implemented a way to observe the stable and unstable values of the
event and return the used one accordingly, if no one is present the
default will be used
- Set the value (will only use unstable type for now)
This patch removes the `RoomInfo::prev_room_state` field, along with the
`RoomInfo::prev_state` method.
This data was introduced during the knocking project but was never used,
and is not used nowadays. Let's remove it.
This is used to configure EC on devices that need to control media outputs on their own (android, ios).
If set, EC will display a list of devices provided by the app.
This patch adds the `m.room.tombstone` state event to the list of
events in `required_state` used by the `RoomListService`. The goal is to
offer the possibility for the consumers to know whether a room has been
tombstoned or not.
The `read_receipts::compute_unread_counts` function needs the _previous
events_ to compute the read receipt correctly. These previous events
were store in `SlidingSyncRoom::timeline_queue`. Since the removal of
`timeline_queue` in the previous patches, this patch uses the Event
Cache to fetch them. It only uses events that are loaded in memory.
This is as correct as the prior behaviour, even this is still incorrect
since it doesn't back-paginate to get a better view. This is for
later. The goal of this patch is to restore the same behaviour, without
`timeline_queue`.
The main problem is that read receipts are computed in
`matrix-sdk-base`, and that the Event Cache lives in `matrix-sdk`. Thus,
we change the `SlidingSyncResponseProcessor` to handle read receipt
in particular.
The
`matrix_sdk_base::response_processors::rooms::msc4186::extensions::dispa
tch_ephemeral_events` function has been split in
two methods `dispatch_typing_ephemeral_events`, and
`dispatch_receipt_ephemeral_event_for_room`. The workflow has been a
little bit redesigned to fit in the new `SlidingSyncResponseProcessor`
constraints.
This patch moves one test from `matrix-sdk-base` into `matrix-sdk`,
because to compute the read receipt, the Event Cache must be
enabled/listening to sync updates.
`RoomEventCache::subscribe` returns the set of events + the
`RoomEventCacheListener`. However, creating this listener isn't
cheap, especially dropping it. That's why this patch creates
`RoomEventCache::events` to replace `subscribe` when the listener is
not necessary.
This patch reduces the sizes of `Verification` from 376 bytes to
16 bytes, and `VerificationRequestState` from 424 bytes to 96 bytes.
It also reduces the size of a couple of other types in the same vain.
They have never been set and there was no way of telling if stickers and polls belong on a thread or come in reply of any other message.
This patch also exposes methods for setting these relations on the event factory level.
This moves the logic from `Room::join()` to these two methods. This is
isofunctional, because `Room::join()` does call into
`Client::join_room_by_id()` internally.
The then() function can be used with booleans and futures to execute a
piece of code if the boolean is true or once the future is completed.
In contrast, the then() function in the widget driver is not executed
immediately. Instead it only adds a callback that is later on executed
by the widget driver.
… that allows building a timeline instance specific to a particular
thread root.
Creating a timeline in this mode will start by backpaginating root event
relations with `num_events` and automatically insert the thread root
event when reaching the end. It will include
`RelationsOfType(RelationType::Thread)` but also recurse over the
retrieved events to fetch reactions.
It will not however react to new events received over sync or that the
user sends (for now).
This patch will also help incrementally deliver the upstream client
support for creating such a timeline.
Part of #4833 (meta #4869).
Under some very particular circumstances, the "database is locked" error
can still happen, even in WAL mode, even if the database connection is
not being upgraded from a read transaction to a write transaction.
We *think* this might be the reason behind errors like
github.com/element-hq/element-x-ios/issues/3582, so we're enabling the
sqlite busy_timeout, which will retry the operation after a short sleep,
until the busy timeout is being hit.
This variant will be used to cause side-effects to existing timeline
items, because the event they relate to is for an aggregation
(edit/reaction/etc.).
This is used here for reactions.
This patch updates `EventHandler` to use the correct regions where
appropriate, thus reducing the complexity of the code, and removing
classes of bugs.
In the case of `Flow::Remote { position: TimelineItemPosition::At { …
}}`, we no longer need to skip the local timeline items, and to handle
the presence of the `TimelineStart` timeline item. The code is less
complex.
In the case of `Flow::Remote { position: TimelineItemPosition::End { …
}}`, that's exactly the same at the previous case.
In the case of `recycle_local_or_create_item`, the `try_fold` approach
is replaced entirely with a simple `iter_locals_region`, reducing the
size of the comments explaining the code, reducing the complexity of the
code, and reducing the surface of bugs.
This patch updates `TimelineMetadata` to work on the _remotes_ region
only, excluding the _start_ and the _locals_ regions. It helps to reduce
the risk of inserting items in an incorrect regions.
This patch also removes on more `rfind_event_by_id` usage, which is
nice.
This patch updates `ReadReceiptTimelineUpdate` to work on the _remotes_
region only, excluding the _start_ and the _lcoals_ regions. It helps
to reduce the risk of inserting a `ReadMarker` inside the _start_ or the
_locals_ regions.
This patch updates `DateDividerAdjuster` to work on _remotes_ and
_locals_ regions only, excluding the _start_ region. It helps to reduce
the risk of inserting a `DateDivider` inside the _start_ region.
This patch also uses the new `push_date_divider` method, which provides
a couple of invariants.
This patch defines a new concept in the `Timeline`: Regions.
The `ObservableItems` holds all the invariants about the _position_ of the
items. It defines three regions where items can live:
1. the _start_ region, which can only contain a single `TimelineStart`,
2. the _remotes_ region, which can only contain many `Remote` timeline
items with their decorations (only `DateDivider`s and `ReadMarker`s),
3. the _locals_ region, which can only contain many `Local` timeline items
with their decorations (only `DateDivider`s).
The `iter_all_regions` method allows to iterate over all regions.
`iter_remotes_region` will restrict the iterator over the _remotes_
region, and so on. These iterators provide the absolute indices of the
items, so that it's harder to make mistakes when manipulating the indices of
items with operations like `insert`, `remove`, `replace` etc.
Other methods like `push_local` or `push_date_divider` insert the items
in the correct region, and check a couple of invariants.
This patch implements the `has_local` method on
`ObservableItemsTransaction`, which is way faster than the previous the
previous solution which was to iterate over all items to find at least
one local timeline item.
This patch adds the `push_local` method on `ObservableItemsTransaction`
to add semantics and hardcode the invariant in a single place for the
different timeline items.
This patch adds the `push_timeline_start_if_missing` method on
`ObservableItemsTransaction` to add semantics and hardcode the
invariant in a single place for the different timeline items.
This patch adds the `--generate-link-to-definition`
argument to `rustdoc` for `docs.rs`. This is using
https://github.com/rust-lang/rust/pull/84176 to add links in the source
code page.
If we already have cross-signing details for the owner of the device at the
point we receive the to-device message, we should store that rather than just
the device info.
create a new method `SenderData::from_device` which does the last few steps of
`SenderDataFinder`: turns out we want it elsewhere. Add some tests to test that
functionality in isolation.
- Doc comment for the SQLite-based state store incorrectly referred to
it as a "cryptostore".
- Consistent capitalisation of SQLite.
- Consistent use of indefinite article "an" before SQLite.
- Fix line length.
Imagine we have the following events:
| event_id | room_id | chunk_id | position |
|----------|---------|----------|----------|
| $ev0 | !r0 | 42 | 0 |
| $ev1 | !r0 | 42 | 1 |
| $ev2 | !r0 | 42 | 2 |
| $ev3 | !r0 | 42 | 3 |
| $ev4 | !r0 | 42 | 4 |
`$ev2` has been removed, then we end up in this state:
| event_id | room_id | chunk_id | position |
|----------|---------|----------|----------|
| $ev0 | !r0 | 42 | 0 |
| $ev1 | !r0 | 42 | 1 |
| | | | | <- no more `$ev2`
| $ev3 | !r0 | 42 | 3 |
| $ev4 | !r0 | 42 | 4 |
We need to shift the `position` of `$ev3` and `$ev4` to `position - 1`,
like so:
| event_id | room_id | chunk_id | position |
|----------|---------|----------|----------|
| $ev0 | !r0 | 42 | 0 |
| $ev1 | !r0 | 42 | 1 |
| $ev3 | !r0 | 42 | 2 |
| $ev4 | !r0 | 42 | 3 |
Usually, it boils down to run the following query:
```sql
UPDATE event_chunks
SET position = position - 1
WHERE position > 2 AND …
```
Okay. But `UPDATE` runs on rows in no particular order. It means that
it can update `$ev4` before `$ev3` for example. What happens in this
particular case? The `position` of `$ev4` becomes `3`, however `$ev3`
already has `position = 3`. Because there is a `UNIQUE` constraint
on `(room_id, chunk_id, position)`, it will result in a constraint
violation.
There is **no way** to control the execution order of `UPDATE` in
SQLite. To persuade yourself, try:
```sql
UPDATE event_chunks
SET position = position - 1
FROM (
SELECT event_id
FROM event_chunks
WHERE position > 2 AND …
ORDER BY position ASC
) as ordered
WHERE event_chunks.event_id = ordered.event_id
```
It will fail the same way.
Thus, we have 2 solutions:
1. Remove the `UNIQUE` constraint,
2. Be creative.
The `UNIQUE` constraint is a safe belt. Normally, we have
`event_cache::Deduplicator` that is responsible to ensure there is no
duplicated event. However, relying on this is “fragile” in the sense it
can contain bugs. Relying on the `UNIQUE` constraint from SQLite is more
robust. It's “braces and belt” as we say here.
So. We need to be creative.
Many solutions exist. Amongst the most popular, we see _dropping and
re-creating the index_, which is no-go for us, it's too expensive. I
(@hywan) have adopted the following one:
- Do `position = position - 1` but in the negative space, so
`position = -(position - 1)`. A position cannot be negative; we are
sure it is unique!
- Once all candidate rows are updated, do `position = -position` to move
back to the positive space.
'told you it's gonna be creative.
This solution is a hack, **but** it is a small number of operations, and
we can keep the `UNIQUE` constraint in place.
This patch updates the `test_linked_chunk_remove_item` to handle
6 events. On _my_ system, with _my_ SQLite version, it triggers the
`UNIQUE` constraint violation without the bug fix.
This is necessary because the `NotificationClient` runs a sliding sync loop and the retrieved data isn't pushed back into the parent client stores (because of cross process locking shenanigans).
This will be used with the previously introduced `org.matrix.msc3401.call.member` required state to check whether a room still has an ongoing call before showing the ringing screen.
This is apparently the right way to do it, both because some HS expect only this mimetype and also so we don't leak the mime type of the encrypted media.
This commit simplifies the filter public api.
Rethinking the public api we only need:
- to know if events can be sent based on the capabilities
- to know if events can be sent to the widget (read) based on the capabilities
- if it even makes sense to sent a cs api read request or if all possibly returned events
would not match the type.
To simplify the code in the machine it also made sense to add `From` implementation
to the FilterInputs instead of gathering the relevant data from all kinds of Raw events.
The new api is simpler:
All possible events we need to check can be converted into filter inputs (using `into()`).
`capabilites` has two allow_read/allow_send that consume filter inputs.
`capabilites` can be asked if there is any filter for specific event types
to allow not send unnecassary requests.
This is (sadly) required since we cannot do `as_str()` for `TimlineEventType` and other `ruma` event types.
So we need to use `String`. We also never used them more specifically than strings.
Previously, the `share_strategy` was breaking the abstraction provided
by `OutboundGroupSession` by accessing its internal fields in an
inconsistent and adhoc way. Now all fields are private and a proper
abstraction was added to access the required state in a consistent API.
This patch moves the `session_id` field from EncryptionInfo to
AlgorithmInfo::MegolmV1AesSha2 as it is specific to Megolm. We provide
transparent migration of the serialized data from one format to the other.
In the future we plan to reuse `EncryptionInfo` for to_device decryption
(using olm not megolm). So megolm session_id should move to algorithm
specific data.
This method had a confusing name: it didn't receive a key bundle, but
rather the data *about* a key bundle.
Remove the unused `sender_key` parameter while we are at it: we use the
embedded (and already-checked) `event.sender_device_keys` here.
This avoids conditions where a key may be shared with a device only
after we decided that it is fine to reuse (and not rotate) the session
based on the wrong assumption that that particular device does not have
the keys.
Signed-off-by: Niklas Baumstark
[niklas.baumstark@gmail.com](mailto:niklas.baumstark@gmail.com)
Previously, `is_session_overshared_for_user` did not take into account
that `shared_with_set` also contains withheld device IDs who explicitly
have never received the session keys. This would lead to it mistakenly
determining oversharing for those devices for every event being sent in
the presence of blacklisted/withheld devices in the room, and rotating
the group session accordingly.
The fix is to correctly exclude devices with `ShareInfo::Withheld` from
the enumeration.
Signed-off-by: Niklas Baumstark niklas.baumstark@gmail.com
This patch updates the sending side of the `sender_device_keys` field
introduced in MSC4147.
Since the MSC got merged, we're switching from the unstable identifier
to the stable one.
A couple of snapshot tests were added modified to make this happen.
This PR is a revived version of:
https://github.com/matrix-org/matrix-rust-sdk/pull/4707 since it is now
possible to easily add wasm support thanks to:
https://github.com/matrix-org/matrix-rust-sdk/pull/4572
Using the `executer::spawn` will select `tokio::spawn` or
`wasm_bindgen_futures::spawn_local` based on what platform we are on.
~~They behave differently in that `tokio` actually starts the async
block inside the spawn but the wasm `spawn` wrapper will only start the
part that is not inside the `async` block and the join handle needs to
be awaited for it to work.~~
This has now changed with:
https://github.com/matrix-org/matrix-rust-sdk/pull/4572.
Now they behave the same and this PR becomes a very simple change.
Rather than attempting to trigger Alice's encryption sync loop with an incoming
message from Bob, instead have Alice send a message once Bob has joined the
room.
Simplifies the `ProcessedToDeviceEvent` enum by converting its variants to tuple variants.
This change improves code readability and conciseness by removing the need for named fields within the variants.
This patch introduces a temporary hack.
So here is the thing. Ideally, we DO NOT want to emit this reason.
It does not makes sense. However, all notable update reasons
are not clearly identified so far. Why is it a problem? The
`matrix_sdk_ui::room_list_service::RoomList` is listening this stream
of [`RoomInfoNotableUpdate`], and emits an update on a room item if
it receives a notable reason. Because all reasons are not identified,
we are likely to miss particular updates, and it can feel broken.
Ultimately, we want to clearly identify all the notable update reasons,
and remove this one.
This patch creates the `room::display_name::update_for_rooms` response
processor. It also creates the `changes::save_only` response processor.
Finally, this patch uses both to compute the new display name for all
rooms and save them, for sliding sync only.
This patch extracts the “save” part of the `save_and_apply` response
processor into its own function. Thus we have `save_changes` (new) and
`apply_changes` that are used by `save_and_apply`.
This patch introduces a new enum: `UpdatedRoomDisplayName`, which
will be used to know if a room display name is different or not when
computing the room display name.
…that holds information on the thread the given item is the root of
- it holds the latest event content and sender at the moment but will
hold more information in the future e.g. number of replies, if it's
unread etc.
- the field is not currently being populate but is delivered earlier so
it can power shipping the UI side on the embedders
See #4891 that shows a case where we should have saved the
previous-batch token, and instead ditched it, in the previous version.
Changes include:
- moving the code deciding to keep or ditch the `previous-batch` token
into `append_events_locked`.
- tweak the condition to ditch, so that the `previous-batch` token is
ditched only if we didn't have events in the event cache in the first
place, in addition to having storage + the timeline not being marked as
limited explicitly.
Credits to @zecakeh for the test case.
This patch is twofold:
1. it transforms the `&mut` to a `&` for the room account data in the
sliding sync flow, which allows to remove one big clone!
2. it adds the `room_account_data` response processor.
This patch creates the new `room::msc4186::extensions::ephemeral_events`
response processor.
Ideally we would like to merge this with `ephemeral_events`, but they
are a bit different. Let's see how it evolves in the future.
This patch uses the `room::Room` structure in `room::sync_v2` to
reduce the number of arguments of the `update_joined_room` and the
`update_left_room` processors.
This patch uses the `room::Room` structure in
`room::msc4186::update_any_room` to reduce its number of arguments. It
results in the removal of te `allow(clippy::too_many_arguments)`.
This patch removes the `SessionMeta` argument of
`room::msc4186::update_any_room`. Tracking its usage, it reveals that
providing a `UserId` is sufficient. Happily, we already provide a
`UserId`, hence making `SessionMeta` useless here.
This patch renames the fields in `RoomUpdate`: `join` becomes `joined`,
`leave` becomes `left`, `invite` becomes `invited`, to match their
associate type.
This patch simplifies the flow of `membership` when the room is an
invited room.
Previously, we were creating the room as `Invited`. Then, later
overriding it to `Knocked` if we found it was such a room. Otherwise, we
were overriding it again to `Invited`. Well. Now the flow is simpler.
This patch extracts the `BaseClient::process_sliding_sync_room` method
into the new `msc4186::update_any_room` response processor. So far, this
is purely a code move, without any changes (modulo method-to-function
transformation).
With `BaseClient:process_sliding_sync_room` comes
`BaseClient::process_sliding_sync_room_membership`,
`BaseClient::handle_own_room_membership`, `process_room_properties` and
`cache_latest_events`.
This patch removes the re-exports of `Notification` and `E2EE` in
`timeline::builder`. Let's make things simple now that they are used
outside the `timeline` processors.
This patch groups arguments behind the `Notification` struct
if it makes sense. This patch also uses the new method
`Notification::push_notification_from_event_if` method to replace
duplicated code.
This patch moves the `timeline::builder::Notification` into
its own module, `notification`. This patch adds two methods:
`push_notification` and `push_notification_from_event_if`.
This patch adds two `From` implementations on
`RawAnySyncOrStrippedTimelineEvent` to create the correct variant of
this enum. This is useful when we get generic types and want to build
this type.
This patch creates the `update_left_room` processor, extracted from the
sync v2 flow.
I've noticed that `new_user_ids` is not used, so it has been put behind
a `_` variable for the moment, but some computations have been removed.
We need to clean this user ID flow.
This patch moves the `timeline::builder::E2EE` type into the `e2ee`
module. Gathering these 3 E2EE values in the same type was a good idea,
and is now applied to more places in the response processors.
This patch makes `Context` to derive `Default`. The `new` constructor
also no longer takes a `RoomInfoNotableUpdates`, since it was always
used with a default value, it generates this value by itself.
This patch removes all mentions of `include_heroes`. This field isn't
part of the MSC4186. We have the `test_compute_heroes_from_sliding_sync`
which continues to pass, everything fine on that side. There is no
mention of `include_heroes` in Synapse at the time of writing. It seems
to be an artifact from the sliding sync proxy experiment.
This patch moves the `state_events::dispatch_and_get_new_users`
processor inside the `state_events::sync` module. Why? Because a similar
processor for `state_events::stripped` is about to be created.
MSC4147 added a `sender_device_keys` property to olm-encrypted to-device
messages, with recommendations about checking the values in that propety. We do
(most of?) those checks for `m.room_key` messages today, but not other types of
to-device message.
MSC4147 added a `sender_device_keys` property to the plaintext of *all*
olm-encrypted events. 03d4a30eb added the field to `DecryptedOlmV1Event`, but
due to Reasons, there is an almost-parallel struct `ToDeviceCustomEvent` which
is used for event types other than the 4 we have content types for.
To complete the set, let's add the field to `ToDeviceCustomEvent`.
This patch tries to clarify the processors in `state_events` by
putting them in two modules: `sync` and `stripped`, along with a bit
of renaming:
- `collect_sync` becomes `sync::collect`,
- `collect_sync_from_timeline` becomes `sync::collect_from_timeline`,
- `collect_stripped` becomes `stripped::collect`.
I believe this is an improvement.
This patch does the following things:
1. moves the `BaseClient::handle_timeline` method as the
`timeline::build` response processor.
2. moves the `BaseClient::update_push_room_context` method as a private
function of the `timeline` response processor module,
3. moves the `BaseClient::get_push_room_context` method as a public
function of the `timeline` response processor module.
The scope of the `timeline::build` processor is a bit too broad, but at
least the number of methods on `BaseClient` are greatly reduced.
This patch creates a new `verification::process_if_candidate` response
process. The idea is the check whether a `AnySyncTimelineEvent` suits to
be a verification event that must be processed.
This piece of code was repeated in three different places in the code.
Now it's unique.
This patch removes the `ignore_state_events` argument from
`BaseClient::handle_timeline`. This method no longer handles state
events. Consequently, the `user_ids` and `ambiguity_cache` arguments are
also removed.
This patch updates the flow to use the new
`state_events::collect_sync_from_timeline` processor, and to re-use the
`state_events::dispatch_and_get_new_users` processor.
The positive impact of this patch is that it re-uses existing code
(hurray). However, the downside is that state events are deserialized
twice for the moment.
This patch extracts the `BaseClient::handle_state` method as a new
response processor named `state_events::dispatch_and_get_new_users`.
It appears that we can do something more elegant with the returned new
users. See the next patch.
This patch rewrites `e2e::decrypt::sync_timeline_event` to avoid calling
a `verification` twice with the same arguments, and to avoid a string
allocation (the `to_string`).
Also, I wasn't comfortable with the `starts_with` which wasn't included
a trailing `.`. At least now we rely on strongly typed event type.
This will hold the info for both the room member whose sender id was provided and the sender of the `m.room.member` event from which the `RoomMember` is built.
We don't need to list the `server_versions` every time we do an http
request. Further, `config` is listed under both `skip` and `fields`, which ends
up being a no-op. I don't think it's very useful (and is quite noisy), so
let's remove it.
In parallel to a sliding sync request, we also check for and send pending outgoing
requests from the crypto stack. Currently, these drop the tracing span, which
loses valuable data. We should propagate the span.
We also add an extra layer of instrumentation so that we can differentiate
between the results of the sliding sync itself, and the outgoing requests.
Currently, if you have two clients both syncing away inside the same process,
it's approximately impossible to see which logs belong to which client. It
would be much better if we could use a tracing span to distinguish the two
clients.
I expect this to be mostly useful in integration tests, but it might be useful
elsewhere too.
This reduces the framerate from ~62fps to 10fps.
This is a workaround for a problem in my terminal, so apologies for inflicting
it on everyone else, but here we are, and 10fps seems like it should be enough
for anyone.
The problem in question is specifically when I try to select some text by
dragging the mouse (eg, to copy a generated recovery key). If I start a drag,
but a redraw happens before the mouse has moved [a certain distance?], then the
drag doesn't work, and nothing gets selected. By reducing the framerate, I have
a much better chance of successfully starting a drag.
Currently, `h` and `l` are intercepted by the parent view to change tab,
meaning it's impossible to enter a recovery key which contains those
characters.
The fix here is very blunt: it just disables `h` and `l` for tab-changing. I
considered making it dependent on which tab is open, or what's going on in the
'Encryption' tab, but given you need to know about the alternatives (tab/cursor
keys) to switch away from the Encryption tab, I don't think that makes sense.
This patch moves the `handle_room_account_data` method as the new
`account_data::for_room` response processor.
Instead of taking the `BaseClient` to fetch the room in `on_room_info`,
it now takes a `BaseStateStore`, which is safer and more straight to
the point.
This patch changes `AccountDataProcessor` to
`processors::account_data::global`. The next patch will introduce a
per-room account data processor, hence the renaming to `Global` to make
the difference between the twos.
This patch renames a couple of variables because it's important
to understand that those stripped state events are coming from the
`invite_state` value of a sliding sync response. This is a pretty
important detail.
This patch creates the `state_events::collect_sync`
and `collect_stripped` response processors. It
removes the `BaseClient::deserialize_state_events` and
`deserialize_stripped_state_events` methods. It also removes a couple of
`Vec` allocations and a couple of clones.
This patch creates the `changes::save_and_apply` response
processor. It consists of moving `BaseClient::apply_changes` plus
the code that is always around. This patch helps to remove the
`BaseClient::load_previous_ignored_user_list` method.
This patch moves the `handle_room_member_event_for_profiles`
function inside the collection of response processors under the name
`profiles::upsert_or_delete`.
Reduce the size of the tuple that this thing returns by defining a new result
type.
I haven't put the `share_infos` in the struct, because I'm going to reuse the
same struct for another method where `share_infos` aren't needed.
This is needed to identify the event as being in a thread, and to reply
to it inside the thread instead of in the room's timeline.
<!-- description of the changes in this PR -->
- [ ] Public API changes documented in changelogs (optional)
<!-- Sign-off, if not part of the commits -->
<!-- See CONTRIBUTING.md if you don't know what this is -->
Signed-off-by:
This allows us to view the debug screen at the same time as the
timeline. The details view can be to the right of the timeline or bellow
the timeline and we can switch using ALT-t.
This patch adds `ClientBuilder::system_is_memory_constrained`
so that the client can be built with that in mind.
Behind the scene, for the moment, it only calls
`SqliteStoreConfig::with_low_memory_config` instead of
`SqliteStoreConfig::new`, but this flag can be used for other use cases.
This patch adds a new constructor for `SqliteStoreConfig`, which sets
some defaults tailored for low memory usage.
This patch adds tests asserting the defaults for `new` and
`with_low_memory_config`.
This patch explores a way to simplify the call sites of the `e2ee`
response processor by creating one response processor for `/v3/sync` and
one for MSC4186. The idea is to:
- simplify the call site by having less code,
- isolating the “dispatch” of a the response values into the `e2ee`
response processor,
- make it easier to test this response processor based on a `Response`
structs directly.
This patch updates
`BaseClient::receive_sync_repsonse_with_requested_required_states` to
use the `response_processors`. This patch also removes duplicated code
with the processors.
This patch introduces a new `Context` type that holds the state changes
and the room info notable updates. Processors will exchange this
context. That's the only data that is mutable and exchangeable between
processors.
There were a lot of changes, and it was hard to follow, especially with
methods and types that changed and were then removed.
This groups everything under a single entry, with a short summary of
the changes, followed by a list of per-PR changes.
The detailed list is reorganized to put the biggest changes first, like
the renaming, and a few entries were cleaned up or removed, because
they mention a method or type that is removed in another entry.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch boxes `local_echo` in
`DependentQueuedRequestKind::FinishUpload`, this variant' size is 512
bytes, compared to the second largest variant which is 104 bytes. To
reduce the global size of this enum, `local_echo` is now a `Box<_>`.
This patch reduces the size of `anySyncOrStrippedState`. The `Sync`
variant is 528 bytes, the `Stripped` variant is 232. First off, there is
a non-negligible difference in size between the two, but still, we can
reduce the size of the enum by boxing all values.
It simplifies code for users, and avoids to have to match on
`AuthApi`, which is a non-exhaustive enum.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
It is actually unused, and now that we only need homeserver URLs for
static registrations, users don't need to access it easily.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
MSC2966 was updated, clients should re-register for every log in, so we
don't need to store the client IDs between logins.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This makes it possible to reply with a media, as part of a thread or not.
Fixes#4835.
---------
Signed-off-by: Johannes Marbach <n0-0ne+github@mailbox.org>
Clearing only the store backend, while not emptying the in-memory linked
chunks, will lead to out-of-sync state across the in-memory caches and
the database. As a result, it's safer to call the existing
`EventCacheInner::clear_all_rooms`, which will clear all the rooms
manually.
A redaction event would be either applied a priori (by the server, when
returning the sync response), or the event cache would handle it, and
redact it in the database; in any case, we'd never see the original
event in its non-redacted form, so there's no point in returning the
redaction event itself.
It's unclear whether it's useful, especially in the case where it would
return an entire reply chain. It's not possible to filter in replies
only, using the function either, which is a sign that replies shouldn't
be indexed, IMO. In any case, that's something we can add back in the
future, if we want to.
Getting the position when reading an event is no longer required:
- the only use case for reading the position out of the event cache was
when we wanted to replace a redacted item into the linked chunk; now
with save_event(), we can replace it without having to know its
position.
As an extra measure of caution, I've also included the room_id in the
`events` table, next to the event_id, so that looking for an event is
still restricted to a single room.
This is necessary to save out-of-band items into the relational linked
chunk. I'm not quite sure of the value to keep it generic, at this
point, but at least it makes testing easy.
This patch adds 3 methods on `ClientBuilder`:
1. `session_pool_max_size`,
2. `session_cache_size`,
3. `session_journal_size_limit`.
Respective fields are also added.
These values control the `SqliteStoreConfig`, used to control the
stores, especially their memory consumption.
This patch removes the `path` and `passphrase` fields from
`BuilderStoreConfig::Sqlite`, and replaces them by `SqliteStoreConfig`.
This patch then opens the stores with `open_with_config` instead of
`open`.
At the default `INFO` log level, the log gets inundated with thousands
of emitted statements, primarily related to
`is_display_name_ambiguous()`. The execution of this tiny function
certainly doesn't need to be traced every time, at least not at the info
log level.
Note: some of these could be debatably reduced to debug level rather
than trace level, but I went with "trace" because they all seem to be
trace statements rather than actual debug dump outputs (there is no
actual program state dumped out).
Signed-off-by: Kevin Boos
[kevinaboos@gmail.com](mailto:kevinaboos@gmail.com)
---------
Signed-off-by: Kevin Boos <1139460+kevinaboos@users.noreply.github.com>
This patch updates `BaseStateStore` and the `StateStore` trait along
with its implementors, to return all rooms or a single room from
`StateStore::get_room_infos`.
See the previous patch for more context.
This patch introduces the `RoomLoadSettings` enum. It is helpful to load
either all rooms or one room when activating a `BaseClient`, i.e. when a
session is initialized or restored.
It addresses a broader problem where, for large accounts with large
caches, creating a `BaseClient` takes many resources. In a resource
constrainted context, like a push notification process, it can eat all
resources up to the point the process is killed (then notifications can
be missed).
The idea is then to force the `BaseClient` to load a single room.
This patch installs the `RoomLoadSettings` argument everywhere it needs
to be. The next patch will use `RoomLoadSettings` to load either all
rooms or a single one.
Since we don't depend on mas anymore, we don't depend on RSA either.
Let's remove the exception, lest we reintroduce the dependency and
security issue.
This patch splits `BaseStateStore::set_or_reload_session` into 3
distinct methods:
1. `set_session_meta` (hello again!),
2. `load_rooms`,
3. `load_sync_token`.
This patch also renames
`BaseStateStore::set_or_reload_session_from_other` into
`derive_from_other` to clarify its semantics. It calls these 3 methods
above as a combo.
Finally, this patch also updates `BaseClient::activate` to call these 3
methods above individually.
This patch renames `BaseClient::set_or_reload_session`
to `BaseClient::activate`, and `BaseClient::logged_in` to
`BaseClient::is_activated`.
The idea behind these renamings is to introduce a “state” for the
`BaseClient`: it is activated when is has a `SessionMeta`, has loaded
its data from the storages, and has an `OlmMachine`. Consequently, the
`logged_in` method is renamed `is_activated` for the symmetry. If one
wants to know if the client is logged in, it can use `is_activated`, or
also `MatrixAuth::logged_in`.
This patch renames the various `set_session_meta` methods to
`set_or_reload_session`. The idea is to highlight that the method is not
a simple setter: it sets but it _also_ updates the store' state.
The private shortcut method in `matrix_sdk::Client::set_session_meta`
is removed, and caller uses `client.base_client().set_or_reload_session`
instead. Why removing this `Client::set_session_meta` shortcut
method? Because it was creating confusion with another method:
`Client::restoring_session`. This private shortcut method wasn't used in
a lot of places, then I believe it's a nice improvement.
This patch updates `StoreOpenConfig` to hold a new type: `RuntimeConfig`.
This `RuntimeConfig` type is passed to a new `SqliteAsyncConnExt`
method, named `apply_runtime_config`. Depending on the values passed
here, the `optimize`, `cache_size` (new!) and `journal_size_limit`
methods will be called automatically.
The goal of this type is to automate a flow we keep repeating in
all the stores. This is error-prone. This type brings uniformity and
consistency.
This patch also makes all `open_with_pool` methods on the stores private
(they were public before):
1. they were never used as far as I know because getting a `SqlitePool`
isn't possible since the `pool` attribute is private…
2. it's better to keep control of this flow.
This patch pursues the same goal as the previous one: `Store` has
been renamed `BaseStateStore`, so the `store` field holding this
`BaseStateStore` is renamed `state_store`.
This patch renames `Store` to `BaseStateStore`. Ideally, I would
like to rename to `StateStore` but that's already a trait name
(`traits::StateStore`).
Why this renaming? To clarify what store it is.
In preparation for threads we have realised that the `reactions`,
`thread_root` and `in_reply_to` were only available on `Message` types,
which doesn't play well with Stickers and Polls.
This PR introduces a new `Aggregated` `TimelineItemContent` variant
which holds the message `kind` (Message, Sticker, Poll) as well as well
as any related aggregated data. it will help treat them all in a similar
fashion as well as account for future changes.
There are no functional changes, it's mostly about moving code around
and the FFI interfaces haven't changed.
Part of #4833.
If we have a device whose Curve25519 key we don't know, then
self-evidently we can't have any active Olm sessions with that device.
Currently, we return an `EventError::MissingSenderKey` in this case, but
(a) the definition of that error doesn't match this situation, and (b)
it complicates handling in methods that call `DeviceData::encrypt`
(currently only `DeviceData::maybe_encrypt_room_key`, but I want to add
a second).
Other than `DeviceData::encrypt`, the only place where
`get_most_recent_session` is called is `mark_device_as_wedged`. In that
case, we have just looked up the device by its Curve25519 key, so we
know it must have one.
We can therefore be reasonably certain that this change is a no-op.
This renames the `BaseClient::with_store_config` constructor to `new`:
1. there is no alternative constructor, it's the only one,
2. the `with_` prefix is usually reserved to (i) builders or (ii)
alternative constructors,
3. I believe it clarifies the code.
This patch:
* fixes the mention of “no IO”, it lacks the “network” part,
* adds an example of how to build a `BaseClient`,
* mentions that it is better be used via `matrix_sdk`.
It allows to reuse the URL for different actions more easily than having
to call `OAuth::account_management_url` every time for a different
action.
It also adds a method with fallback if we want to ignore action
serialization errors, to always present a URL.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch adds a new `StoreOpenConfing` type to configure the store
when opening it and when creating the pool of connections to SQLite via
`deadpool_sqlite`.
This patch also adds a new `open_with_config` constructor on all
stores, namely `SqliteCryptoStore`, `SqliteEventCacheStore` and
`SqliteStateStore`.
The previous code in `LinkedChunk::drop()` would call `clear()`, which
would reset the chunk to an empty events chunk; preinitializing a vector
of 128 items. But this item chunk would never be dropped, so this would
cause a leak.
The solution is to split the semantics of *resetting* a linked chunk
(what was called `clear` before), from *clearing* it: clearing will put
it in an dangling state, and it's the caller's responsibility to do
something about it; as such, it's marked unsafe. `LinkedChunk::drop()`
may now call `clear()` (which is fine; last use of the linked chunk);
and `LinkedChunk::reset()` will call `clear()` and reset the first
chunk, which is fine too.
This is a nice simplification, because this means that:
1. we use a single way to get the event (event-cache-or-network)
2. we don't have to reconstruct a `RoomMessageEventContent` from a
timeline's message, which seems a bit error prone
3. there's a single way to get the replied-to info for an event,
4. and that's actually independent of the timeline, so we can improve
the code for #4835
`send_reply` is geared towards clients that don't render threads themselves. It sends a reply and optionally forwards it onto any existing thread.
This PR adds `send_thread_reply` for clients that do render threads themselves. This sends a message onto a thread, regardless of whether a thread existed before, and includes the fallback for non-threaded clients only if the message is not itself a reply on the thread.
Signed-off-by: Johannes Marbach <n0-0ne+github@mailbox.org>
Accept a URL or a query string for simplicity.
That way we don't need to expose AuthorizationResponse.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Since they perform blocking I/O we probably don't want to block a thread on that.
We use spawn_blocking, the alternative would be to use tokio::fs functions, which do the same thing and would require to load the whole file content in memory before (de)serialization.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Use the same prefix as the other types in the OAuth 2.0 API, and use the
same suffix as other data-persisting APIs for consistency.
It also avoids to have two modules with very similar names, the only
difference being a trailing `s`.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Since we are the ones generating the device ID, we have a way to avoid this error. Even if in practice, it's probably always included in the server's response.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Can be useful with soft logouts, without requiring the user to recreate a new Client to log in again.
Returns an error if the new session is different from the current one.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
That way users only need to call finish_login, since there is no other
reason to call finish_authorization currently.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Fixes: #4793
There was a previous PR https://github.com/matrix-org/matrix-rust-sdk/pull/4802 which attempted to implement this, but missed some backwards compatibility needs.
This updated PR has the original commit and then additional commits to add the compatibility (along with tests for the new intent param generally).
This patch updates `RoomEventCacheState::remove_events` to check whether
the set of events are not empty before removing them. When removing
`in_memory_events`, it avoids taking a write lock on the `RoomEvents`
for nothing for example.
This patch adds a new check when inserting a new first chunk. It makes
some tests to fail but because they were not realistic. This patch then
updates these tests.
Because yes, some weeks, we are very productive!
This patch adds `--limit 100` to `gh pr list` so that we are sure to not
miss pull requests if there are many.
This patch fixes a bug where events coming from the event cache might be
encrypted, see https://github.com/matrix-org/matrix-rust-sdk/issues/4762
to learn more.
This patch updates the `room_event_cache_updates_task` to call
`TimelineController::retry_event_decryption` if the origin is `Cache`.
This patch removes the room `&Room` argument of
`TimelineController::retry_event_decryption`. The `TimelineController`
already has the room with its `room(&self) -> &Room` method. This method
was always used to fetch the room, let's expect `retry_event_decryption`
to do that by itself.
It also prevents passing the “wrong” room. This is more robust this way.
When `BaseClient` is created, the rooms aren't loaded yet. Then,
calculating the size of the channel for `RoomInfoNotableUpdate` is
useless, as it will always be zero, and we will fallback to 500 every
time. By the way, 500 is a nice default. This patch uses this value as
the only channel's size.
Allow use of a server url (eg `http://localhost:8008`), enabling connection to
a local server rather than one which supports well-known, TLS, and the rest.
This patch updates `replace_all_events_by` to take an `EventsOrigin`.
It is called in two places: by `add_initial_events`, in which case it
is `EventsOrigin::Cache`, and by `handle_timeline`, in which case it is
`EventsOrigin::Sync`.
This patch updates the `EventsOrigin` value sent by `Room::clear` to be
`Cache` instead of `Sync`. No events are added, but the `VectorDiff`s
are generated from the event cache itself, not from a sync.
This patch changes the log message. A `TimelineItemPosition::UpdateAt`
can only happen for remote event, not local event. The log message
was talking about _decryption_ but an `UpdateAt` can also happen for
redaction. This was misleading.
This patch updates the `RemoteEventOrigin` value passed to
`TimelineController::replace_with_initial_remote_events` when there is
a lag with the event cache. The previous value was `Sync`, but it should
be `Cache` since these events come from the event _cache_ (it was in the
name, easy).
This patch updates the sync code to include the
`RequestedRequiredStates` type.
This patch also adds `RoomInfo::handle_encryption_state` which
is able to mark an encryption state as synced depending of
`RequestedRequiredStates` (read the comment in the code).
This patch also updates the documentation of
`RoomInfo::handle_state_event` to clarify the impact of a
`m.room.encryption` state event.
This patch introduces a new type: `RequestedRequiredStates`, which keeps
track of all `required_states` passed to a sync request. So far, there
is only a `From` implementation for MSC4186.
These prompts were used in the Element X app, probably in some other clients too.
Since Ruma removed these cases, we're just passing them as `_Custom(value)` ones, which do work.
And make it take a boolean indicating whether we want to set up a
lightweight tokio runtime or not, instead of having
`setup_lightweight_tokio_runtime` as a public function + another
function, both of which would have to be called anyways.
cc @stefanceriu @jmartinesp
Creating many threads may use a bit of memory: on a machine with N
devices, exactly N*2 MB of memory may be consumed.
That might be a lot for a NSE process on iOS, which can only have up to
16 MB of RAM allocated for it. For this case, we introduce a new FFI
method `setup_lightweight_tokio_runtime` which will spawn at most 4
worker threads and 1 blocking thread. This should be sufficient for most
use cases.
As opposed to WAL mode, foreign keys must be enabled for each database
connection, according to
https://www.sqlite.org/foreignkeys.html#fk_enable
Unfortunately, we can't track which connection objects have already
executed the pragma, so the safer we can do is enable it everytime we
try to acquire a connection from the pool.
Fixes#4785.
A linked chunk never wants to be empty. However, after a limited gap
that doesn't contain events, it may be shrunk to the latest chunk that's
a gap.
If later we decide to remove the gap (because it's been resolved with no
events), then we would try to remove the last chunk, which is not
correct.
Ideally, we'd keep an events chunk around; but if we have an events
chunk *before* a gap, that may look like missing events to the user, at
least until the gap has been resolved.
The fix to this problem is to *not* optimize / remove the gap, if it's
the only chunk kept in memory. This was only a memory optimization, but
it's not absolutely required per se.
This patch reduces the memory usage of the broadcast channel used by
`BaseClient::room_info_notable_update_sender`. So far, its size was
`u16::MAX`. Considering `RoomInfoNotableUpdate` is 24 bytes, the channel
was allocating 1.5Mb of memory, which is way too much. It is creating
problems on systems where the process has limited resources, like the
Notification Service Extension on iOS.
For a regular users with 200 rooms, the memory usage becomes 24Kb, which
is 65'536 times less.
Being able to always use the first redirect URI in the client metadata
seems to be very specific to the FFI bindings.
For example clients that need to bind a port on localhost need to
provide a custom redirect URI each time.
So we ask for the redirect URI, and keep the current behavior only for
the bindings.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch introduces the new `EncryptionState` to represent the 3
possible states: `Encrypted`, `NotEncrypted` or `Unknown`. All the
`is_encrypted` methods have been replaced by `encryption_state`.
The most noticable change is in `matrix_sdk::Room` where `async fn
is_encrypted(&self) -> Result<bool>` has been replaced by `fn fn
encryption_state(&self) -> EncryptionState`. However, a new `async
fn latest_encryption_state(&self) -> Result<EncryptionState>` method
“restores” the previous behaviour by calling `request_encryption_state`
if necessary.
The idea is that the caller is now responsible to call
`request_encryption_state` if desired, or use `latest_encryption_state`
to automate the call if necessary. `encryption_state` is now non-async
and infallible everywhere.
`matrix-sdk-ffi` has been updated but no methods have been added for
the moment.
It's unusual to have the method on the parent type when the field type
could also hold the method. In fact, this was the only bool getter
inspecting the timeline's content, so let's move the method next to as
its siblings, for consistency, and let's spell it out fully for clarity.
Instead of MockBuilder::respond_with. This reduces duplcation and will
allow to add some common logic when building the endpoints.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
The SyncService::stop method guarantees that the sync service will be
stopped after it has completed so there's no need to wait for state
changes.
The state change might not even come, if you pressed `S` to stop the
sync service manually.
This is done to fix an issue with these events being received and processed twice when `NotificationProcessSetup` is `SingleProcess`, causing issues with user verification.
This can be used to ignore verification requests in this sliding sync instance, preventing issues found where several sliding sync instances with the same client process events simultaneously and re-process the same verification request events during their initial syncs.
Many fields here are not argument of the `send` method, but are set
later with `Span::record`. Grepping all these fields reveal they are all
set except `request_body` apparently.
This patch is twofold. First off, it provides a new schema allowing to
improve the performance of `SqliteEventCacheStore` for 100_000 events
from 6.7k events/sec to 284k events/sec on my machine.
Second, it now assumes that `EventCacheStore` does NOT store invalid
events. It was already the case, but the SQLite schema was not rejecting
invalid event in case some were handled. It's now explicitely forbidden.
This patch removes the `EventsPostProcessing` type, it assumes
`with_events_muts` will always return events that will be post-process.
The case where `EventsPostProcessing::None` becomes a `vec![]`.
This patch updates `maybe_apply_new_redaction` to remove the first
`&RoomVersionId` argument. Indeed, due to the refactoring, it's now
possible for `maybe_apply_new_redaction` to read this value directly
from `Self::room_version`.
This patch updates `maybe_apply_new_redaction` to use `find_event`, so
that the target event is looked up in memory or in the store.
The case where it is in the store is a simple `todo!()` for the moment.
I wanted to separate the update of the `maybe_apply_new_redaction`
signature from the `InStore` implementation. The method is now async and
returns a `Result`.
This patch introduces `EventLocation` to know if an event has been found
in the memory (in `RoomEvents`) or in the store (in `EventCacheStore`).
This is used by the `RoomEventCacheState::find_event`.
This patch moves the `maybe_apply_new_redaction` method from
`RoomEvents` inside `RoomEventCacheState` so that it has an access
to the store (necessary for the next patch). This patch creates a new
`RoomEvents::replace_event_at` method, which is a thin wrapper around
`LinkedChunk::replace_item_at`.
This patch renames `sync_timeline_events` into `timeline_events`.
Moreover, this change has spotted a possible improvement
in `AllEventsCache` where it now receives events from
`collect_valid_and_duplicated_events`, which allows to only store valid
events in it.
This patch updates the callback passed to `with_events_mut`. It now
returns an `EventsPostProcessing` which can automatically run the, now
inlined, `on_new_events`.
This patch updates where the `RoomVersionId` is also stored. It's not
held by `RoomEventCacheState` instead of `RoomEventCacheInner`.
I have to disable LTO every time I'm building any final binary using the
SDK, because otherwise, the builds can take easily more than 10 minutes
to complete, killing iteration times, and making it almost impractical
to use the programs (benchmarks, or multiverse).
I think it should be the decision of the final embedder to enable or
disable LTO, and that for the purpose of our own binaries hosted in the
SDK repository, we don't need the absolute best performance (or, for the
sake of benchmarking, we can tweak the profiling profile).
The linked chunk always starts with an empty events chunk. If we receive
a gap from sync, then we will immediately push a gap chunk; in this
case, it might be better to replace the events chunk with a gap chunk.
This is equivalent to removing the empty events chunk, after pushing
back the first one (we can't do it before, otherwise we might get rid of
the only chunk in the linked chunk, which breaks the invariant that a
linked chunk is never empty).
And don'y rely on the `Paginator`. This simplifies the code a bit,
avoids a few methods on the `Paginator`, and makes it more
straightforward the pagination happens.
We keep the mock backend for endpoints that require an ID token for now,
as it would involve generating them on the fly.
And since support for ID tokens is going to be removed, it is not worth
it to implement that.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Instead of keeping state for the `Paginator` instance, we create one
when needs be, in the `run_backwards_impl` method, and initialize it
with a previous-batch token. This is simpler than keeping one alive, and
making sure that we reset it in the right places.
The event cache doesn't use the paginator for forwards pagination, so it
doesn't make sense to expose this method. Moreover, the event cache
always listens to sync in real-time, so technically it's always hit the
timeline end.
As a proof of this, this method wasn't even tested.
This patch adds support for the `shared_history` flag from MSC3061 to
the `m.room_key` content, exported room keys, and backed-up room keys.
The flag is now persisted in our `InboundGroupSession`. Additionally,
when creating a new `InboundGroupSession`, we ensure the
`shared_history` flag is set appropriately.
MSC3061: https://github.com/matrix-org/matrix-spec-proposals/pull/3061
In several places, we access almost all the fields of a struct to
create an `InboundGroupSession` from a pure data struct.
When new fields are added to these data structs, it's easy to
overlook updating the `InboundGroupSession` accordingly.
By using struct destructuring, we ensure that newly added fields are
explicitly considered, making it harder to forget one of the newly added
fields.
`prompt=create` is defined in MSC2964, and
`login_hint=mxid:@user:server.name` is defined in MSC4198.
The other parameters came from OpenID Connect.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
It is a small optimization which makes the URL smaller, but it is not
part of the next-gen auth MSCs and is not supported by the oauth2 crate,
so let's drop it.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch fixes `RoomEventCache::event` to look inside `RoomEvents` but
also inside the `EventCacheStore` to look for an event. It ultimately
fallbacks to `AllEventsCache` because we can't get rid of it for the
moment.
This patch makes the code more robust around event removals. Sorting
events by their position is no longer done in the `Deduplicator` but in
a new `RoomEventCacheState::remove_events` method, which removes events
in the store and in the `RoomEvents`. This method is responsible to sort
events, this stuff is less fragile like so.
This patch adds a test for `sort_events_by_position_descending`. It
also updates this function so that events are sorted by their chunk
identifier from newest to oldest, it makes no difference but it matches
the order of the position indices too. Everything “dimension” is
descending.
This patch uses `DeduplicationOutcome` to remove events either in
memory, or in the store, when required. The `remove_events_by_id` method
has been renamed `remove_events_by_position`.
This patch redesigns `Deduplicator::filter_duplicate_events`.
First off, `filter_duplicate_events` does remove events with no valid
ID. At the same time, it removes duplicate events within the new events
(`events`). This check was done in the `BloomFilterDeduplicator` but
not in the `StoreDeduplicator`. Now it's done at the front of these
implementations, directly inside `Deduplicator`.
Second, this patch introduces `DeduplicationOutcome` to replace the
return type `(Vec<Event>, Vec<OwnedEventId>)`, especially because
now it would have become `(Vec<Event>, Vec<(OwnedEventId, Position)>,
Vec<(OwnedEventId, Position)>)`. Why?
1. Because the positions of the duplicated events are returned,
2. We differentiate between in-memory vs. in-store duplicated events.
Third, now there are positions associated to duplicated events, events
must be sorted. It's the role of `sort_events_by_position_descending`.
This way, `DeduplicatorOutcome` brings guarantees and less checks are
required.
It could be that we have a mismatch between network and disk, after
running a back-pagination:
- network indicates start of the timeline, aka there's no previous-batch
token
- but in the persisted storage, we do have an initial empty events chunk
Because of this, we could have weird transitions from "I've reached the
start of the room" to "I haven't actually reached it", if calling the
`run_backwards()` method manually.
This patch rewrites the logic when returning `reached_start`, so that
it's more precise:
- when reloading an events chunk from disk, rely on the previous chunk
property to indicate whether we've reached the start of the timeline,
thus avoiding unnecessary calls to back-paginations.
- after resolving a gap via the network, override the result of
`reached_start` with a boolean that indicates 1. there are no more gaps
and 2. there's no previous chunk (actual previous or lazily-loaded).
In the future, we should consider NOT having empty events chunks, if we
can.
Token revocation was split out from MSC2964 to MSC4254, and RP-Initiated
logout is now mentioned only as an alternative.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Will allow to share code when the backend is switched to the oauth2 crate too.
It will also allow to expose the device authorization grant directly in Oidc, if necessary.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
The `.from()`, `.with_delay()` and `.limit()` functions are not very
explicit about what they did, and the `with_delay` one in particular led
me to think that it would introduce a delay *in the response*, while it
indicated a delay was expected as part of the matched URL.
Instead, this patch proposes to prefix all matchers with `match_`, to
make it clearer that… they will match the incoming query: match_from,
match_delay, match_limit.
Thanks to this change, the `RoomMessagesResponseTemplate` can be renamed
from `delayed` to `with_delay`, which was my original intent, before I
noticed another `with_delay` with totally different semantics.
This patch adds the new `from_all_chunks` function in the
`linked_chunk::lazy_loader` module. It is only used for testing
purposes. It aims at replacing `LinkedChunkBuilderTest` (see next
patches). Why? Because `from_all_chunks` uses `from_last_chunk` and
`insert_new_first_chunk`: if `from_all_chunks` is able to find all
errors that `LinkedChunkBuilderTest` finds, it's a bingo. Transitively,
it proves that `from_last_chunk` and `insert_new_first_chunk` are
correct!
This patch updates `Update::RemoveChunk` to emit `VectorDiff::Remove`.
Until now, `RemoveChunk` was expecting the chunk to be
empty, because it is how it is used so far. However, with
https://github.com/matrix-org/matrix-rust-sdk/pull/4694, it can change
rapidly.
The code before this patch was doing this:
- look if there's any prev-batch token available right now, aka look if
there's a gap in the in-memory linked chunk
- look at the first chunk; if it's a gap, return to the caller so it
resolves it
The check is done twice at two different levels, which is confusing.
Instead, this patch rewrites it so that the chunk is done only in
`load_more_event_backwards()`.
Note this is also correct for the case storage is disabled; in this
case, we early return and always try to resolve the gap anyways.
This patch renames the `builder` module to `lazy_loader`.
The `LinkedChunkBuilder`'s methods are now functions.
The `LinkedChunkBuilder` struct is removed. Finally,
`LinkedChunkBuilderError` is renamed `LazyLoaderError`.
The `LinkedChunkBuilderTest` struct is kept for the moment. It's going
to be replaced soon.
This patch adds an index on `events.event_id` and on `events.room_id`
so that queries on this column are faster. It mostly happens for the
`Deduplicator`, which runs for every backwards pagination or sync.
This patch also updates the query in `filter_duplicated_events` to
sort event by their `chunk_id` and `position` so that the results are
constant, it helps when testing.
This is the method to get the server metadata in the latest draft of
[MSC2965](https://github.com/matrix-org/matrix-spec-proposals/pull/2965).
We still keep the old behavior with `GET /auth_issuer` as fallback for
now because it has wider server support.
There are some pre-main commit cleanups to simplify the main commit.
This can be reviewed commit by commit.
The changes were tested with the oidc_cli example on beta.matrix.org.
Closes#4550.
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch moves `chunk_debug_string` from `rooms/mod.rs` to
`rooms/event.rs`. In addition, it restores (and rewrites) a test,
initially for `chunk_debug_string`, now for `RoomEvents::debug_string`
whichh is the public API.
This patch changes the semantics regarding what to do in case of
duplicated events received during a backwards pagination.
Previously, the strategy for both sync and backwards pagination was
the same: With the new received events, when duplicated events are
detected, the old events are removed and the new ones are kept.
The strategy is reversed for backwards pagination: the old events are
always kept, and the duplicated events are removed from the new events.
The rest of the patch is about removing dead code because of this
change.
This patch updates `Pagination::run_backwards_impl` to paginate in the
event cache. The flow is now as follow:
- backwards pagination tries to load and to insert a new previous chunk
from the store
- if the new chunk contains events, they are returned, pagination done
- if the new chunk is a gap, the flow continues
- (as previously) check for a prev batch token (it exists in the newly
inserted gap)
- (as previously) run a network request, replace the gap by the new
events
- etc.
The new part is to load and to insert a new previous chunk. The rest
is stays the same. The code has been moved in code to keep the lock
releases happy and to clarify the code.
This patch adds the `RoomEventCacheState::load_more_events_backwards`
method to load a new chunk and to insert it at the beginning of the
`LinkedChunk`.
It uses the new `EventCacheStore::load_previous_chunk` method, along
with the new `LinkedChunkBuilder::insert_new_first_chunk` method.
This patch updates `RoomEventCacheState::new` to load a single chunks
of events instead of all events. It solves bugs where all events were
loaded, while removing the gaps in between, thus the `Timeline` wasn't
able to load the missing events to fill the gaps.
This patch removes a `TODO` in `BackPaginationOutcome`.
Events it contains are deduplicated by the `EventCache` (see
`event_cache::Deduplicator`) when inserted inside `RoomEventCache`.
This patch adds a test for all event cache store
implementations that tests a linked chunk incremental
loading, i.e. the `EventCacheStore::load_last_chunk` and
`EventCacheStore::load_previous_chunk` methods.
Most of the methods on `LinkedChunkBuilder` are now only used
for testing. This patch splits `LinkedChunkBuilder` and creates
`LinkedChunkBuilderTest`. This new type is part of the public
API because it's used in other crates, but it's hidden from the
documentation.
This patch adds two methods on `LinkedChunkBuilder`: `from_last_chunk`
to build a new `LinkedChunk` with a single provided chunk, and
`insert_new_first_chunk` to insert a new chunk at the beginning of the
provided `LinkedChunk`.
This patch renames `RoomEvents::with_initial_chunks` to
`with_initial_linked_chunk`. It avoids a confusion between several
chunks, like `RawChunk`s, and `LinkedChunk` which represents several
`Chunk`s.
This patch update the `EventCacheStore` trait to:
1. rename `reload_linked_chunk` into `load_all_chunks` and put this
method behind `#[cfg(test)]` so that it is removed from the public API,
2. add `load_last_chunk`,
3. add `load_previous_chunk`.
These 2 new methods are implemented inside the `MemoryStore` (with its
real implementation in the `RelationalLinkedChunk`), but `todo!()` are
added for the SQLite implementation.
The format changed 10 months ago and since it contains the tokens, it should have be reserialized already in that time.
Afaict EX clients do not serialize that type, the bindings have their own `Session` type for that.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch introduces `Chunk::lazy_previous` which is a key feature to
support lazy-loading of a `LinkedChunk`. When a chunk is loaded, if it
is the first, it keeps in memory whether it has a previous chunk or not.
Thus, it is possible to insert new chunk in front of the `LinkedChunk`,
and `Update`s will correctly continue to link chunks between them (with
`NewItemsChunk` and `NewGapChunk`).
Example, imagine the following chunks: [0] <-> [1] <-> [2]. If [2] is
the only one being loaded. Then its previous chunk, [1], is loaded from
the store (because [2]'s previous is [1] in the store). Then [1] is
replaced by [3] and [4]. We get this: [4] <-> [3] <-> [1] <-> [2]. If
the `Update::New*Chunk` for [4] doesn't contain a `previous`, the store
is out of sync: in the store, [4] has no previous, but [0] still has [1]
for its `next`.
With this `lazy_previous`, the links are correctly computed.
While implementing user verification on Element X I realized that the
`IdentityStatusChanges` stream does not notify us when an user becomes
`Verified`, which is a shame as it's perfect for powering app updates
after verifying users (i.e. all the decorations we show throughout the
app: room header, room details, room member list, room member profile
etc.)
Had a look at the existing code and, while that seems completely
intentional, there is no reason why we can't expand its purview.
This adds a task to compile the benchmarks in CI, without running them,
and with the lowest level of optimization that's available (the `dev`
profile).
As recommended by BCP 195.
It shouldn't be a problem with rustls that only supports TLS 1.2 and 1.3, but with native-tls it depends on the implementation.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This should be the most common case, and is already the only case
supported by the higher level APIs like `url_for_oidc` and
`login_with_qr_code`. It simplifies the API because we can call
`restore_registered_client` directly from `register_client`, which was a
TODO.
- [x] Public API changes documented in changelogs (optional)
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch adds and implements the
`EventCacheStore::filter_duplicated_events` method. It is implemented on
the `MemoryStore` and the `SqliteEventCacheStore`.
This method remove the unique events and reutrn the duplicated events.
This patch updates `RoomEventCache::subscribe` to be infallible. This
method wasn't able to return something else than an `Ok`. The return
type has been updated from `Result<T>` to `T`.
The `RoomEvents` doesn't hold the `Deduplicator` instance now, it's the
role of the `RoomEventCacheState`. This slightly simplifies the code, in
a few cases.
This patch removes the invariant stating that a `LinkedChunk` must start
by a chunk of type items. This has never been really useful but it's now
annoying to have this (with iterative loading of a `LinkedChunk` via the
`EventCache`, it's now possible to get a gap as the first chunk). Let's
remove this invariant.
We encountered this warning a lot in the logs after upgrading the SDK today.
My understanding is that this path is expected if the event is not yet in the timeline, so it's nothing to warn about.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
A room can be associated to a lot of data, depending on the number of members in the room.
So freeing space on the filesystem should be worth it in some cases.
An (extreme) example: I have a test account that is in ~60 rooms, a few of those big public rooms, including Matrix HQ. The size of the matrix-sdk-state.sqlite3 file is 542 MB. Using this PR and leaving, then forgetting Matrix HQ brings the DB down to 255 MB.
The `EventTimelineItem` would get from a bool to an `Option<bool>`. If
the metadata's `is_room_encrypted` was set to `None`, then it would use
`false` as the value, before wrapping it again into an `Option`.
Since the only reader of `EventTimelineItem::is_room_encrypted()`
doesn't really care about the difference between `Some(false)` and
`None`, in `EventTimelineItem::get_shield()`, we can use a plain `bool`
instead of an `Option`, and not distinguish `Some(false)` from `None`.
At worst, it means that sometimes we don't know the room encryption
status yet, and consider the room unencrypted; as soon as we'll get an
update about encryption state, all the items will be marked as encrypted
anyways, if needs be.
## Some context
An aggregation is an event that relates to another event: for instance,
a
reaction, a poll response, and so on and so forth.
## Some requirements
Because of the sync mechanisms and federation, it can happen that a
related
event is received *before* receiving the event it relates to. Those
events
must be accounted for, stashed somewhere, and reapplied later, if/when
the
related-to event shows up.
In addition to that, a room's event cache can also decide to move events
around, in its own internal representation (likely because it ran into
some
duplicate events, or it managed to decrypt a previously UTD event).
When that happens, a timeline opened on the given room
will see a removal then re-insertion of the given event. If that event
was
the target of aggregations, then those aggregations must be re-applied
when
the given event is reinserted.
## Some solution
To satisfy both requirements, the [`Aggregations`] "manager" object
provided
by this PR will take care of memoizing aggregations, **for the entire
lifetime of the timeline** (or until it's clear'd by some
caller). Aggregations are saved in memory, and have the same lifetime as
that of a timeline. This makes it possible to apply pending aggregations
to cater for the first use case, and to never lose any aggregations in
the
second use case.
## Some points for the reviewer
- I think the most controversial point is that all aggregations are
memoized for the entire lifetime of the timeline. Would that become an
issue, we can get back to some incremental scheme, in the future:
instead of memoizing aggregations for the entire lifetime of the
timeline, we'd attach them to a single timeline item. When that item is
removed, we'd put the aggregations back into a "pending" stash of
aggregations. If the item is reinserted later, we could peek at the
pending stash of aggregations, remove any that's in there, and reapply
them to the reinserted event. This is what the [first version of this
patch](https://github.com/matrix-org/matrix-rust-sdk/pull/4576/commits/ec64b9e0bcc9a2d05dd8c910a8710b72466ef51c)
did, in a much more adhoc way, for reactions only; based on the current
PR, we could do the same in a simpler manner
- while the PR has small commits, they don't quite make sense to review
individually, I'm afraid, as I was trying to find a way to make a
general system that would work not only for reactions, poll responses
and ends. As a matter of fact, the first commits may have introduced
code that is changed in subsequent commits, making the review a bit
hazardous. Happy to have a live reviewing party over Element Call, if
that helps, considering the size of the patch.
- future work may include using the aggregations manager for edits too,
leading to more code removal.
The test ends up with checking that the redact endpoint has been hit
once. It's actually the send queue doing the redaction as a dependent
send request, and it doesn't provide any notification mechanism in this
case, so we can't really know when it's done doing it.
One solution would be to not check the number of calls to the redact/
endpoint, but that means checking for fewer things. Instead, I made it
so that when hit, the endpoint will signal it to the main task using a
oneshot channel; then the main task waits with a long timeout for the
receiving end to get the notification it's been sent, which should be
sufficient.
This would help find test failures specific to experimental-oidc, as
well as doctests failing (which would have prevented the failures fixed
in https://github.com/matrix-org/matrix-rust-sdk/pull/4614 to happen in
the first place).
This patch removes the `Clear` variant of the `RoomEventCacheUpdate`
enum. This one is not needed anymore since we have
`UpdateTimelineEvents` which contains updates as `Vec<VectorDiff<_>>`.
`VectorDiff` _has_ a `Clear` variant. It resulted in a double clear
every time.
This patch updates `RoomEventCacheInner::reset` and
`RoomEventCacheInner::with_events_mut` to annotate them with a
`#[must_use]`. Since they return the updates as `VectorDiff`s,
they **must** be broadcasted/propagated somewhere, likely with
`RoomEventCacheUpdate`. This mechanism ensures to not miss updates.
This patch fixes an issue where 0 was used as the initial value for
the `Skip` higher-order stream in the `TimelineSubscriber`. This is
wrong, as the `SkipCount` value may have been modified before the
`TimelineSubscriber` is created.
This patch provides a test to reproduce the problem.
This patch gathers the logic of the `Timeline::subscribe` into a single
type: `TimelineSubscriber`.
The `TimelineController::subscribe_batched_and_limited` method is
renamed `subscribe` to match `Timeline::subscribe`. Things are simpler
to apprehend.
The `TimelineSubscriber` type configures the subscriber/stream in a
single place. It takes an `&ObservableItems` and a `&SkipCount`, and
configures everything. It also provides a single place to document the
behaviour of the subscriber, with the `Skip` higher-order stream.
This patch updates the pagination mechanism of the `Timeline` to support
lazy backwards pagination.
`Timeline::paginate_backwards` already does different things whether the
timeline focus is live or focused. When it's live, the method will first
try to paginate backwards lazily, by adjusting the `count` value of the
`Skip` stream used by the `Timeline::subscribe` method. If there is not
enough items to provide, the greedy pagination will run.
This patch updates `TimelineStateTransaction::commit` to adjust the
count value of the `Skip` higher-order stream.
This patch also creates `TimelineStateTransaction::new` to simplify the
creation of a state transaction.
This patch applies the `Skip` higher-order stream on the `Timeline`
subscriber. The method `TimelineController::subscribe_batched` is
renamed `subscribe_batched_and_limited` at the same time.
The `Skip` stream uses the `TimelineMetadata::subscriber_skip_count`
observable value as the `count_stream`. The initial count value is 0.
No test needs to be changed so far.
This patch adds the `subscriber_skip_count` field to `TimelineMetadata`.
It's going to be used to define the `count` value of the `Skip`
higher-order stream that is going to be applied to the `Timeline`
subscriber.
This patch adds functions like `compute_next`,
`compute_next_when_paginating_backwards` and
`compute_next_when_paginating_forwards`, which are necessary to
correctly compute the `count` value for the `Skip` higher-order stream.
This patch adds the associated test suite.
This patch renames `TimelineStream` to `TimelineWithDropHandle`, as the
former name was too vague and was not clarify what the type was doing.
The new name makes it clear that it attaches a `TimelineDropHandle` to a
subscriber (since it is part of the `subscriber` module!).
This patch updates `TimelineController::subscribe` to use
`VectorSubscriber::into_values_and_batched_stream`. It returns
the cloned items along with the stream. It saves the need to call
`ObservableItems::clone_items`, thus saving the clone of all items.
tl;dr: `clone_items()` clones… items… and `subscribe()` also clones the
items. There is 2 clones. With `into_values_and_batched_stream()`, there
is 1 clone.
This patch improves the performance of
`ReadReceiptTimelineUpdate::apply`, which does 2 things: it calls
`remove_old_receipt` and `add_new_receipt`. Both of them need an
timeline item position. Until this patch, `rfind_event_by_id` was used
and was the bottleneck. The improvement is twofold as is as follows.
First off, when collecting data to create `ReadReceiptTimelineUpdate`,
the timeline item position can be known ahead of time by using
`EventMeta::timeline_item_index`. This data is not always available, for
example if the timeline item isn't created yet. But let's try to collect
these data if there are some.
Second, inside `ReadReceiptTimelineUpdate::remove_old_receipt`, we use
the timeline item position collected from `EventMeta` if it exists.
Otherwise, let's fallback to a similar `rfind_event_by_id` pattern,
without using intermediate types. It's more straightforward here: we
don't need an `EventTimelineItemWithId`, we only need the position.
Once the position is known, it is stored in `Self` (!), this is the
biggest improvement here. Le't see why.
Finally, inside `ReadReceiptTimelineUpdate::add_new_receipt`, we use
the timeline item position collected from `EventMeta` if it exists,
similarly to what `remove_old_receipt` does. Otherwise, let's fallback
to an iterator to find the position. However, instead of iterating over
**all** items, we can skip the first ones, up to the position of the
timeline item holding the old receipt, so up to the position found by
`remove_old_receipt`.
I'm testing this patch with the `test_lazy_back_pagination` test in
https://github.com/matrix-org/matrix-rust-sdk/pull/4594. With 10_000
events in the sync, the `ReadReceipts::maybe_update_read_receipt` method
was taking 52% of the whole execution time. With this patch, it takes
8.1%.
This patch improves the performance of
`RoomEvents::maybe_apply_new_redaction`. This method deserialises
all the events it receives, entirely. If the event is not an
`m.room.redaction`, then the method returns early. Most of the time,
the event is deserialised for nothing because most events aren't of kind
`m.room.redaction`!
This patch first uses `Raw::get_field("type")` to detect the type of
the event. If it's a `m.room.redaction`, then the event is entirely
deserialized, otherwise the method returns.
When running the `test_lazy_back_pagination` from
https://github.com/matrix-org/matrix-rust-sdk/pull/4594 with 10'000
events, prior to this patch, this method takes 11% of the execution
time. With this patch, this method takes 2.5%.
This patch fixes the performance of
`AllRemoteEvents::increment_all_timeline_item_index_after` and
`decrment_all_timeline_item_index_after`.
It appears that the code was previously iterating over all items. This
is a waste of time. This patch updates the code to iterate over all
items in reverse order:
- if `new_timeline_item_index` is 0, we need to shift all items anyways,
so all items must be traversed, the iterator direction doesn't matter,
- otherwise, it's unlikely we want to traverse all items: the item has
been either inserted or pushed back, so there is no need to traverse
the first items; we can also break the iteration as soon as all
timeline item index after `new_timeline_item_index` has been updated.
I'm testing this patch with the `test_lazy_back_pagination` test in
https://github.com/matrix-org/matrix-rust-sdk/pull/4594. With 10_000
events in the sync, the `ObservableItems::push_back` method (that uses
`AllRemoteEvents::increment_all_timeline_item_index_after`) was taking
7% of the whole execution time. With this patch, it takes 0.7%.
The WAL file can grow depending on the transactions that are run. A
critical case is VACUUM which basically writes the content of the DB
file to the WAL file before writing it back to the DB file.
SQLite doesn't try to reduce the size of the file after that unless we
set an explicit limit,
so we could end up taking twice the size of the database on the
filesystem.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
It should have been done in the migration of version 7, to reduce the
size of the database on the filesystem after the media cache was moved
to the SqliteEventCacheStore. Better late than never.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Fixes#4517.
It turns out that the bugs found in that test were due to 2 causes:
- First commit: `TestRoomDataProvider` didn't use `initial_user_receipts` but returned hardcoded values.
- Second commit: Our own read receipts were ignored in `TimelineStateTransaction::load_read_receipts_for_event`, although we need to process all read receipts via `ReadReceipts::maybe_update_read_receipt` because it knows how to filter out our own read receipts were needed.
This patch drastically improves the performance of
`TimelineEventHandler::deduplicate_local_timeline_item`.
Before this patch, `rfind_event_item` was used to iterate over all
timeline items: for each item in reverse order, if it was an event
timeline item, and if it was a local event timeline item, and if it was
matching the event ID or transaction ID, then a duplicate was found.
The problem is the following: all items are traversed.
However, local event timeline items are always at the back of the items.
Even virtual timeline items are before local event timeline items. Thus,
it is not necessary to traverse all items. It is possible to stop the
iteration as soon as (i) a non event timeline item is met, or (ii) a non
local event timeline item is met.
This patch updates
`TimelineEventHandler::deduplicate_local_timeline_item` to replace to
use of `rfind_event_item` by a custom iterator that stops as soon as a
non event timeline item, or a non local event timeline item, is met, or
—of course— when a local event timeline item is a duplicate.
To do so, [`Iterator::try_fold`] is probably the best companion.
[`Iterator::try_find`] would have been nice, but it is available on
nightlies, not on stable versions of Rust. However, many methods in
`Iterator` are using `try_fold`, like `find` or any other methods that
need to do a “short-circuit”. Anyway, `try_fold` works pretty nice here,
and does exactly what we need.
Our use of `try_fold` is to return a `ControlFlow<Option<(usize,
TimelineItem)>, ()>`. After `try_fold`, we call
`ControlFlow::break_value`, which returns an `Option`. Hence the need
to call `Option::flatten` at the end to get a single `Option` instead of
having an `Option<Option<(usize, TimelineItem)>>`.
I'm testing this patch with the `test_lazy_back_pagination` test in
https://github.com/matrix-org/matrix-rust-sdk/pull/4594. With 10_000
events in the sync, the test was taking 13s to run on my machine. With
this patch, it takes 10s to run. It's a 23% improvement. This
`deduplicate_local_timeline_item` method was taking a large part of the
computation according to the profiler. With this patch, this method is
barely visible in the profiler it is so small.
[`Iterator::try_fold`]: https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.try_fold
[`Iterator::try_find`]: https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.try_find
This patch uses `next_back()` instead of `last()`, which is equivalent
but `last()` requires to iterate over the entire iterator, while
`next_back()` is a single operation.
This patch removes a useless type conversion. The `Room::event()` method
returns a `TimelineEvent`, so calling `Into::into` is useless: we map
`TimelineEvent` to `TimelineEvent`.
This patch removes a useless type conversion. The iterator produces
`TimelineEvent`, so mapping to `TimelineEvent::from` is useless: we map
`TimelineEvent` to `TimelineEvent`.
This test helper is the same as the assert_next_eq helper, but it waits
for the stream to be ready for a certain amount of time instead of
expecting it to be ready right away.
The `SyncService::stop()` method could fail for the following reasons:
1. The supervisor was not properly started up, this is a programmer error.
2. The supervisor task wouldn't shut down and instead it returns a JoinError.
3. We couldn't notify the supervisor task that it should shutdown due the channel being closed.
All of those cases shouldn't ever happen and the supervisor task will be
stopped in all of them.
1. Since there is no supervisor to be stopped, we can safely just log an
error, our tests ensure that a `SyncService::start()` does create a
supervisor.
2. A JoinError can be returned if the task has been cancelled or if the
supervisor task has panicked. Since we never cancel the task, nor
have any panics in the supervisor task, we can assume that this won't
happen.
3. The supervisor task holds on to a reference to the receiving end of
the channel, as long as the task is alive the channel can not be
closed.
In conclusion, it doesn't seem to be useful to forward these error cases
to the user.
This patch moves the creations of the child tasks of the SyncService
into the supervisor tasks itself. This should make it easier to let the
supervisor recreate tasks.
This will become useful once we introduce a offline mode where the
supervisor task becomes responsible to restart syncing once we notice
that the server is back online.
This patch moves `TimelineMetadata`, its implementation and companion
types (like `RelativePosition`) into its own module. The idea is to
reduce the size of the `state.rs` module.
This patch replaces all uses of `TimelineController::add_events_at` by
`TimelineController::handle_remote_events_with_diffs` in the `Timeline`
itself.
The idea is to remove `add_events_at`, as we currently have 2 ways to
add or to handle remote events in the `Timeline`. We want a single one,
because it's simpler!
This patch replaces all uses of `TimelineController::add_events_at` by
`TimelineController::handle_remote_events_with_diffs` in the tests.
The idea is to remove `add_events_at`, as we currently have 2 ways to
add or to handle remote events in the `Timeline`. We want a single one,
because it's simpler!
It is already a 3-tuple and we want to add more data so it will be clearer to use a stuct with named fields.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This will be used as a configuration to decide whether or not to keep
media in the cache, allowing to do periodic cleanups to avoid to have
the size of the media cache grow indefinitely.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Now that the various match branches in the start and stop method of the
SyncService are minimized we can remove the early returns.
This should allow us to more easily add new branches.
The supervisor is defined as two optional fields that are set and
removed at the same time.
This patch converts the two optional fields into a single optional
struct. The fields inside the struct now aren't anymore optional. This
ensures that they are always set and destroyed at the same time.
From cambridge a scheduler is defined as:
> someone whose job is to create or work with schedules
While supervisor is defined as:
> a person whose job is to supervise someone or something
Well ok, that doesn't tell us much, supervise is defined as:
> to watch a person or activity to make certain that everything is done correctly, safely, etc.:
In conclusion, supervising a task is the more common and better
understood terminology here I would say.
Previously we had a lock protecting an empty value, but the logic wants
to protect a bunch of data in the SyncService.
Let's do the usual thing and create a SyncServiceInner which holds the
data and protect that with a lock.
This patch renames `Timeline::subscribe_batched` to
`Timeline::subscribe`. Since the `Timeline::subscribe` method has been
removed because unused, it no longer makes sense to have a “batched”
variant here. Let's simplify things!
This patch simplifies the `Timeline` pagination API as follows:
- a unique `paginate_backwards` method (no more
`focused_paginate_backwards` and `live_paginate_backwards`),
- a unique `paginate_forwards` method (no more
`focused_paginate_forwards`, the `live` variant was absent).
The idea is to unify pagination by hiding the `live` and `focused` mode.
It was already partially the case with `paginate_backards`, but the
`live` and `focused` variants were also present. I believe it creates
an unnecessary confusion.
Firstly, build a `CollectRecipientsResult` as we go, rather than building its
components separately and then assembling it at the end.
Then, factor the common code between the two code paths into a method to update
the `CollectRecipientsResult`.
- supports only text room message types
- enumerates through their body's grapheme clusters and check that every
single one of them is an emoji
- part of the `LazyTimelineItemProvider` so that it can be opt in
`split_devices_for_user` returns a superset of the results of
`split_recipients_withhelds_for_user_based_on_identity`: let's reflect that in
the return types so we can start to share code.
Also, rename `split_recipients_withhelds_for_user_based_on_identity` to
`split_devices_for_user_for_identity_based_strategy` while we are here.
A bounds check was recently relaxed in `imbl`'s `Focus::narrow()`
function: https://github.com/jneem/imbl/pull/89,
which fixed a bug that would cause a panic if the downstream user
of `matrix-sdk-ui` attempted to narrow a focus of Timeline items
using a range that included the last item in the Timeline.
Example: https://github.com/project-robius/robrix/issues/330
This fix has been incorporated in `eyeball-im` and `eyeball-im-util`
and has been tested by me to no longer trigger upon the aforementioned
conditions.
As the comment noted, they're essentially doing the same thing. A
`TimelineEvent` may not have computed push actions, and in that regard
it seemed more correct than `SyncTimelineEvent`, so another commit will
make the field optional.
This patch removes `Timeline::subscribe`. There is
`Timeline::subscribe_batched`, which is the only useful API.
`subscribe` was only used by our tests, and `matrix-sdk-ffi` uses
`subscribe_batched` only.
This patch changes all calls to `Timeline::subscribe` to replace them by
`Timeline::subscribe_batched`. Most of them are in tests. It's the first
step of a plan to remove `Timeline::subscribe`.
The rest of the patch updates all the tests to use
`Timeline::subscribe_batched`.
It became an optional default feature in reqwest 0.12, and we disable
the default features,
so I don't think it was meant to be disabled when the crate was
upgraded.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
It was used in places where we could make use of other helpers, in some
cases. Also introduces the `room_avatar` helper to create the room
avatar state event.
This was a regression introduced in
f0d98602a9.
`latest_edit_json` was first set by the call to
`EventTimelineItem::with_content()`.
It was overwritten in the next section because of the
`if let EventTimelineItemKind::Remote(remote_event)`
that uses `item` instead of `new_item`.
It means that the updated `RemoteEventTimelineItem`
inside`new_item` was replaced by the outdated one inside `item`,
so `latest_edit_json` goes back to its previous value.
I believe that part of why that went unnoticed is that the code looks
more
complicated due to the need to set an inner field, so I decided to
change the API and
move `with_encryption_info` to `EventTimelineItem`, which makes the code
look cleaner.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This change ensures that the user's own live location events are
excluded from the location sharing stream. Since the user's location is
already represented on the map by the blue dot, processing their own
events is redundant and unnecessary.
The others should likely be copied at some point as well, but including
the readme as crate documentation is not currently useful since the
readme for the ui crate is empty, and warning about missing docs or
debug implementations would make CI fail without substantial extra work.
This patch removes a `XXX` (which I believe is a TODO) in the
documentation of `RoomEventCache::subscribe`. We are not going to change
this API anytime soon.
Setting the default log level to `debug` results in logs like:
```
log: log_event
log: latest_event
log: log_event
log: log_event
log: room_info
log: latest_event
log: log_event
log: room_info
```
Presumably they're coming out of the custom tracing configuration and we definitely don't need them.
This is unlikely that it will affect us, so not worth adding a test IMO,
but for the sake of completeness: this handles redacted redactions in
the `AllEventsCache` too.
This is intended to prevent the test
`test_when_user_in_verification_violation_becomes_verified_we_report_it`
flaking. I found that sometimes when it called `Room::members` the
result was empty due to it trying to fetch the answer from the server.
This change prevents that behaviour. I don't know why the behaviour was
inconsistent before.
Regenerate Dan's data with new cross-signing and device keys, for which I know
the private keys.
The signatures are manually calculated for now; this will be improved in a
later commit.
This removes one sync that happens in the background, because it's
likely spurious and may be confusing the server about what's been seen
by the current client.
Add a new created_at to the send_queue_events and
dependent_send_queue_events stored records. This will allow clients to
understand how stale a pending message might be in the event that the
queue encounters and error and becomes wedged.
This change is exposed through the FFI on the `EventTimelineItem` struct
as a new optional field named `local_created_at`. It will be `None` for
any Remote event, and `Some` for Local events (except for those that
were enqueued before the migrations were run).
Signed-off-by: Daniel Salinas
---------
Signed-off-by: Daniel Salinas <zzorba@users.noreply.github.com>
Co-authored-by: Daniel Salinas <danielsalinas@daniels-mbp-2.myfiosgateway.com>
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
Co-authored-by: Daniel Salinas <danielsalinas@Daniels-MBP-2.attlocal.net>
This can be accessed through `fn Room::privacy_settings` and will wrap the functionality related to a room's access and privacy settings.
This commit includes the `fn RoomPrivacySettings::update_canonical_alias` to modify the canonical alias of a room.
As an outsider, I am mostly interested in features and new developments
that have happened, not those that *may* happen. An open-but-not-merged
PR may not get merged in the end, or it may not get merged any time
soon, creating false expectations. Merged PRs, on the other hand, have
definitely happened (even if they get undone, that happens via other PRs
that will get merged later). As such, I think it brings more value to
outsiders.
This fixes the deserialization of the SenderData since it switched to
the base64 encoding for serialization of the master key in one of its
variants.
The issue was introduced in 5ff556f6c3.
Having the final clients define the tracing filters / log levels proved
to be tricky to keep in check resulting missing logs (e.g. recently
introduced modules like the event cache) or unexpected behaviors (e.g.
missing panics because we don't set a global filter). As such we decided
to move the tracing setup and default definitions over to the rust side
and let rust developers have full control over them.
We will now take a general log level and optional extra targets and
apply them on top of the rust side defined defaults. Targets that log
more than the requested by default will remain unchanged while the
others will increase their log levels to match. Certain targets like
`hyper` will be ignored in this step as they're too verbose others
like `matrix_sdk` because they're too generic.
This patch fixes a bug where the code assumes a gap has been inserted,
and thus, is always present. But this isn't the case. If `prev_batch`
is `None`, a gap is not inserted, and so we cannot remove it. This patch
checks that `prev_batch` is `Some(_)`, which means the invariant is
correct, and the code can remove the gap.
This patch removes `SlidingSyncRoomInner::client` because, first
off, it's not `Send`, and second, it's useless. Nobody uses it, it's
basically dead code… annoying dead code… bad dead code!
Sliding sync is no longer experimental. It has a solid MSC4186, along
with a solid implementation inside Synapse. It's time to consider it
mature.
The SDK continues to support the old MSC3575 in addition to MSC4186.
This patch only removes the `experimental-sliding-sync` feature flag.
From now on, this patch considers that `VectorDiff`s are the official
input type for the `Timeline`, via `RoomEventCacheUpdate` (notably
`::UpdateTimelineEvents`).
This patch removes `TimelineSettings::vectordiffs_as_inputs`. It thus
removes all deduplication logics, as it is supposed to be managed by the
`EventCache` itself.
This patch removes the `AddTimelineEvents` variant from
`RoomEventCacheUpdate` since it is replaced by `UpdateTimelineEvents`
which shares `VectorDiff`.
This patch also tests all uses of `UpdateTimelineEvents` in existing
tests.
If it's present, we just let it untouched. Otherwise, we set it to
`error` if it's missing. See code comment explaining why we need this.
This makes sure we log panics at the FFI layer, since the `log-panics`
crate will use the `panic` target at the error level.
Some errors can be handled immediately and don't need a request to be
spawned, e.g. invalid mimetype and so on. The returned task handle still
deals about "late" errors about the upload failing (for sync uploads) or
the send queue failing to push the media upload (for async uploads).
This patch renames `RoomEvents::filter_duplicated_events` to
`collect_valid_and_duplicated_events` as I believe it improves the
understanding of the code. The variables named `unique_events` are
renamed `events` as all (valid) events are returned, not only the unique
ones.
This patch adds the `Pagination` variant to the `EventsOrigin` enum.
Not something really mandatory and that is likely to fix a bug, but it's
now correct.
Since 8205da898e it has been possible to
attach (intentional) mentions to _edited_ media captions, but the
send_$mediatype() timeline APIs provided no way to send them with the
initial event. This fixes that.
Signed-off-by: Joe Groocock <me@frebib.net>
We have quite a few `allow(dead_code)` annotations. While it's OK to use
in situations where the Cargo-feature combination explodes and makes it
hard to reason about when something is actually used or not, in other
situations it can be avoided, and show actual, dead code.
With experimental-sliding-sync enabled and e2e-encryption disabled,
there were a bunch of warnings about unused imports. This fixes them
(but a few warnings about other unused items remain).
I was investigating a potential deadlock with the event cache storage,
and only found a few places where to make the code a bit more idiomatic
and more readable.
`compute_display_name` is made private again, and used only within the
base crate. A new public counterpart `Room::display_name` is introduced,
which returns a cached value for, or computes (and fills in cache) the
display name. This is simpler to use, and likely what most users expect
anyways.
It claimed that it would immediately return when the cached display name
value was computed, but that's absolutely not the case.
Spotted while reviewing a PR updating `iamb` to the latest version of
the SDK.
Instead of keeping on handling unwedged events from the sending queue,
it's now required to re-enable the send queue manually for the room that
encountered the sending error, all the time. This is more consistent,
and avoids weird behavior when a user would 1. send an event for which
sending fails, in an unrecoverable manner, 2. send an event that's
actually sendable.
web-time's Instant type is already used elsewhere in the project. It is
an alias for std's Instant type on most targets, but tries to call into
JavaScript on wasm32-unknown-unknown (assuming that the wasm blob is
used in from a browser context). Its Duration type is a plain re-export
of std's Duration, even on wasm32-unknown-unknown.
Sometimes we can get the bytes directly, e.g. in Fractal we can get an
image from the clipboard. It avoids to have to write the data to a
temporary file only to have the data loaded back in memory by the SDK
right after.
The first commit to accept any type that implements `Into<String>` for
the filename is grouped here because it simplifies slightly the second
commit.
Note that we could also use `AttachmentSource` in the other
`send_attachment` APIs, on `Room` and `RoomSendQueue`, for consistency.
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Every caller to `with_events_mut` must propagate the vector diff
updates, otherwise updates would be missing to the room event cache's
observers. This slightly tweaks the signature to make this a bit
clearer, and adjusts the code comment as well.
`AttachmentConfig::with_thumbnail()` is replaced by
`AttachmentConfig::new().thumbnail()`.
Simplifies the use of `AttachmentConfig`, by avoiding code like:
```rust
let config = if let Some(thumbnail) = thumbnail {
AttachmentConfig::with_thumbnail(thumbnail)
} else {
AttachmentConfig::new()
};
```
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
A previous patch deduplicates the remote events conditionnally. This
patch does the same but for timeline items.
The `Timeline` has its own deduplication algorithm (for remote
events, and for timeline items). The `Timeline` is about to receive
its updates via the `EventCache` which has its own deduplication
mechanism (`matrix_sdk::event_cache::Deduplicator`). To avoid conflicts
between the two, we conditionnally deduplicate timeline items based on
`TimelineSettings::vectordiffs_as_inputs`.
This patch takes the liberty to refactor the deduplication mechanism of
the timeline items to make it explicit with its own methods, so
that it can be re-used for `TimelineItemPosition::At`. A specific
short-circuit was present before, which is no more possible with the
rewrite to a generic mechanism. Consequently, when a local timeline
item becomes a remote timeline item, it was previously updated (via
`ObservableItems::replace`), but now the local timeline item is removed
(via `ObservableItems::remove`), and then the remote timeline item is
inserted (via `ObservableItems::insert`). Depending of whether a virtual
timeline item like a date divider is around, the position of the removal
and the insertion might not be the same (!), which is perfectly fine as
the date divider will be re-computed anyway. The result is exactly the
same, but the `VectorDiff` updates emitted by the `Timeline` are a bit
different (different paths, same result).
This is why this patch needs to update a couple of tests.
This patch adds a trick around `SyncTimelineEvent` to avoid reaching the
`recursion_limit` too quickly. Read the documentation in this patch to
learn more.
The `Timeline` has its own remote event deduplication mechanism. But
we are transitioning to receive updates from the `EventCache` via
`VectorDiff`, which are emitted via `RoomEvents`, which already runs its
own deduplication mechanism (`matrix_sdk::event_cache::Deduplicator`).
Deduplication from the `EventCache` will generate `VectorDiff::Remove`
for example. It can create a conflict with the `Timeline` deduplication
mechanism.
This patch updates the deduplication mechanism from the `Timeline`
when adding or updating remote events to be conditionnal: when
`TimelineSettings::vectordiffs_as_inputs` is enabled, the deduplication
mechanism of the `Timeline` is silent, it does nothing, otherwise it
runs.
This patch adds the `AllRemoteEvents::range` method. This
is going to be useful to support `VectorDiff::Insert` inside
`TimelineStateTransaction::handle_remote_events_with_diffs`.
This patch adds the `ObservavbleItems::insert_remote_event` method.
This is going to be useful to implement `VectorDiff::Insert` inside
`TimelineStateTransaction::handle_remote_events_with_diffs`.
This patch adds a new variant to `RoomEventCacheUpdate`, namely
`UpdateTimelineEvents. It's going to replace `AddTimelineEvents` soon
once it's stable enough. This is a transition. They are read by the
`Timeline` if and only if `TimelineSettings::vectordiffs_as_inputs` is
turned on.
This patch adds `with_vectordiffs_as_inputs` on `TimelineBuilder` and
`vectordiffs_as_inputs` on `TimelineSettings`. This new flag allows to
transition from one system to another for the `Timeline`, when enabled,
the `Timeline` will accept `VectorDiff<SyncTimelineEvent>` for the
inputs instead of `Vec<SyncTimelineEvent>`.
Recently we started to differentiate between rooms we've been banned
from from rooms we have left on our own.
Sadly the non-left rooms matcher only checked if the room state is not
equal to the Left state. This then accidentally moved all the banned
rooms to be considered as non-left.
We replace the single if expression with a match and list all the
states, this way we're going to be notified by the compiler that we need
to consider any new states we add in the future.
When starting to back-paginate, in this test, we:
- either have a previous-batch token, that points to the first event
*before* the message was sent,
- or have no previous-batch token, because we stopped sync before
receiving the first sync result.
Because of the behavior introduced in 944a9220, we don't restart
back-paginating from the end, if we've reached the start. Now, if we are
in the case described by the first bullet item, then we may backpaginate
until the start of the room, and stop then, because we've back-paginated
all events. And so we'll never see the message sent by Alice after we
stopped sync'ing.
One solution to get to the desired state is to clear the internal state
of the room event cache, thus deleting the previous-batch token, thus
causing the situation described in the second bullet item. This achieves
what we want, that is, back-paginating from the end of the timeline.
This cleanup task will run while the knock request subscription runs and will use the `Room::room_member_updates_sender` notification to call `Room::remove_outdated_seen_knock_requests_ids` and remove outdated seen knock request ids automatically.
This will check the current seen knock request ids against the room members related to them and will remove those seen ids for members which are no longer in knock state or come from an outdated knock member event.
This sender will notify receivers when new room members are received: this can happen either when reloading the full room member list from the HS or when new member events arrive during a sync.
The sender will emit a `RoomMembersUpdate`, which can be either a full reload or a partial one, including the user ids of the members that changed.
The code would use a chunk iterator that moves forward, but call
`push_front()` repetitively on each chunk, semantically storing the
lengths in *reverse* order.
This could result in subsequent panics, when a new chunk was added,
because the links would not match what's expected (e.g. the last chunk
must have no successor, etc.).
The previous code would skip based on the position's index, but not the
position's chunk. It could be that the position's chunk is different
from the first items chunk, as shown in the example, where the linked
chunk ends with a gap; in this case, the position's index would be 0,
while the first chunk found while iterating backwards had 3 items. As a
result, items 'd' and 'e' would be skipped incorrectly.
The fix is to take into account the chunk id when skipping over items.
This is needed to tell apart rooms in left and banned state in places like `RoomInfo` or `RoomPreview`.
The banned rooms will still count as left rooms in the sync processes.
This subscription will combine 3 streams: one notifying the members in the room have changed, another notifying the seen join requests have changed, and finally a third one notifying when the room members are no longer synced.
With this info we can track when we need to generate a new list of join requests to be emitted so the client can always have an up to date list.
This will allow us to keep track of which join room requests are marked as 'seen' by the current user and return them as such.
Also, add some methods to `Room` to mark new join requests as seen and to get the current ids for the seen join requests.
The conditions required to cause the bug might have been impossible to
reach in the real world, because it assumes a mix of:
- events present in the linked chunk
- no prev-batch token
However: now that we have storage, we could end up in this situation,
when reaching the start of the timeline (since there'll be no previous
gap in that case). We need to handle that better in the linked chunk
representation itself, but in the meanwhile, we should insert the gap
and the events in a relative correct order.
If a linked chunk update starts with a RemoveChunk update, then the
transaction may start with a SELECT query and be considered a read
transaction. Soon enough, it will be upgraded into a write transaction,
because of the next UPDATE/DELETE operations that happen thereafter. If
there's another write transaction already happening, this may result in
a SQLITE_BUSY error, according to
https://www.sqlite.org/lang_transaction.html#deferred_immediate_and_exclusive_transactions
One solution is to always start the transaction in immediate mode. This
may also fail with SQLITE_BUSY according to the documentation, but it's
unclear whether it will happen in general, since we're using WAL mode
too. Let's try it out.
This patch fixes an edge case where the member is alone in the room with
a service member. We already subtracted the number of service members
in the case we calculated the room summary ourselves, but we did not do
so when the server provided the room summary.
This lead to the room, instead of being called `Empty`, being called
`Foo and N others`.
Update the workaround for https://github.com/rust-lang/rust/issues/109717 to
avoid hardcoding the clang version; instead, run `clang -dumpversion` to figure
it out.
While we're there, use the `CC_x86_64-linux-android` env var, which should
point to clang, rather than relying on `ANDROID_NDK_HOME` to be set.
The `MemoryStore` implementation of the `StateStore` has grown into a
monster, with one lock per field. It's probably overkill, as individual
fields don't need fine-grained locks like this; after all, accesses to
the store shouldn't be reentrant in general.
Fixes#3720.
- rename RawLinkedChunk -> RawChunk
- rename RawChunk::id -> RawChunk::identifier
- precise that a `RawChunk` is mostly a `Chunk` with different
previous/next links.
And let the caller rebuild the linked chunk. This is slightly nicer in
that it allows us to display the raw representation of a reloaded linked
chunk, before checking its internal state is consistent; this will allow
for better debug of issues related to the linked chunk internal state.
No functional changes.
The `MediaFormat` reflects only the request that would be made to the homeserver.
There is no link between the format of the files stored in the media cache and their purpose in an event.
`MediaFormat::Tumbnail` means that we request a server-generated thumbnail of a file in the media repository.
Since the thumbnail is its own file in the media repository, it makes more sense to use `MediaFormat::File`.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
By default, the event cache store will be disabled. If disabled, it will
clean all the events in the cache store; most of the time this will do
nothing, since the store will not even be filled with any event data, so
it would be cheap to do. If some data was filled in the cache store
before, then it would be cleared after the cache store has been
disabled.
This makes a less awkward API than the previous one, where `None` and
`Some(false)` carried different semantics.
This patch creates `ObservableItemsTransactionEntry` that mimics
`ObservableVectorTransactionEntry`. The differences are `set` is
renamed `replace`, and `remove` is unsafe (because I failed to update
`AllRemoteEvents` in this method due to the borrow checker).
This patch creates `ObservableItemsEntries` and particularly
`ObservableItemsEntry` that wraps the equivalent
`ObservableVectorEntries` and `ObservableVectorEntry` with the
noticeable difference that `ObservableItemsEntry` does **not** expose
the `remove` method. It only exposes `replace` (which is a renaming
of `set`).
This patch moves `AllRemoteEvents` inside `observable_items` so that
more methods can be made private, which reduces the risk of misuses
of this API. In particular, the following methods are now strictly
private:
- `clear`
- `push_front`
- `push_back`
- `remove`
- `timeline_item_has_been_inserted_at`
- `timeline_item_has_been_removed_at`
In fact, now, all `&mut self` method (except `get_by_event_id_mut`) are
now strictly private!
This patch adds the Room::send_raw method to the bindings, making it usable from
e.g. Swift.
Signed-off-by: Jonas Richard Richter <jonas-richard.richter@telekom.de>
Whenever it needs to back-paginate, the event cache should start with
the *most recent* backpagination token, not the oldest one.
This isn't a functional change, until the persistent storage is enabled.
The reason is that, currently, there is one previous-batch token alive;
after it's used, it's replaced with another gap and the events it served
to request from the server.
When persistent storage will be enabled, we'll have situations like the
one shown in the test code, where we can have multiple previous-batch
token alive at the same time. In that case, we'll need to back-paginate
from the most recent events to the least recent events, and not the
other way around, or we'll have holes in the timeline that won't be
filled until we got to the start of the timeline.
Extracted from #4329. This does not change the `MediaFormat` of the
request used in the media cache by the send queue.
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
It is triggered by the `xshell::cmd!` macro, and is fixed in xshell 0.2.7, which we cannot upgrade to.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
The test requires subtle conditions to trigger:
- initialize a timeline from a room-list-service's room
- start a backpagination with that timeline (so the room event cache's
paginator is busy)
- try to initialize another timeline with the same room-list-service's
room (e.g. because the first room has been closed, and the app using it
doesn't have a room cache)
This would fail, because initializing a timeline calls
`EventCache::add_initial_events()` all the time, which tries to reset
the paginator's state, which assumes the paginator's not paginating at
this point. In a soon future, we'll get rid of the
`add_initial_events()` function because the event cache will handle its
own persistent storage; in the meantime, a correct fix is to skip
`add_initial_events()` if there was already something in the linked
chunk. After all, we're likely to fill the initial events with the same
events all the time, or a subset of more recent events. By doing that,
we're likely keeping *more* events in the linked chunk, instead.
Thanks to @stefanceriu for reporting the issue and confirming the fix
works!
Currently the WidgetDriver just returns unspecified error strings to the
widget that can be used to display an issue description to the user. It
is not helpful to run code like a retry or other error mitigation logic.
Here it is proposed to add standardized errors for issues that every
widget driver implementation can run into (all matrix cs api errors):
https://github.com/matrix-org/matrix-spec-proposals/pull/2762#discussion_r1838804895
This PR forwards the errors that occur during the widget processing to
the widget in the correct format.
NOTE:
It does not include request Url and http Headers. See also:
https://github.com/matrix-org/matrix-spec-proposals/pull/2762#discussion_r1839802292
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
Adds new UtdCause variants for withheld keys, enabling applications to display customised messages when an Unable-To-Decrypt message is expected.
refactor(crypto): Move WithheldCode from crypto to common crate
Since xshell 0.2.3 the behavior of the run() function has changed in a
incompatible manner. Namely the stdin for the run() function no longer
inherits stdin from the shell. This makes it impossible for commands
that are executed by the run() function to accept input from the shell.
We don't use this functionality in many places but the `xtask release
prepare` command is now broken.
Let's just pin xshell to a working version while we wait for this to be
resolved upstream.
Upstream-issue: https://github.com/matklad/xshell/issues/63
Virtual timeline items will still be provided and the `default_event_filter` will be applied before everything else.
Instances of these timelines will be used to power the 2 different tabs shown on the new media browser. The client will be responsible for interacting with it similar to a normal timeline and transforming its data into something renderable e.g. section by date separators (which will be made configurable in a follow up PR)
Having a mutable iterator can be dangerous and is probably too generic
regarding the safety we want to add around the `AllRemoteEvents` type.
This patch removes `iter_mut` and replaces it by its only use case:
`get_by_event_id_mut`.
This patch replaces `VecDeque<EventMeta>` by `AllRemoteEvents` which
is a wrapper type around `VecDeque<EventMeta>`, but this new type aims
at adding semantics API rather than a generic API. It also helps to
isolate the use of these values and to know precisely when and how they
are used.
As a first step, `AllRemoteEvents` implements a generic API to not break
the existing code. Next patches will revisit that a little bit step
by step.
The `add_or_update_remote_event` method always returns `true`. This
patch updates the method to return nothing, and cleans up the call sites
accordingly. This patch also adds comments to clarify the code flow.
The bool value returned by `add_or_update_remote_event` was supposed
to be `false` if the event was duplicated. First off, as soon as the
`Timeline` receives its events from the `EventCache` via `VectorDiff`,
the `event_cache::Deduplicator` will take care of deduplication,
so the `Timeline` won't have to handle that itself. Second,
`add_or_update_remote_event` was sometimes removing an event, but it
was re-inserting a new one immediately without returning `false`: it was
never returning `false` because a new event was always added.
This test was checking that a new device which has access to backups
returned an unknown UTD when it was given empty JSON for the event. It
was only passing because we currently have incorrect behaviour when
backups are enabled.
The fix is to make the device old and without access to backups, so that
we still consider this UTD unexpected.
Any `.remove` operation called on a `ObservableMap` did not re-index
`values` _after_ the removed position, which means any later operation
on elements inserted after the previously removed key would either be
fetched wrongly, or panic when using `.get_or_create`.
This PR fixes these two related bugs and adds extra test cases.
---------
Signed-off-by: g@leirbag.net
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
This patch changes `TimelineStateTransaction::add_remote_events_at`
to take an `IntoIterator<Item = Into<SyncTimelineEvent>>`
for `events`. In the current code, it saves one
`iter().map(Into::into).collect::<Vec<_>>()`, but it will save another
one when we will support `VectorDiff`s coming from the `EventCache`.
It also avoids to allocate a vector to pass new events (this mostly
happens in the test, but it can happen in real life).
If no server names are provided for the room summary request and the
room's server name doesn't match the current user's server name, add the
room alias/id server name as a fallback value. This seems to fix room
preview through federation.
Also, when getting a summary for a room list item, if it's an invite
one, add the server name of the inviter's user id as another possible
fallback.
Changelog: Use the inviter's server name and the server name from the
room alias as fallback values for the via parameter when requesting the
room summary from the homeserver.
This required adding support for *reading* out of the event cache, for
the sqlite backend. This paves the way for the next PR (reload from the
cache), and it should also help with testing at the `EventCacheStore`
trait layer some day.
A comment was duplicating (the first trace! that's removed here), and
the second block comment only applied to new items, and was not as
concise as it could be.
This patch unifies the logic for inserting timeline items at `Start`
and `End` positions. Both `TimelineItemPositions` can share the same
implementation, making separate logic unnecessary. Previously, `End`
included a duplicated events check as well, while `Start` did not, leading
to inconsistency.
The changes strictly involve moving and refactoring, with no functional
modifications.
This PR introduces a new variant to `UtdCause` specifically for device-historical messages (`HistoricalMessage`). These messages cannot be decrypted if key storage is inaccessible. Applications can leverage this new variant to provide more informative error messages to users.
This patch updates `LinkedChunk::new_with_update_history` to emit an
`Update::NewItemsChunk` because the first chunk is created and it must
emit an update accordingly.
This patch fixes the `test_echo` test. It was doing the following:
* client sends an event to the server,
* servers acknowledges with the ID `$wWgymRfo7ri1uQx0NXO40vLJ`,
* client syncs and the server returns one event with ID `$7at8sd:localhost`,
* the test expects those 2 events to be the same!, which is incorrect.
The test was working because the transaction IDs are the same, but
that's an abuse of the existing code (the code will change soon, another
patch is coming). Whatever the code does: the connection must be based
on the event ID, not the transaction ID.
Building the docs for xtask spews a bunch of unexpected cfg warnings. As
these warnings come from a macro in a dependency and the docs for xtask
don't exist nor will, let's just not build them with the rest of the
docs.
These are all coming from macro invocations of macros that are defined
in other crates. It's likely a clippy issue. We should try to revert
this the next time we bump the nightly version we're using.
This patch tries to clear confusion around
`TimelineItemPosition::UpdateDecrypted(usize)`: it does contains
a timeline item index. This patch changes to
`TimelineItemPosition::UpdateDecrypted { timeline_item_index: usize }`
It's unused so it's mostly cosmetic, and it's trivial to reimplement
using `linked_chunk.items().count()`; let's do that instead of keeping
the perfect exact count synchronized with the chunks, which pollutes the
code in a few places.
This patch renames `TimelineEnd` into `TimelineNewItemPosition` for
2 reasons:
1. In the following patches, we will introduce a new variant to insert
at a specific index, so the suffix `End` would no longer make sense.
2. It's exactly like `TimelineItemPosition` except that it's used
only and strictly only to add **new** items, which is why we can't use
`TimelineItemPosition` because it contains the `UpdateDecrypted`
variant. This renaming reflects it's only about **new** items.
This patch takes the opportunity to move the `RemoteEventOrigin` inside
`TimelineNewItemPosition` to simplify method signatures. They always
go together.
These two are required to properly compute the room preview of a joined
room:
- m.room.create ends up filling the `room_type` (space or not)
- m.room.history_visibility ends up filling the `is_world_readable`
field.
It seems sensible to assume that if a client is able to generate a thumbnail,
it should be able to get all this information for it too.
A thumbnail with no information is not really useful, as we don't know when it could be used instead of the original image.
Removes `BaseThumbnailInfo`.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch fixes a bug in `AsVector`: when an `Update::Clear` value
is received, `AsVector`'s internal state must be cleared too, i.e. the
`UpdateToVectorDiff::chunks` field should be reset to an initial value!
This patch adds a test to ensure this works as expected.
Ruma doesn't currently validate mxuri's and as such `MediaSource`s passed over FFI can contain invalid/empty URLs. This change introduces a wrapper type around Ruma's and failable transformations so that appropiate actions can be taken beforehand e.g. returning a `TimelineItemContent::FailedToParseMessageLike` or nil-ing out the thumbnail info.
This patch implements `LinkedChunk::clear`. The code from `impl Drop
for LinkedChunk` has been moved inside `Ends::clear`, and replaced by
a simple `self.links.clear()`. In addition, `LinkedChunk::clear` indeed
calls `self.links.clear()` but also resets all fields.
This patch adds the `Clear` variant to `Update`.
This patch updates `AsVector` to emit a `VectorDiff::Clear` on
`Update::Clear`.
Finally, this patch adds the necessary tests.
This patch updates `EventCacheStore::handle_linked_chunk_updates` to
take a `Vec<Update<Item, Gap>>` instead of `&[Update<Item, Gap>]`.
In fact, `linked_chunk::ObservableUpdates::take()` already returns a
`Vec<Update<Item, Gap>>`; we can simply forward this `Vec` up to here
without any further clones.
This patch creates a new `MemoryStoreInner` and moves all fields from
`MemoryStore` into this new type. All locks are removed, but a new lock
is added around `MemoryStoreInner`. That way we have a single lock.
A `RelationalLinkedChunk` is like a `LinkedChunk` but with a relational
layout, similar to what we would have in a database.
This is used by memory stores. The idea is to have a data layout that
is similar for memory stores and for relational database stores, to
represent a `LinkedChunk`.
This type is also designed to receive `Update`. Applying `Update`s
directly on a `LinkedChunk` is not ideal and particularly not trivial
as the `Update`s do _not_ match the internal data layout of the
`LinkedChunk`, they have been designed for storages, like a relational
database for example.
This type is not as performant as `LinkedChunk` (in terms of memory
layout, CPU caches etc.). It is only designed to be used in memory
stores, which are mostly used for test purposes or light usages of the
SDK.
When instantiating a room preview, previously it would try to just check if the room exists locally either as joined, invited, knocked, left, etc., and then retrieve the info we cached about it.
While this seems fine for most cases, it turns out for non-joined rooms, the info we have locally will **always** be the one we received when the invite/knock/leave action took place and it'll never be updated,
so we may have the case where we knock into a room, never receive a response, someone changes the join rule of the room to something else and we'll think about this room as a 'request to join' room until we clear the local cache.
To prevent that, we can only use the local data for joined rooms, which are constantly updated, and try to use the room summary API and other fallbacks for the rest, even if they're rooms known to us.
There's a lock making sure we're not doing multiple refreshes of an OIDC
token at the same time. Unfortunately, this lock could be dropped, if
the task spawned by the inner function was detached.
The lock must be held throughout the entire detached task's lifetime,
which this refactoring ensures, by setting the lock's result after
calling the inner function.
It has the same semantics used when creating a caption (if no formatted
caption is provided, assume a provided caption is markdown and use that
as the formatted caption).
This PR makes audio, file, image and video messages be editable so that
the timeline signals when it is possible to use #4277/#4300 for editing
captions.
The (matrix-sdk-)common crate used the (matrix-sdk-)test crate only to
benefit from the `async_test` proc macro, which is conveniently defined
in another crate.
My goal is to make `EventFactory`, at this point in the commit history,
defined in the main SDK crate, available in the test crate.
`EventFactory` makes use of some types defined in common, so there's a
circular dependency at the moment.
To split this circular dependency, I've changed the common crate to
depend on the test-macro crate directly; now the test crate can depend
on the common crate, and everybody's happy.
test(timeline): add an integration test for sending an attachment
test(timeline): add tests for multiple caption edits and local reaction to a media upload
There has been a recent change on `Decryptor::decrypt_event_impl` causing
the function to return an TimelineEvent of kind unable to decrypt
instead of failing with an error.
The `late_decrypt` detection code was not changed, causing any retry to
mark UTDs as late decrypt.
This patch introduces a struct that normalizes and sanitizes display
names. Display names can be a source of abuse and can contain characters
which might make it hard to distinguish one display name from the other.
This struct attempts to make it easier to protect against such abuse.
Changelog: Introduce a DisplayName struct which normalizes and sanitizes
display names.
Co-authored-by: Denis Kasak <dkasak@termina.org.uk>
There was an implicit relationship that the `being_sent` lock needed to
be taken in order to do non-atomic state store operations. With the
change from this commit, the relationship is now more explicit: to get a
handle to the state store, or being_sent, you have to obtain a
`StoreLockGuard` by locking against the store itself. The `WeakClient`
isn't stored in the QueueStorage data structure itself, so it's the only
way to get a `dyn StateStore` from the `QueueStorage`.
Prior to this patch, the send queue would not maintain the ordering of
sending a media *then* a text, because it would push back a dependent
request graduating into a queued request.
The solution implemented here consists in adding a new priority column
to the send queue, defaulting to 0 for existing events, and use higher
priorities for the media uploads, so they're considered before other
requests.
A high priority is also used for aggregation events that are sent late,
so they're sent as soon as possible, before other subsequent events.
Add the `markdown` feature to the SDK crate, otherwise we can't use `FormattedBody::markdown`.
Refactor the pattern matching into an if, add tests to check its behaviour.
Changelog: For `Timeline::send_*` fns, treat the passed `caption` parameter as markdown and use the HTML generated from it as the `formatted_caption` if there is none.
Changelog: This patch introduces a mechanism similar to
`Client::add_event_handler` and `Client::add_room_event_handler`
but with a reactive programming pattern. This patch adds
`Client::observe_events` and `Client::observe_room_events`.
```rust
// Get an observer.
let observer =
client.observe_events::<SyncRoomMessageEvent, (Room, Vec<Action>)>();
// Subscribe to the observer.
let mut subscriber = observer.subscribe();
// Use the subscriber as a `Stream`.
let (message_event, (room, push_actions)) = subscriber.next().await.unwrap();
```
When calling `observe_events`, one has to specify the type of event
(in the example, `SyncRoomMessageEvent`) and a context (in the example,
`(Room, Vec<Action>)`, respectively for the room and the push actions).
This patch updates `eyeball-im` and `eyeball-im-util` to integrate
https://github.com/jplatte/eyeball/pull/63/. With this new feature, we
can have a single implementation of `ObservableMap` (instead of 2: one
for all targets, one for `wasm32-u-u`). It makes it possible to get
`Client::rooms_stream` available on all targets now.
The caption and filenames were weirdly duplicated in each media content,
when the expected behavior is well defined:
- if there's both a caption and a filename, body := caption, filename is
its own field.
- if there's only a filename, body := filename.
We can remove all duplicated fields, knowing this, and reconstruct the
body based on that information. This should make it clearer to FFI users
which is what, and provide a clearer API when creating the caption and
so on.
This patch continues to simplification of the `matrix_sdk_ffi::Client`.
The constructor can receive a `enable_oidc_refresh_lock: bool` instead
of `cross_process_refresh_lock_id: Option<String>`, which was a copy of
`matrix_sdk::Client::cross_process_store_locks_holder_name`.
Now there is a single boolean to indicate whether
`Oidc::enable_cross_process_refresh_lock` should be called
or not. If it has to be called, it is possible to re-use
`matrix_sdk::Client::cross_process_store_locks_holder_name`. Once
again, there is a single place to read this data, it's not copied over
different semantics.
This patch simplifies a little the `ClientBuilder` API:
* `enable_cross_process_refresh_lock` is removed
* `enable_oidc_refresh_crypto_lock` + `set_session_delegate` must be
used instead.
This patch removes the `process_id` argument from
`EncryptionSyncService::new()` and replaces it by
`Client::cross_process_store_locks_holder_name`. The “process ID” is
set when the `Client` is converted into another `Client` tailore for
notification in `NotificationClient` with `Client::notification_client`
which now has a new `cross_process_store_locks_holder_name` argument.
See the Changelog Section to get the details.
Changelog: `Client::cross_process_store_locks_holder_name` is used everywhere:
- `StoreConfig::new()` now takes a
`cross_process_store_locks_holder_name` argument.
- `StoreConfig` no longer implements `Default`.
- `BaseClient::new()` has been removed.
- `BaseClient::clone_with_in_memory_state_store()` now takes a
`cross_process_store_locks_holder_name` argument.
- `BaseClient` no longer implements `Default`.
- `EventCacheStoreLock::new()` no longer takes a `key` argument.
- `BuilderStoreConfig` no longer has
`cross_process_store_locks_holder_name` field for `Sqlite` and
`IndexedDb`.
This patch adds `ClientInner::cross_process_store_locks_holder_name` and
its public method `Client::cross_process_store_locks_holder_name`. This
patch also adds `ClientBuilder::cross_process_store_locks_holider_name`
to configure this value.
Changelog: Implement proper redact handling in the widget driver.
This allows the Rust SDK widget driver to support widgets that
rely on redacting.
Co-authored-by: Damir Jelić <poljar@termina.org.uk>
Previously this only used the Ruma checks, which only handled the initial `#` char and the domain part. With these changes, the name part is also validated, checking it's lowercase, with no whitespaces and containing only allowed chars, similar to what `DisplayName::to_room_alias_name` does.
Moved the code to the SDK crate so it can be properly tested.
This one is used when caching a thumbnail everywhere, and when
attempting to retrieve it; it gives us a single place where to
coordinate the default `MediaThumbnailSettings` parameters.
The previous values would lead to super large memory allocations, as
observed with `valgrind --tool=massive` on the tiny test added in this
commit:
- for 400 rooms each having 100 events, this led to 540MB of
allocations.
- for 1000 rooms each having 100 events, this led to 1.5GB of
allocations.
This is not acceptable for any kind of devices, especially for mobile
devices which may be more constrained on memory. The bloom filter is an
optimisation to avoid going through events in the room's event list, so
it shouldn't cause a big toll like that; instead, we can reduce the
parameters values given when creating the filters.
With the given parameters, 1000 rooms each having 100 events leads to
1.2MB of allocations.
With this, we get notified of the current verification state almost immediately.
Without it, you may either call it too soon and receive an `Unknown` state or you might have to call `Encryption::wait_for_e2ee_initialization_tasks()` and wait until it's finished to request a valid state value.
This patch introduces `EmptyChunk`, a new enum used to represent whether
empty chunks must be removed/unlink or kept from the `LinkedChunk`. It
is used by `LinkedChunk::remove_item_at`.
Why is it important? For example, imagine the following situation:
- one inserts a single event in a new chunk (possible if a (sliding)
sync is done with `timeline_limit=1`),
- one inserts many events at the position of the previous event,
with one of the new events being a duplicate of the first event
(possible if a (sliding) sync is done with `timeline_limit=10` this
time),
- prior to this patch, the older event was removed, resulting in an
empty chunk, which was removed from the `LinkedChunk`, invalidating
the insertion position!
So, with this patch:
- `RoomEvents::remove_events` does remove empty chunks, but
- `RoomEvents::remove_events_and_update_insert_position` does NOT remove
empty chunks, they are kept in case the position wants to insert in this
same chunk.
This fixes a problem when doing an incremental sync at launch,
where `NotLoaded` event would not be dispatched until data became
available or timeout is reached, leading to app waiting for it.
Changelog: the parameter `room_info_notable_update_receiver` was removed
from `RoomList::entries_with_dynamic_adapters`, since it could be
inferred internally instead.
Notably, make it super clear what parameters are required to create the
attachment type, since the function doesn't consume the whole
`AttachmentConfig` for realz.
Changelog: The send queue will now store a serialized
`QueuedRequestKind` instead of a raw event, which breaks the format.
As a result, all send queues have been emptied.
This makes it possible to have different kinds of *parent key*, to
update a dependent request. A dependent request waits for the parent key
to be set, before it can be acted upon; before, it could only be an
event id, because a dependent request would only wait for an event to be
sent. In a soon future, we're going to support uploading medias as
requests, and some subsequent requests will depend on this, but won't be
able to rely on an event id (since an upload doesn't return an
event/event id).
Since this changes the format of `DependentQueuedRequest`, which is
directly serialized into the state stores, I've also cleared the table,
to not have to migrate the data in there. Dependent requests are
supposed to be transient anyways, so it would be a bug if they were many
of them in the queue.
Since a migration was needed anyways, I've also removed the `rename`
annotations (that supported a previous format) for the
`DependentQueuedRequestKind` enum.
Breaking: `ffi::Client::resolve_room_alias` now returns `Result<Option<ResolvedRoomAlias>, ClientError>` instead of `Result<ResolvedRoomAlias, ClientError>`. This allows the client to match the 3 possible cases:
- The room alias exists.
- The room alias does not exist.
- The function failed internally.
This should take care of a bug that caused pinned events to be incorrectly removed when the new pinned event ids list was based on an empty one if the required state of the room didn't contain any pinned events info
This patch removes `RoomListService::new_with_encryption`. This feature
is not used, not useful since it's best to use `EncryptionSyncService`,
and it can be racy depending on how it's used. To avoid potential errors
and bugs, it's preferable to remove this code.
Changelog: a new optional `via_server` parameter was added to `sdk::RoomDirectorySearch::search`, to specify which homeserver to use for searching rooms. In the FFI layer, this parameter is called `via_server_name`.
This patch uses the new `Deduplicator` type, along with
`LinkeChunk::remove_item_at` to remove duplicated events. When a new
event is received, the older one is removed.
This patch introduces `Deduplicator`, an efficient type to detect
duplicated events in the event cache. It uses a bloom filter, and
decorates a collection of events with `Decoration`, which an enum that
marks whether an event is unique, duplicated or invalid.
Changelog: Renamed all the send-queue related "events" to "requests", so
as to generalize usage of the send queue to not-events (e.g. medias,
redactions, etc.).
In a next commit, the `QueuedEvent` will be renamed to `QueuedRequest`.
This specifies which kind of request we want to send with the send
queue; for now, it can only be an event.
This patch fixes a bug in an optimisation inside `UpdateToVectorDiff`
when an `Update::PushItems` is handled. It can sometimes create
`VectorDiff::Append` instead of a `VectorDiff::Insert`. The tests will
be part of the next patch.
This patch adds an `Event` type alias to `SyncTimelineEvent` to (i) make
the code shorter, (ii) remove some cognitive effort, (iii) make things
more convenient.
This method will return a `RoomPreview` for the provided room id.
Also added `fn RoomPreview::leave()` action to be able to decline
invites or cancel knocks, since there wasn't a
`Client::leave_room_by_id` counterpart as there is for join.
The PR also deprecates `RoomListItem::invited_room`, since we have a
better alternative now.
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
Changelog: Support for preallocated media content URI has been added in
`Media::create_content_uri()`, and uploading the content for such a
preallocated URI is possible with `Media::upload_preallocated()`.
Modify the SendQueue in order to persist the error that cause the event
to fail to send as a `QueueWedgeError`. The `QueueWedgeError` is not a
1:1 mapping for all kinds of errors, but holds variant and information
that the client can react to in order to propose "quick fixes"/solution
before retrying to send.
Fixes https://github.com/matrix-org/matrix-rust-sdk/issues/3973
Also fixes https://github.com/element-hq/element-x-ios/issues/3287
because when a timeline reset occurs the fail to send reason is also
lost.
This PR starts with a refactoring commit
https://github.com/matrix-org/matrix-rust-sdk/commit/e7696003e846b64c41761109f326fd37c4506040
to introduce the new `QueueWedgedError` and move the logic that was in
the ffi layer to convert api errors to SendState error variant. This
`QueueWedgedError` can be directly use in the `SendingFailed` variant
and expose to ffi.
Second commit
https://github.com/matrix-org/matrix-rust-sdk/commit/109c1337465ba7965825fec858fd2a4b8b954611
adds the persistence, `QueuedEvent` now have an optional error field
instead of a `is_weged` boolean. Same for LocalEchoContent::Event.
Adds also Migration for sqlite and indexeddb
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
Changelog: We now persist the error that caused an event to fail to send. The error `QueueWedgeError` contains info that client can use to try to resolve the problem when the error is not automatically retry-able. Some breaking changes occurred in the FFI layer for `timeline::EventSendState`, `SendingFailed` now directly contains the wedge reason enum; use it in place of the removed variant of `EventSendState`.
Before we do any more work here, give this variant a better name
Breaking-Change: `matrix_sdk_crypto::type::events::UtdCause::Membership` has
been renamed to `...::SentBeforeWeJoined`.
The FFI will request a scaled version of the thumbnail by default; let's
use the same cache key when caching the thumbnail after an upload.
Thanks @zecakeh for flagging the issue.
This patch removes the `settings` argument of
`RoomListService::subscribe_to_rooms`. The settings were mostly composed
of:
* `required_state`: now shared with `all_rooms`, so that we are
sure they are synced; except that `m.room.create` is added for
subscriptions.
* `timeline_limit`: now defaults to 20.
This patch thus creates the `DEFAULT_REQUIRED_STATE` and
`DEFAULT_ROOM_SUBSCRIPTION_TIMELINE_LIMIT` constants.
Finally, this patch updates the tests, and updates all usages of
`subscribe_to_rooms`.
This patch adds the `m.room.topic` and `m.room.pinned_events` state
events in the `required_state` of the `all_rooms` sliding sync list of
`RoomListService`.
We can't know which key is going to be used precisely for the thumbnail,
so assume non-animated cropped same-size thumbnail media request.
Changelog: when `SendAttachment::store_in_cache()` is set, the thumbnail
is also cached with a sensible default media request (not animated,
cropped, same dimensions as the uploaded thumbnail).
Make the tests behave the same way as the network code, by returning UTDs
as `TimelineEventKind::UnableToDecrypt` instead of `TimelineEventKind::PlainText`.
There's no need for this API anymore.
Changelog: `Timeline::get_event_timeline_item_by_transaction_id` has
been removed. There's no API that makes use of an `EventTimelineItem`
now, those APIs are using a `TimelineEventItemId` instead.
I feel like the ability to convert straight from a `Raw<AnySyncTimelineEvent>>`
into a `SyncTimelineEvent` is somewhat over-simplified: the two are only
occasionally equivalent, and it's better to be explicit.
Changelog: `SyncTimelineEvent` no longer implements `From<Raw<AnySyncTimelineEvent>>`.
The event cache stores its events in a linked chunk. The linked chunk
supports updates (`ObservableUpdates`) via `LinkedChunk::updates()`.
This `ObservableUpdates` receives all updates that are happening inside
the `LinkedChunk`. An `ObservableUpdates` wraps `UpdatesInner`, which
is the real logic to handle multiple update readers. Each reader has a
unique `ReaderToken`. `UpdatesInner` has a garbage collector that drops
all updates that are read by all readers. And here comes the problem.
A category of readers are `UpdatesSubscriber`, returned by
`ObservableUpdates::subscribe()`. When an `UpdatesSubscriber` is
dropped, its reader token was still alive, thus preventing the garbage
collector to clear all its pending updates: they were kept in memory
for the eternity.
This patch implements `Drop` for `UpdatesSubscriber` to correctly remove
its `ReaderToken` from `UpdatesInner`. This patch also adds a test that
runs multiple subscribers, and when one is dropped, its pending updates
are collected by the garbage collector.
The name wasn't very descriptive, and it's tweaking the content, so
let's do that in place, instead of deferring to another method somewhere
else in the codebase.
The tails of the prepare_attachment_message and
prepare_encrypted_attachment_message were almost the same, with the one
different that they were using different ctors for the `EventContent`
types. In fact, all these `EventContent` types also expose a plain `new`
function that can take in either an encrypted or a plain media source,
so we can commonize the code there.
This allows clients to set custom join rules for a room, as would be needed for the knock-only rooms, or restricted rooms (those that can only be joined if the user is part of some other room or space).
See previous commit for explanations. This makes for a simpler API
anyways.
Changelog: `Timeline::edit` doesn't return a bool anymore to indicate it
couldn't manage the edit in some cases, but will return errors
indicating what the cause of the error is.
I think this can't happen, but the send queue can return an error if a
local echo identified by a transaction id doesn't exist anymore in the
database. The only reason the latter could happen is because the local
echo has been sent, in which case an update to the timeline would be
dispatched, and the timeline item would have morphed into a remote echo
in the meantime. So it's really rare that this would happen, and the
`Timeline::redact()` method doesn't have to return a boolean to indicate
success in general.
Changelog: `Timeline::redact()` doesn't return a boolean; previously, it
would only return false if the internal state was invalid, so a new
error `RedactError::InvalidLocalEchoState` has been introduced to
represent that.
This maintains functionality we had prior to the previous commit: if an
event's missing from the timeline (e.g. timeline's been cleared after a
gappy sync response), then still allow editing it.
In particular, this means that trying to edit an event that's not
present anymore in a timeline (e.g. after a timeline reset) will fail,
while it worked before.
Changelog: `Timeline::edit_by_id` has been fused into `Timeline::edit`,
which now takes a `TimelineEventItemId` as the identifier for the local
or remote item to edit. This also means that editing an event that's not
in the timeline anymore will now fail. Callers should manually create
the edit event's content, and then send it via the send queue; which the
FFI function `Room::edit` does.
Changelog: `Timeline::redact_by_id` has been fused into
`Timeline::redact`, which now takes a `TimelineEventItemId` as an
identifier of the item (local or remote) to redact.
We can now use this type instead of passing a string, which means
there's no way to confuse oneself in methods like
`toggle_reaction_local`.
Changelog: Introduced `TimelineUniqueId`, returned by
`TimelineItem::unique_id()` and serving as an opaque identifier to use
in other methods modifying the timeline item (e.g. `toggle_reaction`).
This would not report the `txn_id` field because of the `skip_all`. It's
actually interesting to also get the error, so I'm only skipping self
from now on.
I suppose these were useful at the FFI layer at some point, but they
aren't anymore, so they could be removed.
Changelog: Got rid of `From<String/&str>` for `TimelineEventItemId`.
This would have avoided a few hours of debugging where we thought there
was an issue with multiple timelines spawned at the same time, and then
realized it was expected because of the existence of the pinned timeline
in EX apps.
Older versions of Git require --interactive when we supply --autosquash,
and it's also probably a good idea generally.
See https://stackoverflow.com/a/77663575/22610 for more info.
Changelog: `Client::url_for_oidc_login` is now `Client::url_for_oidc` with an additional `OidcPrompt` parameter. `abort_oidc_login` has been renamed to `abort_oidc_auth`.
This allows clients to directly open the web page they want: the login one, the registration one, consent, etc. It should improve the UX in the registration flow since we can now skip the login one.
We depend on the `futures_util::steam_select` macro since 9b36a04b. This
macro requires the async-await-macros and std feature of futures-util.
These features are the default features so let's just stop disabling the
default features for futures-util.
Signed-off-by: Damir Jelić <poljar@termina.org.uk>
Co-authored-by: Jonas Platte <jplatte@matrix.org>
This patch updates the `required_state` of `all_rooms` inside the
`RoomListService` to add `m.room.name`. Apparently, Synapse doesn't
always update the `response.rooms.*.avatar` field when the avatar is
updated. It's being investigated, but it doesn't hurt to ensure we get
it from the state events.
To fix the `test_room_avatar_group_conversation`, we need to ask for the
`m.room.avatar` state event from `required_state`. The rest of the patch
rewrites the test a little bit to make it more Rust idiomatic.
The `response.rooms.*.avatar` field from sliding sync should contain the
new avatar, but for the moment, it doesn't. It seems to be a bug.
To fix the `test_left_room`, we need to ask for the `m.room.member`
state event from `required_state`. The rest of the patch rewrites the
test a little bit to make it more Rust idiomatic.
Sliding sync expects all state events to be in `required_state`. State
events in `timeline` **must be ignored**. However, in sync v2, state
events in `timeline` **must be handled**.
In the sync response flow, both sliding sync and sync v2 uses the same
`handle_timeline` method. This patch adds an argument to ignore state
events. This is not ideal, but it's a temporary solution as a first
step. The next step is to refactor this code, but let's start easy.
The rest of the patch updates the tests accordingly.
With sliding sync, we must handle state events from `required_state`
only, not from `timeline`, this is a mistake as they might be incomplete
or _staled_.
I'm going to be replacing the inner structure of `TimelineEvent` with an
implementation that holds a `Raw<AnySyncTimelineEvent>`, rather than a
`Raw<AnyTimelineEvent>`. Prepare for that by changing the accessors to return
`Raw<AnySyncTimelineEvent>`.
... and add accessors instead.
Give `TimelineEvent` the same treatment we just gave `SyncTimelineEvent`: make
the fields private, and use accessors where we previously used direct access.
... and add accessors instead.
I'm going to change the inner structure of `SyncTimelineEvent`, meaning that
access will have to be via an accessor in future. Let's start by making the
fields private, and use accessors where we previously used direct access.
I'm going to change the internal structure of `SyncTimelineEvent`, and since
it implements `Deserialize`, we need to not break it. Let's add a test for the
current format.
`bootstrap_cross_signing` holds a lock on the private identity. In case
a new identity is created, it will try to acquire a lock on `account`.
The latter is locked by `sync`, which tries to acquire a lock on the private identity.
Note that the `bootstrap_cross_signing` call is executed in a separate
task e.g. in `restore_session`. In particular, this task and `sync` both
race to acquire locks described above.
Signed-off-by: boxdot <d@zerovolt.org>
This is useful for the knocking feature since we'll be able to differentiate between rooms that you were just invited from rooms that you knocked and then were granted access, or rooms that you left and rooms where your knocking attempt was rejected.
The `mark_room_as_*` functions have been updated so they reuse the same `set_state` function underneath, which only updates the previous state if the new one doesn't match.
Was added post-merge to the MSC for servers that support
authenticated media but do not support all of Matrix 1.11 yet.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch changes the behaviour of `timeline_limit` in sliding sync
requests. It previously was sticky, but since it's now mandatory
with MSC4186, it's preferable it to be non-sticky, otherwise in
some scenarios it might default to 0 (its default value). How?
If the server doesn't reply with our `txn_id` (because it doesn't
support sticky parameters or because it misses a `txn_id`), the
next request will be built with a default `timeline_limit` value,
which is zero, and won't get updated to the `timeline_limit` value
from `SlidingSyncListStickyParameters`. This is not good. Instead,
we must consider `timeline_limit` as non-sticky, and moves it from
`SlidingSyncListStickyParameters` to `SlidingSyncListInner`. This is
what this patch does.
The content of a poll timeline item goes to the content directory. The
data structure handling pending poll events goes into state.
No functional changes, only code motion.
It can occur that the data contains rooms that were forgotten.
Knowing that we now update that data after every sync, that creates
a lot of noise in the logs.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
The `handle_reaction_redaction` method is called by `handle_redaction`
for every single redaction event that we receive as a first step to
check if the redaction matches a reaction.
It means that not finding a reaction to redact is perfectly fine and is not worthy of a warning.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This old method was checking invariants that were
spooky-action-at-a-distance: these invariants have changed since then,
so this would panic instead of returning a proper error to the caller.
Signed-off-by: oliverw@element.io
---------
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
This was because we used a `BTreeSet`, which doesn't make sense anymore
since the data part of the key got mangled with some value unrelated to
the key itself.
This seems to be the only way to make the log rotation fix work and avoid build warnings like:
```
warning: patch for the non root package will be ignored, specify patch at the workspace root:
package: matrix-rust-sdk/bindings/matrix-sdk-ffi/Cargo.toml
workspace: = matrix-rust-sdk/Cargo.toml
Finished `dev` profile [unoptimized] target(s) in 0.30s
```
This improves parsing times in mobile Clients. On Android, this means a 5-10x faster parsing of timeline events.
To do that I had to:
- Make functions like `edit/redact/forward` take an identifier (EventId/TransactionId) instead of the actual event. This id will be used to look for the actual SDK timeline event in the timeline. This change will make these functions a bit less performant.
- Make `InReplyToDetails` an object instead since a record can't recursively contain itself.
- Turn `EventTimelineItem` into a record type. Do the same with `Message`, which is now `MessageContent`.
These tests were failing since the migration from the sliding sync proxy
to Synapse.
Since the previous fixes to re-enable other tests, these 2 passes for
free.
This test was failing since the migration from the sliding sync proxy
to Synapse.
This patch fixes the test. The failing parts were:
1. The `timeline_limit` wasn't set, so Synapse was returning an error,
2. The `unread_notifications` was set to 0 and could not be set to 1
because that's an encrypted room.
The fact `timeline_limit` is now mandatory has been mentioned in the MSC:
https://github.com/matrix-org/matrix-spec-proposals/pull/4186/files#r1775138458
A patch in Ruma has been created. The previous patch in this repository
also contains the fix for the SDK side.
The assertions around `unread_notifications` have been removed. We no
longer use this API anymore (and it should be deprecated by the way).
Since MSC4186, the `timeline_limit` value is required.
This patch uses 1 as the default value for `timeline_limit`, and forces
the `timeline_limit` to be defined everywhere.
When Bob receives the invite, the room has the correct name. Bob to sync
more to receive the new name. This is not a bug.
This patch updates the `CreateRoomRequest` to set the correct name
immediately.
This test was failing since the migration from the sliding sync proxy
to Synapse.
This patch fixes the test. The failing part was:
```rust
assert_eq!(notification.joined_members_count, 1);
```
This patch changes the value from 1 to 0. Indeed, Synapse doesn't share
this data for the sake of privacy because the room is not joined.
A comment has been made on MSC4186 to precise this behaviour:
https://github.com/matrix-org/matrix-spec-proposals/pull/4186#discussion_r1774775560.
Moreover, this test was asserting a bug (which is alright), but now
a bug report has been made. The patch contains the link to this bug
report.
The code has been a bit rewritten to make it simpler, and more comments
have been added.
Instead of forcing the room encryption to be known when the timeline is created and failing if it's not known, take the latest room encryption info as a base value and update it when processing timeline events.
At the time of writing this commit, the encryption info is only used to decide whether shields should be calculated for timeline items or not.
- explain what these tests are
- mention that it's sometimes needed to rebuild the synapse image
---------
Signed-off-by: Benjamin Bouvier <benjamin@bouvier.cc>
Co-authored-by: Ivan Enderlin <ivan@mnt.io>
This patch updates `merge_stream_and_receiver` to display an `error!`
when the room info receiver reads an error, like `Closed` or `Lagged`.
This is helpful when debugging.
The NotificationClient, responsible for handling, fetching, and
potentially decrypting events received via push notifications, creates a
copy of the main Client object.
During this process, the Client object is adjusted to use an in-memory
state store to prevent concurrency issues from multiple sync loops
attempting to write to the same database.
This copying unintentionally recreated the OlmMachine with fresh data
loaded from the database. If both Client instances were used for syncing
without proper cross-process locking, forks of the vodozemac Account and
Olm Sessions could be created and later persisted to the database.
This behavior can lead to the duplication of one-time keys, cause
sessions to lose their ability to decrypt messages, and result in the
generation of undecryptable messages on the recipient’s side.
This patch adds a new `cancel_in_flight_request` argument to
`SlidingSync::subscribe_to_rooms`, which tells the method cancel the
in-flight request if any.
This patch also updates `RoomListService::subscribe_to_rooms` to turn
this new argument to `true` if the state machine isn't in a “starting”
state.
The problem it's solving is the following:
* some apps starts the room list service
* a first request is sent with `pos = None`
* the server calculates a new session (which can be expensive)
* the app subscribes to a set of rooms
* a second request is immediately sent with `pos = None` again
* the server does possibly NOT cancel its previous calculations, but
starts a new session and its calculations
This is pretty expensive for the server. This patch makes so that the
immediate room subscriptions will be part of the second request, with
the first request not being cancelled.
This method is now private inside `matrix_sdk_ui` and removed
from `matrix_sdk_ffi`. This method is returning a stream of rooms,
but updates on rooms won't update the stream (only new rooms
will be seen on the stream). Nobody uses it as far as I know, and
`entries_with_dynamic_adapters` is the real only API we want people
to use.
This field is helpful as it tells us the sequence number of the message in the
megolm session, which gives us a clue about how long it will have been since
the session should have been shared with us.
This fn will loop until it finds an at least partially synced room with the given id. It uses the `ClientInner::sync_beat` listener to wait until the next check is needed.
This patch adds `m.room.canonical_name` in the `required_state` of the
`all_rooms` list defined by `RoomListService`.
This is useful to better compute the room name in a more robust way.
## Changes
Takes care of [this
TODO](https://github.com/matrix-org/matrix-rust-sdk/blob/9df1c480795c42afcee39e4c7e553d5a927a2680/crates/matrix-sdk-ui/src/timeline/mod.rs#L520).
- sdk & sdk-ui: unify `Timeline::edit` and `Timeline::edit_polls`, the
new fn takes an `EditedContent` parameter now, which includes a
`PollStart` case too.
- ffi: also unify the FFI fns there, using `PollStart` and a new
`EditContent` enum that must be passed from the clients like:
```kotlin
val messageContent = MessageEventContent.from(...)
timeline.edit(event, EditedContent.RoomMessage(messageContent))
```
Since the is mainly about changing the fns signatures I've reused the
existing tests, including one that used `edit_poll` that now uses the
new fn.
---------
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
There is a non-negligible difference MSC3575 and MSC4186 in how the
`e2ee` extension works. When the client sends a request with no `pos`:
* MSC3575 returns all device lists updates since the last request
from the device that asked for device lists (this works similarly to
to-device message handling),
* MSC4186 returns no device lists updates, as it only returns changes
since the provided `pos` (which is `null` in this case); this is in
line with sync v2.
Therefore, with MSC4186, the device list cache must be marked as to be
re-downloaded if the `since` token is `None`, otherwise it's easy to
miss device lists updates that happened between the previous request and
the new “initial” request.
This patch adds the `OldMachine::mark_all_tracked_users_as_dirty`.
This patch rewrites a bit `OlmMachine::new_helper` by extracting some
piece of it inside `OlmMachine::new_helper_prelude`. With that, we
can rewrite `OlmMachine::migration_post_verified_latch_support` to use
`IdentityManager::mark_all_tracked_users_as_dirty`.
This latter is the shared implementation with
`OlmMachine::mark_all_tracked_users_as_dirty`.
This patch adds a test for `OlmMachine:mark_all_tracked_users_as_dirty`.
This patch improves `Client::available_sliding_sync_versions` when
trying to detect the sliding sync proxy. Previously, we were relying
on the `Client::server` to send the `discover_homeserver::Request`.
Sadly, this value is an `Option<_>`, meaning it's not always defined
(it depends how the `Client` has been built with `HomeserverConfig`:
sometimes the homeserver URL is passed directly, so the server cannot
be known).
This patch tries to find to discover the homeserver by using
`Client::server` if it exists, like before, but it also tries by using
`Client::user_id`. Another problem arises then: the user ID indeed
contains a server name, but we don't know whether it's behind HTTPS or
HTTP. Thus, this patch tries both: it starts by testing with `https://`
and then fallbacks to `http://`.
A test has been added accordingly.
The term session is usually only used in the crypto crate to reference a
Megolm session, the rest of the SDK uses the name from the event and the
Matrix spec, this should lower the amount of confusion since the main
crate has already a session concept and its unrelated to end-to-end
encryption.
My theory is that the intermittent failure depends on the ordering of
the requests, and if the /keys/upload request happened before the key
backup request, then after failing the next key backup request wouldn't
run.
This is likely a small typo that the key upload returns a 404 error
instead of a 200, let's see if this improves the situation.
It does not make much sense to create an UI client that does not support
end-to-end encryption, besides disabling the feature was broken for
quite some time.
* Not verifying the remote device/user is normal: log it at debug rather than
info.
* On the other hand, if we do verify something, let's log that at info rather
than trace.
Also fix a comment, while we're here.
* Upgrade the log when we get the "reciprocate" message (which tells us the
other side has scanned our QR code) to debug, instead of trace.
* Warn if we get a reciprocate we don't understand
* Log when the user confirms that the other side has scanned successfully.
This is the message that tells us whether the other side wants to do QR code or
SAS (emoji) verification. Knowing which they have chosen is really helpful for
following the flow!
Attach the flow_id (the transaction ID or message ID from the `request`
message) to the span, so that it is displayed alongside loglines that happen
when processing the request.
Take the logging that happens when a QR code verification is added to the
`verification cache`, and push it down to the `VerificationCache` itself. Doing
so means that we will log when we *show* a QR code as well as when we scan it.
I would have found this helpful when trying to debug a verification flow this
week.
For debugging, it's useful to have a record of what we believe our own public
cross-signing keys to be. Currently, we log the keys at startup if we restore
them from the database, but if we subsequently create, or download, a set of
keys, they aren't logged.
When we receive an `/keys/query` response, look for existing
inboundgroupsessions created by updated devices, and see if we can update any
of their senderdata settings.
This module has a number of useful types (in particular, error types). Rather
than addding even more types to the top level module, let's export the
`sender_data_finder` module as a whole.
Turns out that most of the places we call `find_using_device_keys`, we already
have a DeviceData. So we might as well pass that in directly, rather than
extracting the device keys and then rebuilding a DeviceData.
This patch renames `HomeserverConfig::Url` to `HomeserverUrl`, and
`HomeserverConfig::ServerNameOrUrl` to `ServerNameOrHomeserverUrl`.
Funnily, the methods on `ClientBuilder` doesn't need to be renamed to
match the new naming since they already use this naming!
When subscribing to an already subscribed room, the subscription state
was reset, the subscription was resent by cancelling the in-flight
request. All this is useless and can create a feeling of lag.
This patch checks if a room is subscribed first. If it is, nothing
happens. If it is not, the room subscription is created, and the
in-flight request is cancelled.
Tests are updated to reflect this new change.
Note: room subscription settings are not taken into account in this
“presence” pattern. It means that if we subscribe to an already
subscribed room but with different settings, nothing will happen. The
server will ignore the new settings anywhere for the moment.
This patch moves `client/builder.rs` into `client/builder/mod.rs`.
This patch also moves the `HomeserverConfig` type and its siblings
(`discover_homeserver`, `UrlScheme` etc.) into its own module `client/
builder/homeserver_config.rs`.
This is purely for cleaning up.
This patch adds `Client::server`: the URL of the server.
Not to be confused with the `Client::homeserver`. The `server`
holds some information, like the `.well-known` file, to discover the
homeserver. The homeserver is the client-server Matrix API.
`server` is usually the server part in a user ID, e.g. with
`@mnt_io:matrix.org`, here `matrix.org` is the server, whilst
`matrix-client.matrix.org` is the homeserver.
This patch also moves the code about homeserver discovery in the
`HomeserverConfig::discover` new method. A new struct is introduced to
hold the result, to replace a 4-tuples.
`Client::server` is also now a `Option<_>` because in the case of
`HomeserverConfig::Url`, the server cannot be known.
This patch also removes several clones here and there.
Finally, this patch updates a test to quickly test the new behaviour. A
next patch will introduce proper tests.
Especially for remote items, it should be in sync with `should_add` as
it's used in this method, otherwise read receipt tracking will not work
correctly.
This will only apply to `async_test` functions, but I think this is a
win:
1. for consistency within the codebase, since I've started doing so in
many places,
2. because these function names will clearly identify these functions as
tests, in the call tree interfaces, when rendered using the LSP
show-callers/show-callees functionality.
Including non-live timelines (pinned event timelines and permalinked
timelines). This makes it possible to see that you're adding a reaction
etc. in real time, while it wasn't the case anymore.
Fixes#3906.
This was copied from SqliteStateStore, but the reason
that they are separated there is because some migrations
require the store cipher.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Setting the version number only when all migrations are done
means that the version will be wrong if a migration fails.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Otherwise, this would mean that logged out clients would infinitely
repeat network requests failing in the background.
Without this fix, the added test will time out, endlessly reattempting
network requests.
This patch adds bindings to `Client::available_sliding_sync_versions`
to `matrix-sdk-ffi`.
This patch also moves `Client::sliding_sync_version` from “private” to
“public” FFI API, in thee sense that this method is now exported with
UniFFI.
Previous patches have unified all sliding sync versions behind a
single type: `Version`. More recent previous patches have introduced
`VersionBuilder` so that a `ClientBuilder` can use them to coerce or
find the best `Version` possible. This patch implements a last missing
piece: `Client::available_sliding_sync_versions` will report all
available `Version`s at a given time. This is useful when a `Client`
is already built, and a session has been opened/a user is logged,
but someone has to take the decision whether it's useful to switch to
another sliding sync version or not.
This patch adds a builder for `sliding_sync::Version`. It is a similar
enum except that it has `DiscoverProxy` and `DiscoverNative` to
automatically configure `Version::Proxy` or `Version::Native`.
This patch changes the type of
`discover_homeserver_from_server_name_or_url`. It now returns a `Url`
instead of a `String` for the homeserver URL. It also returns an
`Option<get_supported_versions::Response>` in addition to the other
values.
The change from `String` to `Url` is necessary to avoid a
double-parsing. It was parsed in `build()` but previously in
`discover_homeserver_from_server_name_or_url` in the last branch.
The addition of `get_supported_versions::Response` is necessary for the
next patch. It's going to be helpful to auto-discover sliding sync in
Synapse. The change happens here because
`get_supported_versions::Response` is already received in
`discover_homeserver_from_server_name_or_url`. This patch makes it easy
to re-use it so that the request is sent only once.
This patch therefore changes `check_is_homeserver` a little bit to
become `get_supported_versions`, and inlines its previous call inside
`discover_homeserver_from_server_name_or_url`.
This patch replaces all the API using simplified sliding sync, or
sliding sync proxy, by a unified `sliding_sync::Version` type!
This patch disables auto-discovery for the moment. It will be re-enable
with the next patches.
If we're building for the wasm architecture, jump through the hoops to
tell wasm_bindgen about `ShieldStateCode`. This solves the need to
declare an identical copy of `ShieldStateCode` in the wasm bindings.
This is part of https://github.com/matrix-org/matrix-rust-sdk/pull/3662,
pulled out to into a separate PR. Recent changes in `main` made it
pretty much impossible to merge this section of code from `main` into
that PR, and Rich wanted to see the refactoring bits separate from the
behavioural changes. So I've re-written the refactoring.
Pulls the `match` on `sharing_strategy` outside of the `for` loop, and
moves any code that is specific to one strategy into the appropriate
branch.
This commit contains a new `assert_next_matches_with_timeout!` macro that will take a `Stream` and wait for a little while until its next item is ready, then match it to a pattern.
It was found that it's making the whole build slower because it's
hitting a pathologically slow path in type-checking. Considering that it
doesn't do much, let's get rid of it and inline it instead.
After this, compiles times are reduced from 30 seconds to 22 seconds on
my machine
This patch uses the new `RoomSubscriptionState` enum to filter
room subscriptions that have already been sent, when building a
`http::Request` with the sticky parameters.
By default, to start, a `http::request::RoomSubscription` is in the
state `Pending`, i.e. it's not sent yet. Once the sticky parameters are
committed, the state is updated to `Applied`. When the sticky parameters
are applied, only the `Pending` room subscriptions are added.
This patch contains one test to specifically assert this behaviour.
This patch adds the `commit` method on the `StickyData` trait. It is
called by `SlidingSyncStickyManager::maybe_commit` when we are sure the
data can be validated because of a valid response to the sent request.
## Changes
- Creates a separate `AllEventsCache` struct holding both the actual all
events cache map and the relationship one. This new cache wrapper also
holds a single `RwLock` to both inner caches, as they are independent
from each other.
- When a new event is saved either from `/sync` or another HS request,
it'll be saved to both the 'all events' cache and the relationship map.
- Add tests for the relationship cache.
Allows to save media in a different path than the state store.
This adds a "last_access" field to the SQLite implementation, to prepare
for future work on a media retention policy.
This removes the IndexedDB media cache implementation, because as far as
I know it is currently unused, and I have no idea how to implement
efficiently the planned media retention policy with a key-value store.
Closes#1810.
- [x] Public API changes documented in changelogs (optional)
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch clears all sliding sync room subscriptions when a session
expires. Indeed, we might not want to request all room subscriptions
when the session restarts. Imagine if the client has subscribed to 400
rooms and the session expires: once the session restarts, it will ask
for 400 room subscriptions, which is a lot and will result in a quite
slow response.
We're finding ourselves in the situation in which we can't interact with
invites through normal Room APIs as `full_room`s can't be build from the
RoomListItem. Full rooms require the timeline to be configured before
use and the timeline can't be configured because encryption cannot be
fetched for invited rooms on homeservers that have previews disabled
(see #3848 and #3850)
In response we now expose the room's membership directly from the
`RoomListItem` so that the final client can chose which of the 2 rooms
types (invited or full) to ask for before using aforementioned APIs.
Powers https://github.com/element-hq/element-x-ios/pull/3189
This patch updates when sliding sync requests have a `timeout`.
Prior to this patch, all requests had a `timeout` query, set to the
`poll_timeout` duration value. However it means: if there is no data
to return, wait `timeout` milliseconds for new data before returning.
This definition is correct. Problem: if the current range of a list
has no data, the server will wait! It means that, in a situation where
there is no update at all, but the client is fetching all rooms batch by
batch, it will wait `poll_timeout` for each batch!
The behaviour described above is absolutely correct. Some server
implementations are less strict though, and we didn't realise our code
was doing that, because the server had some optimisations to ignore the
timeout if the range wasn't covering all the rooms. Nonetheless, a new
server implementation (namely Synapse) is strict, and it confirms we
have a bug here.
This patch then configures a `timeout` if all lists require a timeout,
otherwise there is no `timeout`, which is equivalent to `timeout=0`.
- this prevents issues where spoofing the sender field is enough to spoof and edit and display wrong decorations in the app
- fixes matrix-org/internal-config/issues/1549
Most times we pulled an InboundGroupSession from the sqlite DB, we were
overriding whatever value for `backed_up` was stored inside the pickled
value, and using the value stored in the SQL column.
But when we pulled a single InboundGroupSession from the DB by ID, we
did not override it.
I am fairly sure this was an accidental oversight, so this change
corrects it, and unifies the code with other places we create these
objects.
It seems that the fact that matrix-rust-sdk contains `olm-rs` in its
`Cargo.lock` sparked panic, so let's attempt to fend off future concerns by
adding a comment.
This patch fixes a bug in the tracing system. It introduces one
fields formatter _per layer_ to force the fields to be recorded in
different span extensions, and thus to remove the duplicated fields in
`FormattedFields`.
The patch contains links to the bug report in `tokio-rs/tracing`. This
patch is a workaround.
This patch removes `NotificationSettings::subscribe_to_changes` because
it's not used anywhere in our code except in tests. It is indeed part of
the public API but I'm not aware of anyone using it for the moment. It
only adds complexity in the code.
This patch replaces `RoomInfo::user_defined_notification_mode` by
its cached variant: `cached_user_defined_notification_mode`, and
call `Room::cached_user_defined_notification_mode` which will boost
performance when computing a new `RoomInfo`.
This patch splits/copies the
`test_room_info_deserialization_without_optional_items` test into the
same test + `test_room_info_deserialization`.
It appears that the _without optional items_ part has been forgotten.
In the past, the test has been updated to test optional items. The
initial idea (based on my understanding of the comments) is to test
potentially old `RoomInfo` can still be deserialized today. So this test
must never be changed, except if a non-optional field is added.
For this reason, this patch removes from this test the assertions about
optional fields. A new test, named `test_room_info_deserialization` is
created, and tests all the fields, including the optional ones.
One important thing:
`test_room_info_deserialization_without_optional_items` now runs
even if the `experimental-sliding-sync` feature is absent. It was
required only because of `latest_event`, but that's an optional
field! However, the `test_room_info_deserialization` requires the
`experimental-sliding-sync` feature, as it tests `latest_event` but
also `recency_stamp`.
Finally, `test_room_info_deserialization` tests
`cached_user_defined_notification_mode`.
This patch updates `matrix_sdk::Room::user_defined_notification_mode`
to cache its result if some mode has been found. The cached result can
be retrieve with
`matrix_sdk_base::Room::cached_user_defined_notification_mode`.
Use a for loop rather than `partition_map`. We're about to add a third list, so
partition_map won't work.
(partition_map ends up using Vec::push under the hood, so this is pretty much
equivalent.)
Update Ruma dependency to expect call membership state events with state
keys that are arbitrary strings, not just pure MXIDs.
When a call membership state key does not exactly match the format of an
MXID, treat it as a valid state key if it starts with an MXID followed
by an underscore, with that MXID designating the owner of the event.
(The state key may also be optionally prefixed with an underscore, which
is permitted as a way to bypass pre-MSC3757 authorization rules against
sending state events with state keys that do not exactly match the
sender's MXID.)
---------
Signed-off-by: Andrew Ferrazzutti <andrewf@element.io>
Co-authored-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
The timeline already listens to changes to the pinned events list (via a
stream), so there's no need to fully reload all the pinned events every
time we receive a new event that's pinned. Technically it may avoid one
or a few lookups, but this is cheap and a subsequent commit/PR will
merge the pinned event cache into the event cache.
This patch restricts the call to `compute_limited` to the sliding sync
proxy implementation (aka MCS3575). It is not necessary for the sliding
sync native implementation (aka Simplified MSC3575). The proxy doesn't
implement the `limited` flag, contrary to Synapse. Let's not run
workarounds when we don't need them.
The patch https://github.com/matrix-org/matrix-rust-sdk/pull/2395 has
introduced `SlidingSyncInner::past_positions` as a mechanism to filter
duplicated responses. It was a problem because the sliding sync `ops`
could easily create corrupted states if they were applied more than
once.
Since https://github.com/matrix-org/matrix-rust-sdk/pull/3664/, `ops`
are ignored.
Now, `past_positions` create a problem with the sliding sync native
implementation inside Synapse because `pos` can stay the same between
multiple responses.
While `past_positions` was helpful to fix bugs in the past, it's no
longer necessary today. Moreover, it breaks an invariant about `pos`: we
must consider it as a blackbox. It means we must ignore if a `pos` value
has been received in the past or not. This invariant has been broken for
good reasons, but it now creates new issues.
This patch removes `past_positions`, along with the associated code
(like `Error::ResponseAlreadyReceived` for example).
This patch changes the visibility of
`SlidingSyncList::invalidate_sticky_data` from `pub` to `pub(super)`.
This is the only place where it must be accessible from.
This patch asserts that when subscribing to a new room, the old room
subscriptions are still present. Is it the behaviour we want? Probably
not, but this is the standard behaviour right now, and we need to assert
it.
We had *two* copies of `response_from_file`, and all calls to them were always
immediately followed by an operation to parse the response as a Ruma response
object.
We can save a whole lot of boilerplate with a generic function that wraps the
json into an HTTP response *and* parses it into a Ruma object.
We weren't updating the database schema version immediately after the v10 -> v11
migration. This was fine in practice, because (a) for now, there is no v12
migration so we ended up setting the schema version immediately anyway; (b) the
migration is idempotent.
However, it's inconsistent with the other migrations and confusing, and is
about to make my test fail, so let's clean it up.
It's better to have fewer public APIs, especially when there's little
annoyance to have it. We could use a request builder that converts into
a Future, too, but considering there's only a single optional parameter,
it's fine to include it in the function's signature.
We should only show the spinner if the *first* sliding sync request is
taking a while. If we have received some data and the second request
takes a while, that is OK.
For the state transition of `Init -> SettingUp` this is handled
correctly, however for `Terminated -> Recovering -> Running` we waited
until the second request returned before hiding the sync spinner. This
meant that if the first request returned quickly the app would show new
data and *then* the sync spinner would show (if the second request took
time).
This situation occurs frequently with the new SSS API, where if all the
new data was returned in the first sync then the second sync would
block waiting for new data, triggering the sync spinner.
This patch changes the `SlidingSync::subscribe_to_room` method to
`subscribe_to_rooms`. Note the plural form. It's now mandatory to
subscribe to a set of rooms. The idea is to avoid calling this method
repeatedly. Why? Because each time the method is called, it sends a
`SlidingSyncInternalMessage` of kind `SyncLoopSkipOverCurrentIteration`,
i.e. it cancels the in-flight sliding sync request, to start over with
a new one (with the new room subscription). A problem arises when the
async runtime (here, Tokio) is busy: in this case, the internal message
channel can be filled pretty easily because its size is 8. Messages
are not consumed as fast as they are inserted. By changing this API:
subscribing to multiple rooms will result in a single internal message,
instead of one per room.
Consequently, the rest of the patch moves the `subscribe` method of
`room_list_service::Room` to `room_list_service::RoomListService`
because it now concerns multiple rooms instead of a single one.
Our CI used to be quite slow and used up a lot of CI time, so as an
optimization we disabled CI for draft PRs.
This is a bit annoying since people want to open draft PRs to check if
CI passes, but without triggering any review requests.
Since our CI is nowadays a bit more efficient let's see if we can enable
it for draft PRs.
This method works the same as `Room::event` but you can provide a custom `RequestConfig` to it.
It's especially useful for the pinned events timeline, since we need a max number of retries and a max number of concurrent requests. With this we can remove some unnecessary complexity.
This way it matches the rest of timelines in the SDK, I reversed it here because I didn't realise most clients just do this reversal of ordering themselves. As they do, they need the same order for this timeline too to be able to reuse their existing logic.
This ensures the cache keeps the events even when the associated `Room` is dropped, which is what we want when using it to cache the pinned events for rooms in the client.
Add `fn Client::pinned_event_cached()` to get a reference to it.
In `matrix.rs` we add methods to interact with the matrix homeserver. And in `machine/mod.rs` we implement the widgetMachine cases for handling (checking capabilities and using the matrixDriver) delayed events.
Add `PinnedEventsLoader` to encapsulate this logic, being able to provide a max number of events to load and how many can be loaded at the same time. Also implement an event cache for it.
Add `PinnedEventsRoom` trait to use it in the same way as `PaginableRoom`, only for pinned events.
The code used to only add new targets to rooms
but never remove the ones that are not in the event anymore.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch ensures that we retain the master key for a given UserIdentityData object, even when a new and different identity arrives via the `/keys/query` endpoint. This concept, called pinning, is similar to certificate pinning in web browsers.
Retaining the master key allows us to detect changes and notify the user accordingly.
This tiny change allows one to easily name the return type of
`EventTimelineItem::reactions()` such that you can use it in a function
signature. Same goes for the `ReactionInfo` type.
I think this was likely just a small oversight in recent changes to the
reactions API, so hopefully it's not controversial.
Without this, it's impossible to write a function that uses
`ReactionsByKeyBySender` or `ReactionInfo` in the signature.
Signed-off-by: Kevin Boos <kevinaboos@gmail.com>
- disable backups and recovery before requesting the reset handle
- attempt device key upload
- re-enable backups both when UIAA is required and when not
This means we have all the information inside SenderData to populate
VerificationStatus and DeviceId for EncryptionInfo, so we can share the
code between SenderDataFinder and get_verification_state.
This makes it impossible to represent states like "there's a local *and*
a remote echo for the same sender for a given reaction", or multiple
reactions from the same sender to the same event, and so on.
Remove `Result` where it is not needed, and switch to `CryptoStoreError`
instead of `OlmError` where possible. Soon, this will allow us to call
some of these methods from places that don't know about `OlmError`.
These calls are equivalent because the old code called
`self.get_user_devices` with a `timeout` of `None`, which meant the call
to `wait_if_user_pending` inside was a no-op.
As it says on the tin. Needed the functions but they were missing. They are analogous to the GlobalAccountData setters.
Signed-off-by: Benjamin Kampmann <ben@acter.global>
The sliding sync proxy has a bug: despite the presence of
`m.room.create` in `bump_event_types`, an invite with an `m.room.create`
event will not have a `timestamp`. Thus, such a room cannot be sorted
reliably.
This patch fixes this problem with a terrible hack, where it tries to
find an `origin_server_ts` value from within the `invite_state` state
events.
Please read the comment to learn more.
The `UserIdentity::is_verified()` method in the matrix-sdk-crypto crate
before version 0.7.2 doesn't take into account the verification status
of the user's own identity while performing the check and may as a result
return a value contrary to what is implied by its name and documentation.
This patch fixes this and adds a regression test.
The method itself is not used internally and as such has not a larger
impact.
Co-authored-by: Denis Kasak <dkasak@termina.org.uk>
Signed-off-by: Damir Jelić <poljar@termina.org.uk>
The `RoomList` provides a `Stream<Item = Vec<VectorDiff<Room>>>`.
This `Stream` receives updates from 2 sources: `RoomList::entries`,
and `Receiver<RoomInfoNotableUpdate>`. When a `RoomInfo` is
updated, a notable update is emitted and broadcasted. The
`RoomList` was filtering these notable updates by _reasons_ (namely
`RoomInfoNotableUpdateReasons`).
This is great and it's a good idea since we can filter which
`RoomInfoNotableUpdate` will trigger a `RoomList` update. However, too
many _reasons_ were hidden/implicit, and it creates several regressions
because (i) these _reasons_ were implicit, (ii) since the business rules
are not defined, there is no tests for that (not in this SDK, not in
apps like ElementX). It means we discover missing _reasons_ bug after
bug. It's not pleasant.
The reality is: we are in the middle of big changes, mostly with room
list client-side sorting and simplified sliding sync. We want to relax
a little bit. This patch then disable the feature _filter updates by
reasons_. The `RoomList` will update to all `RoomInfoNotableUpdate` for
the moment. We will get back to this optimisation later.
The `UserIdentity::is_verified()` method in the matrix-sdk-crypto crate
before version 0.7.2 doesn't take into account the verification status
of the user's own identity while performing the check and may as a result
return a value contrary to what is implied by its name and documentation.
This patch fixes this and adds a regression test.
The method itself is not used internally and as such has not a larger
impact.
Co-authored-by: Denis Kasak <dkasak@termina.org.uk>
Signed-off-by: Damir Jelić <poljar@termina.org.uk>
Simplified sliding sync no longer has `sort` or `bump_event_types`
because they are static values on the implementation/server side. This
patch removes them here.
This patch renames “recency timestamp” to “recency stamp”. It prepares
the fact that simplified sliding sync has a `bump_stamp` instead of a
`timestamp`. The notion of _timestamp_ must be removed.
This patch extracts most of `SlidingSync::sync_once` into a
method named `SlidingSync::send_sync_request`. The name mimics
the `SlidingSync::generate_sync_request` similar method: first we
_generate_, then we _send_.
The `SlidingSync::send_sync_request` is generic over the `Request` and
`::sync_once` passes the correct type depending of whether Simplified
MSC3575 is enabled.
This patch is the first over two patches to change the support
of Simplified MSC3575 from a compiler feature flag to a runtime
configuration. This configuration is held by the `Client`.
By default, it's turned off inside `matrix-sdk-ffi` and
`matrix-sdk-integration-testing`, otherwise it's turned on.
Currently on Element X if you receive a sticker in a room, the room list
will show the room as updated but it will show the latest event that is
not a sticker. This change fixes that.
Signed-off-by:
Marco Antonio Alvarez <surakin@gmail.com>
---------
Signed-off-by: Marco Antonio Alvarez <surakin@gmail.com>
Signed-off-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <public@benj.me>
New feature from Matrix 1.11.
- [x] Public API changes documented in changelogs (optional)
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
There were some leftovers from the rename of the ReadOnlyDevice and
identity structs. The store still referenced them.
This gets rid of all the mentions and improves the documentation of the
store methods for devices and identities.
This reduces the work we do to calculate changed devices etc. when DEBUG
logging is not enabled, but more importantly (to me) it makes clear
that this code is only used for logging.
It is possible to remove the m.call capability because we now have
merged and released a version that does not request it anymore on
call.element.io
---------
Signed-off-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <public@benj.me>
This patch rewrites the `test_delayed_decryption_latest_event` test a
little bit. It does exactly the same things, but in a simpler way: it
removes multiple `sleep` and remove 2 sliding sync loops.
First off, the `SyncService` already starts the `RoomListService and the
`EncryptionSync` service. Both of them have their own sliding sync
loop. The test doesn't need other sliding sync loops in their own tasks,
this is not necessary at all: it's just pretty confusing and doesn't
reflect the reality, i.e. how these API are supposed to be used.
Second, it also tests the room for Bob is seen as encrypted.
Third, the `VectorDiff::Reset` is tested before the event from Bob
is sent. It's not only for clarity: it makes the test more robust for
future modifications.
Fourth, instead of waiting with a `sleep` for the event from Bob to be
received by Alice, we instead wait on the room list's stream of Alice to
receive an update. It's more robust this way and reflects the real usage
of this API. It also helps to remove an intermediate `assert_pending!`
that is no longer necessary because we are waiting on the stream just
after.
Finally, just like for the previous modification, this patch removes
another `sleep` for the to-device event from Bob to be received by
Alice, and instead wait on the room list's stream to receive an update.
It's again more robust and reflects the real usage of this API. Plus, it
makes the last `assert_pending!` macro to not be flaky.
This patch fixes a bug where rooms stored in the sliding sync cache
aren't restored by the `SlidingSyncBuilder`.
This patch also removes `SlidingSyncBuilder::rooms` fields which was
used but never modified. It's dead code.
Again, a transaction id received from the remote flow doesn't mean it
corresponds to a local echo sent *this particular session*, so no need
to warn about it.
This log can happen when an event is received, has a transaction id (in
the data received from sync), and doesn't have a corresponding timeline
item (be it local or remote). There's no reason to warn about this,
because this would happen in most cases, for new incoming events coming
from sync, and this pollutes the logs of rageshakes.
This patch removes the `test_room_info_notable_update_deduplication`
test. First off, it's flaky because sometimes Synapse lags, or sends
another events, which makes the test to fail. Second, the same feature
is tested inside the `matrix_sdk_ui::room_list_service` test suite,
with `test_room_sorting` and `test_room_latest_event`, and inside the
`matrix_sdk_base::sliding_sync` test suite, with a better granularity.
And lastly, this test doesn't test what it says: there is no room info
notable update deduplication whatsoever. I personally don't believe it
has ever existed. This test isn't necessary.
ReadOnlyDevice is not particularly useful as a description of why we
have two device types. This commit renames it into DeviceData, as this
struct is used to hold the device keys and additional local device data.
I'm not quite sure why it took me so long to come up with a better name.
Please forgive me past readers.
The test database was created using a slightly modified `oidc-cli`
example, to turn of the database encryption, on commit
d6dca91df86413b0cbf193a4be191835dd81862e
Since it's only used for remote events, and a local echo couldn't figure
that out anyways (since it's not sent yet, and that information makes
sense after sending).
This is needed to prevent the race condition where the invite request finished, the `/sync` one didn't fetch the new membership event yet and we send a message in the room. This message won't be encrypted for the newly invited user and will result in an UTD.
I added a new integration test and I can confirm this [complement-crypto test](https://github.com/matrix-org/complement-crypto/pull/98) now passes instead of being skipped.
Fixes#3622.
---
* fix(sdk): force room member reload after inviting a user
This is needed to prevent the race condition where the invite request finished, the `/sync` one didn't fetch the new membership event yet and we send a message in the room. This message won't be encrypted for the newly invited user and will result in an UTD.
* Use `room.mark_members_missing()` instead, add integration test
* Abort syncing before the test ends
* Resolve nit: else after a return
* Fix race condition where bob may try to join the room before the invite is received
* Remove double sync
This patch removes everything related to the computation of `ops`
from a sliding sync response. With the recent `RoomList`'s client-side
sorting project, we no longer need to handle these `ops`. Moreover, the
simplified sliding sync specification that is coming removes the `ops`.
A `SlidingSyncList` was containing a `rooms` field. It's removed by
this patch. Consequently, all the `SlidingSyncList::room_list` and
`::room_list_stream` methods are also removed.
A `FrozenSlidingSyncList` was containing the `FrozenSlidingSyncRoom`.
This patch moves the `FrozenSlidingSyncRoom`s inside
`FrozenSlidingSync`. Why is it still correct? We only want to keep the
`SlidingSyncRoom::timeline_queue` in the cache for the moment (until
the `EventCache` has a persistent storage). Since a `SlidingSyncList`
no longer holds any information about the rooms, and since `SlidingSync`
itself has all the `SlidingSyncRoom`, this move is natural and still
valid.
Bye bye all this code :'-).
The only thing is that our client builder allows setting only the server
versions, so this means the new field has to contain two `Option`al
fields. This causes a bit of churn, but isn't too bad in the end.
I've been debugging a cause of flakey complement-crypto tests for
about a month now. I was pretty convinced it was deadlocking
somewhere in the memory store `save_changes` code. With additional
logging, it's now clear that the there is an ABBA style deadlock
when `save_changes` is called at the same time as `get_state_events`.
I've also adjusted code for `get_user_ids` as it has a very similar
pattern and also acquires locks in reverse order to `save_changes`,
so is potentially vulnerable to this.
This patch avoids to emit a
`RoomInfoNotableUpdateReasons::RECENCY_TIMESTAMP` for rooms that are new.
Otherwise the entries in `matrix_sdk_ui::room_list_service::RoomList`
receive a `VectorDiff` because a new room is inserted, and then a
`VectorDiff` because the recency timestamp is updated. The second
`VectorDiff` is useless in this case.
First off, this patch removes the
`RoomInfoNotableUpdate::trigger_room_list_update` field. It is replaced
by a `reasons: RoomInfoNotableUpdateReasons` 8-bit unsigned integer. It
addresses the following issues:
1. When a subscriber receives a `RoomInfoNotableUpdate`, they have no
idea what has triggered this update.
2. In
`matrix_sdk_base::sliding_sync::BaseClient::process_sliding_sync_e2ee`,
we were triggering an update even if the latest event wasn't
modified: it is a false-positive, it was a bug and a waste of
resources. Now it's more refined, see the why below.
Second, this patch removes the second `trigger_room_list_update`
argument of `matrix_sdk_base::BaseClient::apply_changes`. This method
now knows where to find the reasons for the room info notable updates,
see next point.
Third, this patch adds a new
`matrix_sdk::StateChanges::room_info_notable_updates` field which is a
B-tree map between an `OwnedRoomId` and a
`RoomInfoNotableUpdateReasons`. The idea is that all places that receive
a `StateChanges` can also create a room info notable update with a
specific reason. This is a finer grained mechanism, and it avoids to
pass new arguments everywhere. It's cleaner.
Finally, it's easier than ever to add a new reason and to propagate it
to subscribers.
The patch renames `RoomInfoUpdate` to `RoomInfoNotableUpdate`.
The functions, methods or variables whose names start with
`roominfo_update(.*)`` are renamed `room_info_notable_update$1`.
* ffi: expose methods for SSO login
Signed-off-by: Johannes Marbach <n0-0ne+github@mailbox.org>
* Update bindings/matrix-sdk-ffi/Cargo.toml
Co-authored-by: Ivan Enderlin <ivan@mnt.io>
Signed-off-by: Johannes Marbach <n0-0ne+github@mailbox.org>
* Refactor code to use a separate object to complete the SSO login
* Remove superfluous .workspace
* Remove superfluous field name
* Fix formatting with nightly toolchain
* Fix clippy errors using nightly toolchain
* Move SSO methods over into Client as AuthenticationService will go away very soon
* Add login tests
* Assign tokio runtime and url getter
* Reformat
* Relocate parts of the code as per review comments
* Add url to debugging representation of SsoHandler
* Add example for login_with_sso_callback
* Use unwrap_or_default to avoid creating empty string
* Remove leftover dependencies
---------
Signed-off-by: Johannes Marbach <n0-0ne+github@mailbox.org>
Co-authored-by: Ivan Enderlin <ivan@mnt.io>
When a request fails because of the exponential backoff, it won't be
re-logged again. It would be useful, for the purposes of the send queue
notably, to see when a request is re-attempted.
Synapse returns a bare `{ "membership": "leave" }` as the content of a
room membership event (for leave membership changes and likely others).
In this case, it'd still be nice to have some kind of display
name/avatar URL to show in UIs; it's possible to reuse information from
the previous member event, if available.
Add a new `max_concurrent_requests` parameter in the `RequestConfig` limits the number of http(s) requests the internal sdk client issues concurrently (if > 0). The default behavior is the same as before: there is no limit on concurrent requests issued.
This is especially useful for resource constrained platforms (e.g. mobile platforms), and if your pattern might lead to issuing many requests at the same time (like downloading and caching all avatars at startup).
- [x] Public API changes documented in changelogs (optional)
Signed-off-by: Benjamin Kampmann <ben.kampmann@gmail.com>
It turns out this will cause a network request if the encryption info hasn't been loaded before, which is the case for opening a client in offline mode. It will slow down displaying the room list or loading the room info in general.
Turns out legacy unencrypted objects can take many forms, and we need to be
tolerant of them.
Also expose `deserialize_legacy_value` as a separate function, because we're
going to need to special-case it.
This patch revisits the need to trigger a room list update for
all changes of `RoomInfo`. For the moment, it reduces the scope to
`recency_timestamp` update.
This patch comes with a test to ensure things work as expected.
... and make `deserialize_value` handle both the old and new formats.
`maybe_encrypt_value` uses a much more efficient representation, so let's
migrate to that.
It's currently possible for integ test results to leak from one test run to the
next (for example, the indexeddb stores hang around in the browser), causing
bad test results.
Extend the test setup routine to clear out the store before the test starts.
This patch removes the `LatestEvent::cached_event_origin_ts`.
It's no longer necessary to cache this value as the
`matrix_sdk_ui::room_list_service::sorter::recency` sorter no longer
uses it.
This patch adds a new field in `RoomInfo`: `recency_timestamp:
Option<MilliSecondsSinceUnixEpoch>>`. Its value comes from a Sliding
Sync Room response, that's why all this API is behind `cfg(feature =
"experimental-sliding-sync")`.
This patch removes the `RoomListService::rooms` cache, since now a
`Room` is pretty cheap to build.
This cache was also used to keep the `Timeline` alive, but it's now
recommended that the consumer of the `Room` keeps its own clone of the
`Timeline` somewhere. We may introduce a cache inside `RoomListService`
for the `Timeline` later.
This patch rewrites `merge_stream_and_receiver` to switch the order
of `roominfo_update_recv` and `raw_stream`. The idea is to give the
priority to `raw_stream` since it will necessarily trigger the room
items recomputation.
This patch also remove the `for` loop with `Iterator::enumerate`, to
simply use `Iterator::position`: it's more compact and it removes a
`break` (it makes the code simpler to understand).
Finally, this patch renames `merged_stream` into `merged_streams`.
Complement uses the FFI `RoomList` API. Since the patch set modifies
this API, Complement is broken. We disable it and will re-enable it once
we have updated Complement.
This patch adapts the `RoomList` FFI API to the recent changes to
suport a `Stream<Item = RoomListItem>` instead of a `Stream<Item =
RoomListEntry>`. Behind the scene, it supports client side sorting for
the rooms but this is transparent for this API.
This patch also removes the `RoomListInput` enum as no input is
supporter anymore.
The `entries` method no long returns a `RoomListEntriesResult` but
directly a `TaskHandle`. The given listener will receive the initial
entries as a `VectorDiff::Append`, which first is simpler but also fixe
a potential race condition bug.
This patch adds a new `cached_event_origin_server_ts` field on
`LatestEvent`, which is a copy of the `origin_server_ts` of the inner
`SyncTimelineEvent`.
This patch removes the `visible_rooms` sliding sync list from
`RoomListService`. As we are taking the path of doing client-side
sorting, the ordering of the server-side will most likely always
mismatch the ordering of the client-side, thus using `visible_rooms`
with room indices make no sense (indices from server-side won't map
indices on the client-side, so room ranges from client-side won't map
what the server knows).
We used to use `visible_rooms` to “preload” the timeline of rooms in
the user app viewport, with a `timeline_limit` of 20. This should be
replaced by room subscriptions starting from now. For the moment, the
user of `RoomListService` is responsible to do that manually. Maybe
`RoomListService` will handle that automatically in the future.
This patch is quite big… `RoomList::entries*` now returns `Room`s
instead of `RoomListEntry`s. This patch consequently updates all the
filters to manipulate `Room` instead of `RoomListEntry`. No more
`Client` is needed in the filters.
This patch also disables the `RoomList` integration test suite in order
to keep this patch “small”.
Development of the SDK is proudly sponsored and maintained by [Element](https://element.io). Element uses the SDK in their next-generation mobile apps Element X on [iOS](https://github.com/element-hq/element-x-ios) and [Android](https://github.com/element-hq/element-x-android) and has plans to introduce it to the web and desktop clients as well.
The SDK is also the basis for multiple Matrix projects and we welcome contributions from all.
</div>
## Purpose
The SDK takes care of the low-level details like encryption,
syncing, and room state, so you can focus on your app's logic and UI. Whether
you're writing a small bot, a desktop client, or something in between, the SDK
is designed to be flexible, async-friendly, and ready to use out of the box.
## Project structure
The rust-sdk consists of multiple crates that can be picked at your convenience:
The Matrix Rust SDK is made up of several crates that build on top of each
other. The following crates are expected to be usable as direct dependencies:
-**matrix-sdk** - Highlevel client library, with batteries included, you're most likely
interested in this.
- **matrix-sdk-base** - No (network) IO client state machine that can be used to embed a
Matrix client in your project or build a full fledged network enabled client
lib on top of it.
- **matrix-sdk-crypto** - No (network) IO encryption state machine that can be
used to add Matrix E2EE support to your client or client library.
-[matrix-sdk-ui](https://docs.rs/matrix-sdk-ui/latest/matrix_sdk_ui/) – A high-level client library that makes it easy to build
full-featured UI clients with minimal setup. Check out our reference client,
[multiverse](https://github.com/matrix-org/matrix-rust-sdk/tree/main/labs/multiverse), for an example.
- [matrix-sdk](https://docs.rs/matrix-sdk/latest/matrix_sdk/) – A mid-level client library, ideal for building bots, custom
clients, or higher-level abstractions. You can find example usage in the
This is a rough migration guide to help you upgrade your code using matrix-sdk 0.5 to the newly released matrix-sdk 0.6 . While it won't cover all edge cases and problems, we are trying to get the most common issues covered. If you experience any other difficulties in upgrade or need support with using the matrix-sdk in general, please approach us in our [matrix-sdk channel on matrix.org][matrix-channel].
## Minimum Supported Rust Version Update: `1.60`
We have updated the minimal rust version you need in order to build `matrix-sdk`, as we require some new dependency resolving features from it:
> These crates are built with the Rust language version 2021 and require a minimum compiler version of 1.60
## Dependencies
Many dependencies have been upgraded. Most notably, we are using `ruma` at version `0.7.0` now. It has seen some renamings and restructurings since our last release, so you might find that some Types have new names now.
## Repo Structure Updates
If you are looking at the repository itself, you will find we've rearranged the code quite a bit: we have split out any bindings-specific and testing related crates (and other things) into respective folders, and we've moved all `examples` into its own top-level-folder with each example as their own crate (rendering them easier to find and copy as starting points), all in all slimming down the `crates` folder to the core aspects.
## Architecture Changes / API overall
### Builder Pattern
We are moving to the [builder pattern][] (familiar from e.g. `std::io:process:Command`) as the main configurable path for many aspects of the API, including to construct Matrix-Requests and workflows. This has been and is an on-going effort, and this release sees a lot of APIs transitioning to this pattern, you should already be familiar with from the `matrix_sdk::Client::builder()` in `0.5`. This pattern been extended onto:
- the [login configuration][login builder] and [login with sso][ssologin builder],
Most have fallback (though maybe with deprecation warning) support for an existing code path, but these are likely to be removed in upcoming releases.
### Splitting of concerns: Media
In an effort to declutter the `Client` API dedicated types have been created dealing with specific concerns in one place. In `0.5` we introduced `client.account()`, and `client.encryption()`, we are doing the same with `client.media()` to manage media and attachments in one place with the [`media::Media` type][media typ] now.
The signatures of media uploads, have also changed slightly: rather than expecting a reader `R: Read + Seek`, it now is a simple `&[u8]`. Which also means no more unnecessary `seek(0)` to reset the cursor, as we are just taking an immutable reference now.
### Event Handling & sync updaes
If you are using the `client.register_event_handler` function to receive updates on incoming sync events, you'll find yourself with a deprecation warning now. That is because we've refactored and redesigned the event handler logic to allowing `removing` of event handlers on the fly, too. For that the new `add_event_handler()` (and `add_room_event_handler`) will hand you an `EventHandlerHandle` (pardon the pun), which you can pass to `remove_event_handler`, or by using the convenient `client.event_handler_drop_guard` to create a `DropGuard` that will remove the handler when the guard is dropped. While the code still works, we recommend you switch to the new one, as we will be removing the `register_event_handler` and `register_event_handler_context` in a coming release.
Secondly, you will find a new [`sync_with_result_callback` sync function][sync with result]. Other than the previous sync functions, this will pass the entire `Result` to your callback, allowing you to handle errors or even raise some yourself to stop the loop. Further more, it will propagate any unhandled errors (it still handles retries as before) to the outer caller, allowing the higher level to decide how to handle that (e.g. in case of a network failure). This result-returning-behavior also punshes through the existing `sync` and `sync_with_callback`-API, allowing you to handle them on a higher level now (rather than the futures just resolving). If you find that warning, just adding a `?` to the `.await` of the call is probably the quickest way to move forward.
### Refresh Tokens
This release now [supports `refresh_token`s][refresh tokens PR] as part of the [`Session`][session]. It is implemented with a default-flag in serde so deserializing a previously serialized Session (e.g. in a store) will work as before. As part of `refresh_token` support, you can now configure the client via `ClientBuilder.request_refresh_token()` to refresh the access token automagically on certain failures or do it manually by calling `client.refresh_access_token()` yourself. Auto-refresh is _off_ by default.
You can stay informed about updates on the access token by listening to `client.session_tokens_signal()`.
### Further changes
- [`MessageOptions`][message options] has been updated to Matrix 1.3 by making the `from` parameter optional (and function signatures have been updated, too). You can now request the server sends you messages from the first one you are allowed to have received.
-`client.user_id()` is not a `future` anymore. Remove any `.await` you had behind it.
-`verified()`, `blacklisted()` and `deleted()` on `matrix_sdk::encryption::identities::Device` have been renamed with a `is_` prefix.
-`verified()` on `matrix_sdk::encryption::identities::UserIdentity`, too has been prefixed with `is_` and thus is now called `is_verified()`.
- The top-level crypto and state-store types of Indexeddb and Sled have been renamed to unique types>
-`state_store` and `crypto_store` do not need to be boxed anymore when passed to the [`StoreConfig`][store config]
- Indexeddb's `SerializationError` is now `IndexedDBStoreError`
- Javascript specific features are now behind the `js` feature-gate
- The new experimental next generation of sync ("sliding sync"), with a totally revamped api, can be found behind the optional `sliding-sync`-feature-gate
## Quick Troubleshooting
You find yourself focused with any of these, here are the steps to follow to upgrade your code accordingly:
### warning: use of deprecated associated function `matrix_sdk::Client::register_event_handler`: Use [`Client::add_event_handler`](#method.add_event_handler) instead
As it says on the tin: we have deprecated this function in favor of the newer removable handler approach (see above). You can still continue to use this `fn` for now, but it will be removed in a future release.
### warning: use of deprecated associated function `matrix_sdk::Client::login`: Replaced by [`Client::login_username`](#method.login_username)
We have replaced the login facilities with a `LoginBuilder` and recommend you use that from now on. This isn't an error yet, but the function will be removed in a future release.
### expected slice `[u8]`, found struct ...
We've updated the `send_attachment` and `Media` signatures to use `&[u8]` rather than `reader: Read + Seek` as it is more convenient and common place for most architectures anyways. If you are using `File::open(path)?` to get that handler, you can just replace that with `std::fs::read(path)?`
### no method named `verified` found for struct `matrix_sdk::encryption::identities::Device` in the current scope
Boolean flags like `verified`, `deleted`, `blacklisted`, etc have been renamed with a `is_` prefix. So, just follow the cargo suggestion:
```
|
69 | device.verified()
| ^^^^^^^^ help: there is an associated function with a similar name: `is_verified`
Ruma has been updated to `0.7.0`, you will find some ruma Events names have changed, most notably, the `AnySyncRoomEvent` is now named `AnySyncTimelineEvent` (and not `AnySyncStateEvent`, which cargo wrongly suggests). Just rename the import and usage of it.
### `std::option::Option<&matrix_sdk::ruma::UserId>` is not a future
You are seeing something along the lines of:
```
19 | if room_member.state_key != client.user_id().await.unwrap() {
| ^^^^^^ `std::option::Option<&matrix_sdk::ruma::UserId>` is not a future
|
= help: the trait `Future` is not implemented for `std::option::Option<&matrix_sdk::ruma::UserId>`
= note: std::option::Option<&matrix_sdk::ruma::UserId> must be a future or must implement `IntoFuture` to be awaited
= note: required because of the requirements on the impl of `IntoFuture` for `std::option::Option<&matrix_sdk::ruma::UserId>`
help: remove the `.await`
|
19 - if room_member.state_key != client.user_id().await.unwrap() {
19 + if room_member.state_key != client.user_id().unwrap() {
```
You are using `client.user_id().await` but `user_id()` is no longer `async`. Just follow the cargo suggestion and remove the `.await`, it is not necessary any longer.
-`Client::subscribe_to_room_info` allows clients to subscribe to room info updates in rooms which may not be known yet.
This is useful when displaying a room preview for an unknown room, so when we receive any membership change for it,
we can automatically update the UI.
-`Client::get_max_media_upload_size` to get the max size of a request sent to the homeserver so we can tweak our media
uploads by compressing/transcoding the media.
- Add `ClientBuilder::enable_share_history_on_invite` to enable experimental support for sharing encrypted room history on invite, per [MSC4268](https://github.com/matrix-org/matrix-spec-proposals/pull/4268).
- The `dynamic_registrations_file` field of `OidcConfiguration` was removed.
Clients are supposed to re-register with the homeserver for every login.
- `RoomPreview::own_membership_details` is now `RoomPreview::member_with_sender_info`, takes any user id and returns an `Option<RoomMemberWithSenderInfo>`.
Additions:
- Add `Encryption::get_user_identity` which returns `UserIdentity`
- Add `ClientBuilder::session_pool_max_size`, `::session_cache_size` and `::session_journal_size_limit` to control the stores configuration, especially their memory consumption
- Add `Room::member_with_sender_info` to get both a room member's info and for the user who sent the `m.room.member` event the `RoomMember` is based on.
This uses [`uniffi`](https://mozilla.github.io/uniffi-rs/Overview.html) to build the matrix bindings for native support and wasm-bindgen for web-browser assembly support. Please refer to the specific section to figure out how to build and use the bindings for your platform.
## Features
Given the number of platforms targeted, we have broken out a number of features
### Platform specific
-`rustls-tls`: Use Rustls as the TLS implementation, necessary on Android platforms.
-`native-tls`: Use the TLS implementation provided by the host system, necessary on iOS and Wasm platforms.
### Functionality
-`sentry`: Enable error monitoring using Sentry, not supports on Wasm platforms.
-`bundled-sqlite`: Use an embedded version of sqlite instead of the system provided one.
### Unstable specs
-`unstable-msc4274`: Adds support for gallery message types, which contain multiple media elements.
## Platforms
Each supported target should use features to select the relevant TLS system. Here are some suggested feature flags for the major platforms:
- Use the `DisplayName` struct to protect against homoglyph attacks.
### Features
- Add `BaseClient::room_key_recipient_strategy` field
- `AmbiguityCache` contains the room member's user ID.
- [**breaking**] `Media::get_thumbnail` and `MediaFormat::Thumbnail` allow to
request an animated thumbnail They both take a `MediaThumbnailSettings`
instead of `MediaThumbnailSize`.
- Consider knocked members to be part of the room for display name
disambiguation.
- `Client::cross_process_store_locks_holder_name` is used everywhere:
- `StoreConfig::new()` now takes a
`cross_process_store_locks_holder_name` argument.
- `StoreConfig` no longer implements `Default`.
- `BaseClient::new()` has been removed.
- `BaseClient::clone_with_in_memory_state_store()` now takes a
`cross_process_store_locks_holder_name` argument.
- `BaseClient` no longer implements `Default`.
- `EventCacheStoreLock::new()` no longer takes a `key` argument.
- `BuilderStoreConfig` no longer has
`cross_process_store_locks_holder_name` field for `Sqlite` and
`IndexedDb`.
- Make `ObservableMap::stream` works on `wasm32-unknown-unknown`.
- Allow aborting media uploads.
- Replace the `Notification` type from Ruma in `SyncResponse` and `StateChanges`
by a custom one.
- Introduce a `DisplayName` struct which normalizes and sanitizes
display names.
### Refactor
- [**breaking**] Rename `DisplayName` to `RoomDisplayName`.
- Rename `AmbiguityMap` to `DisplayNameUsers`.
- Move `event_cache_store/` to `event_cache/store/` in `matrix-sdk-base`.
- Move `linked_chunk` from `matrix-sdk` to `matrix-sdk-common`.
- Move `Event` and `Gap` into `matrix_sdk_base::event_cache`.
- The ambiguity maps in `SyncResponse` are moved to `JoinedRoom` and `LeftRoom`.
- Replace the `Notification` type from Ruma in `SyncResponse` and `StateChanges` by a custom one
- The ambiguity maps in `SyncResponse` are moved to `JoinedRoom` and `LeftRoom`
- `AmbiguityCache` contains the room member's user ID
- `Store::get_rooms` and `Store::get_rooms_filtered` are way faster because they
don't acquire the lock for every room they read.
- `Store::get_rooms`, `Store::get_rooms_filtered` and `Store::get_room` are
renamed `Store::rooms`, `Store::rooms_filtered` and `Store::room`.
- `Client::get_rooms` and `Client::get_rooms_filtered` are renamed
- [**breaking**] `Client::get_rooms` and `Client::get_rooms_filtered` are renamed
`Client::rooms` and `Client::rooms_filtered`.
- `Client::get_stripped_rooms` has finally been removed.
- [**breaking**] `Client::get_stripped_rooms` has finally been removed.
- [**breaking**] The `StateStore` methods to access data in the media cache
where moved to a separate `EventCacheStore` trait.
- [**breaking**] The `instant` module was removed, use the `ruma::time` module instead.
# 0.7.0
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Damir Jelić
