This patch improves the performance of
`ReadReceiptTimelineUpdate::apply`, which does 2 things: it calls
`remove_old_receipt` and `add_new_receipt`. Both of them need an
timeline item position. Until this patch, `rfind_event_by_id` was used
and was the bottleneck. The improvement is twofold as is as follows.
First off, when collecting data to create `ReadReceiptTimelineUpdate`,
the timeline item position can be known ahead of time by using
`EventMeta::timeline_item_index`. This data is not always available, for
example if the timeline item isn't created yet. But let's try to collect
these data if there are some.
Second, inside `ReadReceiptTimelineUpdate::remove_old_receipt`, we use
the timeline item position collected from `EventMeta` if it exists.
Otherwise, let's fallback to a similar `rfind_event_by_id` pattern,
without using intermediate types. It's more straightforward here: we
don't need an `EventTimelineItemWithId`, we only need the position.
Once the position is known, it is stored in `Self` (!), this is the
biggest improvement here. Le't see why.
Finally, inside `ReadReceiptTimelineUpdate::add_new_receipt`, we use
the timeline item position collected from `EventMeta` if it exists,
similarly to what `remove_old_receipt` does. Otherwise, let's fallback
to an iterator to find the position. However, instead of iterating over
**all** items, we can skip the first ones, up to the position of the
timeline item holding the old receipt, so up to the position found by
`remove_old_receipt`.
I'm testing this patch with the `test_lazy_back_pagination` test in
https://github.com/matrix-org/matrix-rust-sdk/pull/4594. With 10_000
events in the sync, the `ReadReceipts::maybe_update_read_receipt` method
was taking 52% of the whole execution time. With this patch, it takes
8.1%.
This patch improves the performance of
`RoomEvents::maybe_apply_new_redaction`. This method deserialises
all the events it receives, entirely. If the event is not an
`m.room.redaction`, then the method returns early. Most of the time,
the event is deserialised for nothing because most events aren't of kind
`m.room.redaction`!
This patch first uses `Raw::get_field("type")` to detect the type of
the event. If it's a `m.room.redaction`, then the event is entirely
deserialized, otherwise the method returns.
When running the `test_lazy_back_pagination` from
https://github.com/matrix-org/matrix-rust-sdk/pull/4594 with 10'000
events, prior to this patch, this method takes 11% of the execution
time. With this patch, this method takes 2.5%.
This patch fixes the performance of
`AllRemoteEvents::increment_all_timeline_item_index_after` and
`decrment_all_timeline_item_index_after`.
It appears that the code was previously iterating over all items. This
is a waste of time. This patch updates the code to iterate over all
items in reverse order:
- if `new_timeline_item_index` is 0, we need to shift all items anyways,
so all items must be traversed, the iterator direction doesn't matter,
- otherwise, it's unlikely we want to traverse all items: the item has
been either inserted or pushed back, so there is no need to traverse
the first items; we can also break the iteration as soon as all
timeline item index after `new_timeline_item_index` has been updated.
I'm testing this patch with the `test_lazy_back_pagination` test in
https://github.com/matrix-org/matrix-rust-sdk/pull/4594. With 10_000
events in the sync, the `ObservableItems::push_back` method (that uses
`AllRemoteEvents::increment_all_timeline_item_index_after`) was taking
7% of the whole execution time. With this patch, it takes 0.7%.
The WAL file can grow depending on the transactions that are run. A
critical case is VACUUM which basically writes the content of the DB
file to the WAL file before writing it back to the DB file.
SQLite doesn't try to reduce the size of the file after that unless we
set an explicit limit,
so we could end up taking twice the size of the database on the
filesystem.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
It should have been done in the migration of version 7, to reduce the
size of the database on the filesystem after the media cache was moved
to the SqliteEventCacheStore. Better late than never.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Fixes#4517.
It turns out that the bugs found in that test were due to 2 causes:
- First commit: `TestRoomDataProvider` didn't use `initial_user_receipts` but returned hardcoded values.
- Second commit: Our own read receipts were ignored in `TimelineStateTransaction::load_read_receipts_for_event`, although we need to process all read receipts via `ReadReceipts::maybe_update_read_receipt` because it knows how to filter out our own read receipts were needed.
This patch drastically improves the performance of
`TimelineEventHandler::deduplicate_local_timeline_item`.
Before this patch, `rfind_event_item` was used to iterate over all
timeline items: for each item in reverse order, if it was an event
timeline item, and if it was a local event timeline item, and if it was
matching the event ID or transaction ID, then a duplicate was found.
The problem is the following: all items are traversed.
However, local event timeline items are always at the back of the items.
Even virtual timeline items are before local event timeline items. Thus,
it is not necessary to traverse all items. It is possible to stop the
iteration as soon as (i) a non event timeline item is met, or (ii) a non
local event timeline item is met.
This patch updates
`TimelineEventHandler::deduplicate_local_timeline_item` to replace to
use of `rfind_event_item` by a custom iterator that stops as soon as a
non event timeline item, or a non local event timeline item, is met, or
—of course— when a local event timeline item is a duplicate.
To do so, [`Iterator::try_fold`] is probably the best companion.
[`Iterator::try_find`] would have been nice, but it is available on
nightlies, not on stable versions of Rust. However, many methods in
`Iterator` are using `try_fold`, like `find` or any other methods that
need to do a “short-circuit”. Anyway, `try_fold` works pretty nice here,
and does exactly what we need.
Our use of `try_fold` is to return a `ControlFlow<Option<(usize,
TimelineItem)>, ()>`. After `try_fold`, we call
`ControlFlow::break_value`, which returns an `Option`. Hence the need
to call `Option::flatten` at the end to get a single `Option` instead of
having an `Option<Option<(usize, TimelineItem)>>`.
I'm testing this patch with the `test_lazy_back_pagination` test in
https://github.com/matrix-org/matrix-rust-sdk/pull/4594. With 10_000
events in the sync, the test was taking 13s to run on my machine. With
this patch, it takes 10s to run. It's a 23% improvement. This
`deduplicate_local_timeline_item` method was taking a large part of the
computation according to the profiler. With this patch, this method is
barely visible in the profiler it is so small.
[`Iterator::try_fold`]: https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.try_fold
[`Iterator::try_find`]: https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.try_find
This patch uses `next_back()` instead of `last()`, which is equivalent
but `last()` requires to iterate over the entire iterator, while
`next_back()` is a single operation.
This patch removes a useless type conversion. The `Room::event()` method
returns a `TimelineEvent`, so calling `Into::into` is useless: we map
`TimelineEvent` to `TimelineEvent`.
This patch removes a useless type conversion. The iterator produces
`TimelineEvent`, so mapping to `TimelineEvent::from` is useless: we map
`TimelineEvent` to `TimelineEvent`.
This test helper is the same as the assert_next_eq helper, but it waits
for the stream to be ready for a certain amount of time instead of
expecting it to be ready right away.
The `SyncService::stop()` method could fail for the following reasons:
1. The supervisor was not properly started up, this is a programmer error.
2. The supervisor task wouldn't shut down and instead it returns a JoinError.
3. We couldn't notify the supervisor task that it should shutdown due the channel being closed.
All of those cases shouldn't ever happen and the supervisor task will be
stopped in all of them.
1. Since there is no supervisor to be stopped, we can safely just log an
error, our tests ensure that a `SyncService::start()` does create a
supervisor.
2. A JoinError can be returned if the task has been cancelled or if the
supervisor task has panicked. Since we never cancel the task, nor
have any panics in the supervisor task, we can assume that this won't
happen.
3. The supervisor task holds on to a reference to the receiving end of
the channel, as long as the task is alive the channel can not be
closed.
In conclusion, it doesn't seem to be useful to forward these error cases
to the user.
This patch moves the creations of the child tasks of the SyncService
into the supervisor tasks itself. This should make it easier to let the
supervisor recreate tasks.
This will become useful once we introduce a offline mode where the
supervisor task becomes responsible to restart syncing once we notice
that the server is back online.
This patch moves `TimelineMetadata`, its implementation and companion
types (like `RelativePosition`) into its own module. The idea is to
reduce the size of the `state.rs` module.
This patch replaces all uses of `TimelineController::add_events_at` by
`TimelineController::handle_remote_events_with_diffs` in the `Timeline`
itself.
The idea is to remove `add_events_at`, as we currently have 2 ways to
add or to handle remote events in the `Timeline`. We want a single one,
because it's simpler!
This patch replaces all uses of `TimelineController::add_events_at` by
`TimelineController::handle_remote_events_with_diffs` in the tests.
The idea is to remove `add_events_at`, as we currently have 2 ways to
add or to handle remote events in the `Timeline`. We want a single one,
because it's simpler!
It is already a 3-tuple and we want to add more data so it will be clearer to use a stuct with named fields.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This will be used as a configuration to decide whether or not to keep
media in the cache, allowing to do periodic cleanups to avoid to have
the size of the media cache grow indefinitely.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Now that the various match branches in the start and stop method of the
SyncService are minimized we can remove the early returns.
This should allow us to more easily add new branches.
The supervisor is defined as two optional fields that are set and
removed at the same time.
This patch converts the two optional fields into a single optional
struct. The fields inside the struct now aren't anymore optional. This
ensures that they are always set and destroyed at the same time.
From cambridge a scheduler is defined as:
> someone whose job is to create or work with schedules
While supervisor is defined as:
> a person whose job is to supervise someone or something
Well ok, that doesn't tell us much, supervise is defined as:
> to watch a person or activity to make certain that everything is done correctly, safely, etc.:
In conclusion, supervising a task is the more common and better
understood terminology here I would say.
Previously we had a lock protecting an empty value, but the logic wants
to protect a bunch of data in the SyncService.
Let's do the usual thing and create a SyncServiceInner which holds the
data and protect that with a lock.
This patch renames `Timeline::subscribe_batched` to
`Timeline::subscribe`. Since the `Timeline::subscribe` method has been
removed because unused, it no longer makes sense to have a “batched”
variant here. Let's simplify things!
This patch simplifies the `Timeline` pagination API as follows:
- a unique `paginate_backwards` method (no more
`focused_paginate_backwards` and `live_paginate_backwards`),
- a unique `paginate_forwards` method (no more
`focused_paginate_forwards`, the `live` variant was absent).
The idea is to unify pagination by hiding the `live` and `focused` mode.
It was already partially the case with `paginate_backards`, but the
`live` and `focused` variants were also present. I believe it creates
an unnecessary confusion.
Firstly, build a `CollectRecipientsResult` as we go, rather than building its
components separately and then assembling it at the end.
Then, factor the common code between the two code paths into a method to update
the `CollectRecipientsResult`.
- supports only text room message types
- enumerates through their body's grapheme clusters and check that every
single one of them is an emoji
- part of the `LazyTimelineItemProvider` so that it can be opt in
`split_devices_for_user` returns a superset of the results of
`split_recipients_withhelds_for_user_based_on_identity`: let's reflect that in
the return types so we can start to share code.
Also, rename `split_recipients_withhelds_for_user_based_on_identity` to
`split_devices_for_user_for_identity_based_strategy` while we are here.
A bounds check was recently relaxed in `imbl`'s `Focus::narrow()`
function: https://github.com/jneem/imbl/pull/89,
which fixed a bug that would cause a panic if the downstream user
of `matrix-sdk-ui` attempted to narrow a focus of Timeline items
using a range that included the last item in the Timeline.
Example: https://github.com/project-robius/robrix/issues/330
This fix has been incorporated in `eyeball-im` and `eyeball-im-util`
and has been tested by me to no longer trigger upon the aforementioned
conditions.
As the comment noted, they're essentially doing the same thing. A
`TimelineEvent` may not have computed push actions, and in that regard
it seemed more correct than `SyncTimelineEvent`, so another commit will
make the field optional.
This patch removes `Timeline::subscribe`. There is
`Timeline::subscribe_batched`, which is the only useful API.
`subscribe` was only used by our tests, and `matrix-sdk-ffi` uses
`subscribe_batched` only.
This patch changes all calls to `Timeline::subscribe` to replace them by
`Timeline::subscribe_batched`. Most of them are in tests. It's the first
step of a plan to remove `Timeline::subscribe`.
The rest of the patch updates all the tests to use
`Timeline::subscribe_batched`.
It became an optional default feature in reqwest 0.12, and we disable
the default features,
so I don't think it was meant to be disabled when the crate was
upgraded.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
It was used in places where we could make use of other helpers, in some
cases. Also introduces the `room_avatar` helper to create the room
avatar state event.
This was a regression introduced in
f0d98602a9.
`latest_edit_json` was first set by the call to
`EventTimelineItem::with_content()`.
It was overwritten in the next section because of the
`if let EventTimelineItemKind::Remote(remote_event)`
that uses `item` instead of `new_item`.
It means that the updated `RemoteEventTimelineItem`
inside`new_item` was replaced by the outdated one inside `item`,
so `latest_edit_json` goes back to its previous value.
I believe that part of why that went unnoticed is that the code looks
more
complicated due to the need to set an inner field, so I decided to
change the API and
move `with_encryption_info` to `EventTimelineItem`, which makes the code
look cleaner.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This change ensures that the user's own live location events are
excluded from the location sharing stream. Since the user's location is
already represented on the map by the blue dot, processing their own
events is redundant and unnecessary.
The others should likely be copied at some point as well, but including
the readme as crate documentation is not currently useful since the
readme for the ui crate is empty, and warning about missing docs or
debug implementations would make CI fail without substantial extra work.
This patch removes a `XXX` (which I believe is a TODO) in the
documentation of `RoomEventCache::subscribe`. We are not going to change
this API anytime soon.
Setting the default log level to `debug` results in logs like:
```
log: log_event
log: latest_event
log: log_event
log: log_event
log: room_info
log: latest_event
log: log_event
log: room_info
```
Presumably they're coming out of the custom tracing configuration and we definitely don't need them.
This is unlikely that it will affect us, so not worth adding a test IMO,
but for the sake of completeness: this handles redacted redactions in
the `AllEventsCache` too.
This is intended to prevent the test
`test_when_user_in_verification_violation_becomes_verified_we_report_it`
flaking. I found that sometimes when it called `Room::members` the
result was empty due to it trying to fetch the answer from the server.
This change prevents that behaviour. I don't know why the behaviour was
inconsistent before.
Regenerate Dan's data with new cross-signing and device keys, for which I know
the private keys.
The signatures are manually calculated for now; this will be improved in a
later commit.
This removes one sync that happens in the background, because it's
likely spurious and may be confusing the server about what's been seen
by the current client.
Add a new created_at to the send_queue_events and
dependent_send_queue_events stored records. This will allow clients to
understand how stale a pending message might be in the event that the
queue encounters and error and becomes wedged.
This change is exposed through the FFI on the `EventTimelineItem` struct
as a new optional field named `local_created_at`. It will be `None` for
any Remote event, and `Some` for Local events (except for those that
were enqueued before the migrations were run).
Signed-off-by: Daniel Salinas
---------
Signed-off-by: Daniel Salinas <zzorba@users.noreply.github.com>
Co-authored-by: Daniel Salinas <danielsalinas@daniels-mbp-2.myfiosgateway.com>
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
Co-authored-by: Daniel Salinas <danielsalinas@Daniels-MBP-2.attlocal.net>
This can be accessed through `fn Room::privacy_settings` and will wrap the functionality related to a room's access and privacy settings.
This commit includes the `fn RoomPrivacySettings::update_canonical_alias` to modify the canonical alias of a room.
As an outsider, I am mostly interested in features and new developments
that have happened, not those that *may* happen. An open-but-not-merged
PR may not get merged in the end, or it may not get merged any time
soon, creating false expectations. Merged PRs, on the other hand, have
definitely happened (even if they get undone, that happens via other PRs
that will get merged later). As such, I think it brings more value to
outsiders.
This fixes the deserialization of the SenderData since it switched to
the base64 encoding for serialization of the master key in one of its
variants.
The issue was introduced in 5ff556f6c3.
Having the final clients define the tracing filters / log levels proved
to be tricky to keep in check resulting missing logs (e.g. recently
introduced modules like the event cache) or unexpected behaviors (e.g.
missing panics because we don't set a global filter). As such we decided
to move the tracing setup and default definitions over to the rust side
and let rust developers have full control over them.
We will now take a general log level and optional extra targets and
apply them on top of the rust side defined defaults. Targets that log
more than the requested by default will remain unchanged while the
others will increase their log levels to match. Certain targets like
`hyper` will be ignored in this step as they're too verbose others
like `matrix_sdk` because they're too generic.
This patch fixes a bug where the code assumes a gap has been inserted,
and thus, is always present. But this isn't the case. If `prev_batch`
is `None`, a gap is not inserted, and so we cannot remove it. This patch
checks that `prev_batch` is `Some(_)`, which means the invariant is
correct, and the code can remove the gap.
This patch removes `SlidingSyncRoomInner::client` because, first
off, it's not `Send`, and second, it's useless. Nobody uses it, it's
basically dead code… annoying dead code… bad dead code!
Sliding sync is no longer experimental. It has a solid MSC4186, along
with a solid implementation inside Synapse. It's time to consider it
mature.
The SDK continues to support the old MSC3575 in addition to MSC4186.
This patch only removes the `experimental-sliding-sync` feature flag.
From now on, this patch considers that `VectorDiff`s are the official
input type for the `Timeline`, via `RoomEventCacheUpdate` (notably
`::UpdateTimelineEvents`).
This patch removes `TimelineSettings::vectordiffs_as_inputs`. It thus
removes all deduplication logics, as it is supposed to be managed by the
`EventCache` itself.
This patch removes the `AddTimelineEvents` variant from
`RoomEventCacheUpdate` since it is replaced by `UpdateTimelineEvents`
which shares `VectorDiff`.
This patch also tests all uses of `UpdateTimelineEvents` in existing
tests.
If it's present, we just let it untouched. Otherwise, we set it to
`error` if it's missing. See code comment explaining why we need this.
This makes sure we log panics at the FFI layer, since the `log-panics`
crate will use the `panic` target at the error level.
Some errors can be handled immediately and don't need a request to be
spawned, e.g. invalid mimetype and so on. The returned task handle still
deals about "late" errors about the upload failing (for sync uploads) or
the send queue failing to push the media upload (for async uploads).
This patch renames `RoomEvents::filter_duplicated_events` to
`collect_valid_and_duplicated_events` as I believe it improves the
understanding of the code. The variables named `unique_events` are
renamed `events` as all (valid) events are returned, not only the unique
ones.
This patch adds the `Pagination` variant to the `EventsOrigin` enum.
Not something really mandatory and that is likely to fix a bug, but it's
now correct.
Since 8205da898e it has been possible to
attach (intentional) mentions to _edited_ media captions, but the
send_$mediatype() timeline APIs provided no way to send them with the
initial event. This fixes that.
Signed-off-by: Joe Groocock <me@frebib.net>
We have quite a few `allow(dead_code)` annotations. While it's OK to use
in situations where the Cargo-feature combination explodes and makes it
hard to reason about when something is actually used or not, in other
situations it can be avoided, and show actual, dead code.
With experimental-sliding-sync enabled and e2e-encryption disabled,
there were a bunch of warnings about unused imports. This fixes them
(but a few warnings about other unused items remain).
I was investigating a potential deadlock with the event cache storage,
and only found a few places where to make the code a bit more idiomatic
and more readable.
`compute_display_name` is made private again, and used only within the
base crate. A new public counterpart `Room::display_name` is introduced,
which returns a cached value for, or computes (and fills in cache) the
display name. This is simpler to use, and likely what most users expect
anyways.
It claimed that it would immediately return when the cached display name
value was computed, but that's absolutely not the case.
Spotted while reviewing a PR updating `iamb` to the latest version of
the SDK.
Instead of keeping on handling unwedged events from the sending queue,
it's now required to re-enable the send queue manually for the room that
encountered the sending error, all the time. This is more consistent,
and avoids weird behavior when a user would 1. send an event for which
sending fails, in an unrecoverable manner, 2. send an event that's
actually sendable.
web-time's Instant type is already used elsewhere in the project. It is
an alias for std's Instant type on most targets, but tries to call into
JavaScript on wasm32-unknown-unknown (assuming that the wasm blob is
used in from a browser context). Its Duration type is a plain re-export
of std's Duration, even on wasm32-unknown-unknown.
Sometimes we can get the bytes directly, e.g. in Fractal we can get an
image from the clipboard. It avoids to have to write the data to a
temporary file only to have the data loaded back in memory by the SDK
right after.
The first commit to accept any type that implements `Into<String>` for
the filename is grouped here because it simplifies slightly the second
commit.
Note that we could also use `AttachmentSource` in the other
`send_attachment` APIs, on `Room` and `RoomSendQueue`, for consistency.
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Every caller to `with_events_mut` must propagate the vector diff
updates, otherwise updates would be missing to the room event cache's
observers. This slightly tweaks the signature to make this a bit
clearer, and adjusts the code comment as well.
`AttachmentConfig::with_thumbnail()` is replaced by
`AttachmentConfig::new().thumbnail()`.
Simplifies the use of `AttachmentConfig`, by avoiding code like:
```rust
let config = if let Some(thumbnail) = thumbnail {
AttachmentConfig::with_thumbnail(thumbnail)
} else {
AttachmentConfig::new()
};
```
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
A previous patch deduplicates the remote events conditionnally. This
patch does the same but for timeline items.
The `Timeline` has its own deduplication algorithm (for remote
events, and for timeline items). The `Timeline` is about to receive
its updates via the `EventCache` which has its own deduplication
mechanism (`matrix_sdk::event_cache::Deduplicator`). To avoid conflicts
between the two, we conditionnally deduplicate timeline items based on
`TimelineSettings::vectordiffs_as_inputs`.
This patch takes the liberty to refactor the deduplication mechanism of
the timeline items to make it explicit with its own methods, so
that it can be re-used for `TimelineItemPosition::At`. A specific
short-circuit was present before, which is no more possible with the
rewrite to a generic mechanism. Consequently, when a local timeline
item becomes a remote timeline item, it was previously updated (via
`ObservableItems::replace`), but now the local timeline item is removed
(via `ObservableItems::remove`), and then the remote timeline item is
inserted (via `ObservableItems::insert`). Depending of whether a virtual
timeline item like a date divider is around, the position of the removal
and the insertion might not be the same (!), which is perfectly fine as
the date divider will be re-computed anyway. The result is exactly the
same, but the `VectorDiff` updates emitted by the `Timeline` are a bit
different (different paths, same result).
This is why this patch needs to update a couple of tests.
This patch adds a trick around `SyncTimelineEvent` to avoid reaching the
`recursion_limit` too quickly. Read the documentation in this patch to
learn more.
The `Timeline` has its own remote event deduplication mechanism. But
we are transitioning to receive updates from the `EventCache` via
`VectorDiff`, which are emitted via `RoomEvents`, which already runs its
own deduplication mechanism (`matrix_sdk::event_cache::Deduplicator`).
Deduplication from the `EventCache` will generate `VectorDiff::Remove`
for example. It can create a conflict with the `Timeline` deduplication
mechanism.
This patch updates the deduplication mechanism from the `Timeline`
when adding or updating remote events to be conditionnal: when
`TimelineSettings::vectordiffs_as_inputs` is enabled, the deduplication
mechanism of the `Timeline` is silent, it does nothing, otherwise it
runs.
This patch adds the `AllRemoteEvents::range` method. This
is going to be useful to support `VectorDiff::Insert` inside
`TimelineStateTransaction::handle_remote_events_with_diffs`.
This patch adds the `ObservavbleItems::insert_remote_event` method.
This is going to be useful to implement `VectorDiff::Insert` inside
`TimelineStateTransaction::handle_remote_events_with_diffs`.
This patch adds a new variant to `RoomEventCacheUpdate`, namely
`UpdateTimelineEvents. It's going to replace `AddTimelineEvents` soon
once it's stable enough. This is a transition. They are read by the
`Timeline` if and only if `TimelineSettings::vectordiffs_as_inputs` is
turned on.
This patch adds `with_vectordiffs_as_inputs` on `TimelineBuilder` and
`vectordiffs_as_inputs` on `TimelineSettings`. This new flag allows to
transition from one system to another for the `Timeline`, when enabled,
the `Timeline` will accept `VectorDiff<SyncTimelineEvent>` for the
inputs instead of `Vec<SyncTimelineEvent>`.
Recently we started to differentiate between rooms we've been banned
from from rooms we have left on our own.
Sadly the non-left rooms matcher only checked if the room state is not
equal to the Left state. This then accidentally moved all the banned
rooms to be considered as non-left.
We replace the single if expression with a match and list all the
states, this way we're going to be notified by the compiler that we need
to consider any new states we add in the future.
When starting to back-paginate, in this test, we:
- either have a previous-batch token, that points to the first event
*before* the message was sent,
- or have no previous-batch token, because we stopped sync before
receiving the first sync result.
Because of the behavior introduced in 944a9220, we don't restart
back-paginating from the end, if we've reached the start. Now, if we are
in the case described by the first bullet item, then we may backpaginate
until the start of the room, and stop then, because we've back-paginated
all events. And so we'll never see the message sent by Alice after we
stopped sync'ing.
One solution to get to the desired state is to clear the internal state
of the room event cache, thus deleting the previous-batch token, thus
causing the situation described in the second bullet item. This achieves
what we want, that is, back-paginating from the end of the timeline.
This cleanup task will run while the knock request subscription runs and will use the `Room::room_member_updates_sender` notification to call `Room::remove_outdated_seen_knock_requests_ids` and remove outdated seen knock request ids automatically.
This will check the current seen knock request ids against the room members related to them and will remove those seen ids for members which are no longer in knock state or come from an outdated knock member event.
This sender will notify receivers when new room members are received: this can happen either when reloading the full room member list from the HS or when new member events arrive during a sync.
The sender will emit a `RoomMembersUpdate`, which can be either a full reload or a partial one, including the user ids of the members that changed.
The code would use a chunk iterator that moves forward, but call
`push_front()` repetitively on each chunk, semantically storing the
lengths in *reverse* order.
This could result in subsequent panics, when a new chunk was added,
because the links would not match what's expected (e.g. the last chunk
must have no successor, etc.).
The previous code would skip based on the position's index, but not the
position's chunk. It could be that the position's chunk is different
from the first items chunk, as shown in the example, where the linked
chunk ends with a gap; in this case, the position's index would be 0,
while the first chunk found while iterating backwards had 3 items. As a
result, items 'd' and 'e' would be skipped incorrectly.
The fix is to take into account the chunk id when skipping over items.
This is needed to tell apart rooms in left and banned state in places like `RoomInfo` or `RoomPreview`.
The banned rooms will still count as left rooms in the sync processes.
This subscription will combine 3 streams: one notifying the members in the room have changed, another notifying the seen join requests have changed, and finally a third one notifying when the room members are no longer synced.
With this info we can track when we need to generate a new list of join requests to be emitted so the client can always have an up to date list.
This will allow us to keep track of which join room requests are marked as 'seen' by the current user and return them as such.
Also, add some methods to `Room` to mark new join requests as seen and to get the current ids for the seen join requests.
The conditions required to cause the bug might have been impossible to
reach in the real world, because it assumes a mix of:
- events present in the linked chunk
- no prev-batch token
However: now that we have storage, we could end up in this situation,
when reaching the start of the timeline (since there'll be no previous
gap in that case). We need to handle that better in the linked chunk
representation itself, but in the meanwhile, we should insert the gap
and the events in a relative correct order.
If a linked chunk update starts with a RemoveChunk update, then the
transaction may start with a SELECT query and be considered a read
transaction. Soon enough, it will be upgraded into a write transaction,
because of the next UPDATE/DELETE operations that happen thereafter. If
there's another write transaction already happening, this may result in
a SQLITE_BUSY error, according to
https://www.sqlite.org/lang_transaction.html#deferred_immediate_and_exclusive_transactions
One solution is to always start the transaction in immediate mode. This
may also fail with SQLITE_BUSY according to the documentation, but it's
unclear whether it will happen in general, since we're using WAL mode
too. Let's try it out.
This patch fixes an edge case where the member is alone in the room with
a service member. We already subtracted the number of service members
in the case we calculated the room summary ourselves, but we did not do
so when the server provided the room summary.
This lead to the room, instead of being called `Empty`, being called
`Foo and N others`.
Update the workaround for https://github.com/rust-lang/rust/issues/109717 to
avoid hardcoding the clang version; instead, run `clang -dumpversion` to figure
it out.
While we're there, use the `CC_x86_64-linux-android` env var, which should
point to clang, rather than relying on `ANDROID_NDK_HOME` to be set.
The `MemoryStore` implementation of the `StateStore` has grown into a
monster, with one lock per field. It's probably overkill, as individual
fields don't need fine-grained locks like this; after all, accesses to
the store shouldn't be reentrant in general.
Fixes#3720.
- rename RawLinkedChunk -> RawChunk
- rename RawChunk::id -> RawChunk::identifier
- precise that a `RawChunk` is mostly a `Chunk` with different
previous/next links.
And let the caller rebuild the linked chunk. This is slightly nicer in
that it allows us to display the raw representation of a reloaded linked
chunk, before checking its internal state is consistent; this will allow
for better debug of issues related to the linked chunk internal state.
No functional changes.
The `MediaFormat` reflects only the request that would be made to the homeserver.
There is no link between the format of the files stored in the media cache and their purpose in an event.
`MediaFormat::Tumbnail` means that we request a server-generated thumbnail of a file in the media repository.
Since the thumbnail is its own file in the media repository, it makes more sense to use `MediaFormat::File`.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
By default, the event cache store will be disabled. If disabled, it will
clean all the events in the cache store; most of the time this will do
nothing, since the store will not even be filled with any event data, so
it would be cheap to do. If some data was filled in the cache store
before, then it would be cleared after the cache store has been
disabled.
This makes a less awkward API than the previous one, where `None` and
`Some(false)` carried different semantics.
This patch creates `ObservableItemsTransactionEntry` that mimics
`ObservableVectorTransactionEntry`. The differences are `set` is
renamed `replace`, and `remove` is unsafe (because I failed to update
`AllRemoteEvents` in this method due to the borrow checker).
This patch creates `ObservableItemsEntries` and particularly
`ObservableItemsEntry` that wraps the equivalent
`ObservableVectorEntries` and `ObservableVectorEntry` with the
noticeable difference that `ObservableItemsEntry` does **not** expose
the `remove` method. It only exposes `replace` (which is a renaming
of `set`).
This patch moves `AllRemoteEvents` inside `observable_items` so that
more methods can be made private, which reduces the risk of misuses
of this API. In particular, the following methods are now strictly
private:
- `clear`
- `push_front`
- `push_back`
- `remove`
- `timeline_item_has_been_inserted_at`
- `timeline_item_has_been_removed_at`
In fact, now, all `&mut self` method (except `get_by_event_id_mut`) are
now strictly private!
This patch adds the Room::send_raw method to the bindings, making it usable from
e.g. Swift.
Signed-off-by: Jonas Richard Richter <jonas-richard.richter@telekom.de>
Whenever it needs to back-paginate, the event cache should start with
the *most recent* backpagination token, not the oldest one.
This isn't a functional change, until the persistent storage is enabled.
The reason is that, currently, there is one previous-batch token alive;
after it's used, it's replaced with another gap and the events it served
to request from the server.
When persistent storage will be enabled, we'll have situations like the
one shown in the test code, where we can have multiple previous-batch
token alive at the same time. In that case, we'll need to back-paginate
from the most recent events to the least recent events, and not the
other way around, or we'll have holes in the timeline that won't be
filled until we got to the start of the timeline.
Extracted from #4329. This does not change the `MediaFormat` of the
request used in the media cache by the send queue.
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
It is triggered by the `xshell::cmd!` macro, and is fixed in xshell 0.2.7, which we cannot upgrade to.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
The test requires subtle conditions to trigger:
- initialize a timeline from a room-list-service's room
- start a backpagination with that timeline (so the room event cache's
paginator is busy)
- try to initialize another timeline with the same room-list-service's
room (e.g. because the first room has been closed, and the app using it
doesn't have a room cache)
This would fail, because initializing a timeline calls
`EventCache::add_initial_events()` all the time, which tries to reset
the paginator's state, which assumes the paginator's not paginating at
this point. In a soon future, we'll get rid of the
`add_initial_events()` function because the event cache will handle its
own persistent storage; in the meantime, a correct fix is to skip
`add_initial_events()` if there was already something in the linked
chunk. After all, we're likely to fill the initial events with the same
events all the time, or a subset of more recent events. By doing that,
we're likely keeping *more* events in the linked chunk, instead.
Thanks to @stefanceriu for reporting the issue and confirming the fix
works!
Currently the WidgetDriver just returns unspecified error strings to the
widget that can be used to display an issue description to the user. It
is not helpful to run code like a retry or other error mitigation logic.
Here it is proposed to add standardized errors for issues that every
widget driver implementation can run into (all matrix cs api errors):
https://github.com/matrix-org/matrix-spec-proposals/pull/2762#discussion_r1838804895
This PR forwards the errors that occur during the widget processing to
the widget in the correct format.
NOTE:
It does not include request Url and http Headers. See also:
https://github.com/matrix-org/matrix-spec-proposals/pull/2762#discussion_r1839802292
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
Adds new UtdCause variants for withheld keys, enabling applications to display customised messages when an Unable-To-Decrypt message is expected.
refactor(crypto): Move WithheldCode from crypto to common crate
Since xshell 0.2.3 the behavior of the run() function has changed in a
incompatible manner. Namely the stdin for the run() function no longer
inherits stdin from the shell. This makes it impossible for commands
that are executed by the run() function to accept input from the shell.
We don't use this functionality in many places but the `xtask release
prepare` command is now broken.
Let's just pin xshell to a working version while we wait for this to be
resolved upstream.
Upstream-issue: https://github.com/matklad/xshell/issues/63
Virtual timeline items will still be provided and the `default_event_filter` will be applied before everything else.
Instances of these timelines will be used to power the 2 different tabs shown on the new media browser. The client will be responsible for interacting with it similar to a normal timeline and transforming its data into something renderable e.g. section by date separators (which will be made configurable in a follow up PR)
Having a mutable iterator can be dangerous and is probably too generic
regarding the safety we want to add around the `AllRemoteEvents` type.
This patch removes `iter_mut` and replaces it by its only use case:
`get_by_event_id_mut`.
This patch replaces `VecDeque<EventMeta>` by `AllRemoteEvents` which
is a wrapper type around `VecDeque<EventMeta>`, but this new type aims
at adding semantics API rather than a generic API. It also helps to
isolate the use of these values and to know precisely when and how they
are used.
As a first step, `AllRemoteEvents` implements a generic API to not break
the existing code. Next patches will revisit that a little bit step
by step.
The `add_or_update_remote_event` method always returns `true`. This
patch updates the method to return nothing, and cleans up the call sites
accordingly. This patch also adds comments to clarify the code flow.
The bool value returned by `add_or_update_remote_event` was supposed
to be `false` if the event was duplicated. First off, as soon as the
`Timeline` receives its events from the `EventCache` via `VectorDiff`,
the `event_cache::Deduplicator` will take care of deduplication,
so the `Timeline` won't have to handle that itself. Second,
`add_or_update_remote_event` was sometimes removing an event, but it
was re-inserting a new one immediately without returning `false`: it was
never returning `false` because a new event was always added.
This test was checking that a new device which has access to backups
returned an unknown UTD when it was given empty JSON for the event. It
was only passing because we currently have incorrect behaviour when
backups are enabled.
The fix is to make the device old and without access to backups, so that
we still consider this UTD unexpected.
Any `.remove` operation called on a `ObservableMap` did not re-index
`values` _after_ the removed position, which means any later operation
on elements inserted after the previously removed key would either be
fetched wrongly, or panic when using `.get_or_create`.
This PR fixes these two related bugs and adds extra test cases.
---------
Signed-off-by: g@leirbag.net
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
This patch changes `TimelineStateTransaction::add_remote_events_at`
to take an `IntoIterator<Item = Into<SyncTimelineEvent>>`
for `events`. In the current code, it saves one
`iter().map(Into::into).collect::<Vec<_>>()`, but it will save another
one when we will support `VectorDiff`s coming from the `EventCache`.
It also avoids to allocate a vector to pass new events (this mostly
happens in the test, but it can happen in real life).
If no server names are provided for the room summary request and the
room's server name doesn't match the current user's server name, add the
room alias/id server name as a fallback value. This seems to fix room
preview through federation.
Also, when getting a summary for a room list item, if it's an invite
one, add the server name of the inviter's user id as another possible
fallback.
Changelog: Use the inviter's server name and the server name from the
room alias as fallback values for the via parameter when requesting the
room summary from the homeserver.
This required adding support for *reading* out of the event cache, for
the sqlite backend. This paves the way for the next PR (reload from the
cache), and it should also help with testing at the `EventCacheStore`
trait layer some day.
A comment was duplicating (the first trace! that's removed here), and
the second block comment only applied to new items, and was not as
concise as it could be.
This patch unifies the logic for inserting timeline items at `Start`
and `End` positions. Both `TimelineItemPositions` can share the same
implementation, making separate logic unnecessary. Previously, `End`
included a duplicated events check as well, while `Start` did not, leading
to inconsistency.
The changes strictly involve moving and refactoring, with no functional
modifications.
This PR introduces a new variant to `UtdCause` specifically for device-historical messages (`HistoricalMessage`). These messages cannot be decrypted if key storage is inaccessible. Applications can leverage this new variant to provide more informative error messages to users.
This patch updates `LinkedChunk::new_with_update_history` to emit an
`Update::NewItemsChunk` because the first chunk is created and it must
emit an update accordingly.
This patch fixes the `test_echo` test. It was doing the following:
* client sends an event to the server,
* servers acknowledges with the ID `$wWgymRfo7ri1uQx0NXO40vLJ`,
* client syncs and the server returns one event with ID `$7at8sd:localhost`,
* the test expects those 2 events to be the same!, which is incorrect.
The test was working because the transaction IDs are the same, but
that's an abuse of the existing code (the code will change soon, another
patch is coming). Whatever the code does: the connection must be based
on the event ID, not the transaction ID.
Building the docs for xtask spews a bunch of unexpected cfg warnings. As
these warnings come from a macro in a dependency and the docs for xtask
don't exist nor will, let's just not build them with the rest of the
docs.
These are all coming from macro invocations of macros that are defined
in other crates. It's likely a clippy issue. We should try to revert
this the next time we bump the nightly version we're using.
This patch tries to clear confusion around
`TimelineItemPosition::UpdateDecrypted(usize)`: it does contains
a timeline item index. This patch changes to
`TimelineItemPosition::UpdateDecrypted { timeline_item_index: usize }`
It's unused so it's mostly cosmetic, and it's trivial to reimplement
using `linked_chunk.items().count()`; let's do that instead of keeping
the perfect exact count synchronized with the chunks, which pollutes the
code in a few places.
This patch renames `TimelineEnd` into `TimelineNewItemPosition` for
2 reasons:
1. In the following patches, we will introduce a new variant to insert
at a specific index, so the suffix `End` would no longer make sense.
2. It's exactly like `TimelineItemPosition` except that it's used
only and strictly only to add **new** items, which is why we can't use
`TimelineItemPosition` because it contains the `UpdateDecrypted`
variant. This renaming reflects it's only about **new** items.
This patch takes the opportunity to move the `RemoteEventOrigin` inside
`TimelineNewItemPosition` to simplify method signatures. They always
go together.
These two are required to properly compute the room preview of a joined
room:
- m.room.create ends up filling the `room_type` (space or not)
- m.room.history_visibility ends up filling the `is_world_readable`
field.
It seems sensible to assume that if a client is able to generate a thumbnail,
it should be able to get all this information for it too.
A thumbnail with no information is not really useful, as we don't know when it could be used instead of the original image.
Removes `BaseThumbnailInfo`.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch fixes a bug in `AsVector`: when an `Update::Clear` value
is received, `AsVector`'s internal state must be cleared too, i.e. the
`UpdateToVectorDiff::chunks` field should be reset to an initial value!
This patch adds a test to ensure this works as expected.
Ruma doesn't currently validate mxuri's and as such `MediaSource`s passed over FFI can contain invalid/empty URLs. This change introduces a wrapper type around Ruma's and failable transformations so that appropiate actions can be taken beforehand e.g. returning a `TimelineItemContent::FailedToParseMessageLike` or nil-ing out the thumbnail info.
This patch implements `LinkedChunk::clear`. The code from `impl Drop
for LinkedChunk` has been moved inside `Ends::clear`, and replaced by
a simple `self.links.clear()`. In addition, `LinkedChunk::clear` indeed
calls `self.links.clear()` but also resets all fields.
This patch adds the `Clear` variant to `Update`.
This patch updates `AsVector` to emit a `VectorDiff::Clear` on
`Update::Clear`.
Finally, this patch adds the necessary tests.
This patch updates `EventCacheStore::handle_linked_chunk_updates` to
take a `Vec<Update<Item, Gap>>` instead of `&[Update<Item, Gap>]`.
In fact, `linked_chunk::ObservableUpdates::take()` already returns a
`Vec<Update<Item, Gap>>`; we can simply forward this `Vec` up to here
without any further clones.
This patch creates a new `MemoryStoreInner` and moves all fields from
`MemoryStore` into this new type. All locks are removed, but a new lock
is added around `MemoryStoreInner`. That way we have a single lock.
A `RelationalLinkedChunk` is like a `LinkedChunk` but with a relational
layout, similar to what we would have in a database.
This is used by memory stores. The idea is to have a data layout that
is similar for memory stores and for relational database stores, to
represent a `LinkedChunk`.
This type is also designed to receive `Update`. Applying `Update`s
directly on a `LinkedChunk` is not ideal and particularly not trivial
as the `Update`s do _not_ match the internal data layout of the
`LinkedChunk`, they have been designed for storages, like a relational
database for example.
This type is not as performant as `LinkedChunk` (in terms of memory
layout, CPU caches etc.). It is only designed to be used in memory
stores, which are mostly used for test purposes or light usages of the
SDK.
When instantiating a room preview, previously it would try to just check if the room exists locally either as joined, invited, knocked, left, etc., and then retrieve the info we cached about it.
While this seems fine for most cases, it turns out for non-joined rooms, the info we have locally will **always** be the one we received when the invite/knock/leave action took place and it'll never be updated,
so we may have the case where we knock into a room, never receive a response, someone changes the join rule of the room to something else and we'll think about this room as a 'request to join' room until we clear the local cache.
To prevent that, we can only use the local data for joined rooms, which are constantly updated, and try to use the room summary API and other fallbacks for the rest, even if they're rooms known to us.
There's a lock making sure we're not doing multiple refreshes of an OIDC
token at the same time. Unfortunately, this lock could be dropped, if
the task spawned by the inner function was detached.
The lock must be held throughout the entire detached task's lifetime,
which this refactoring ensures, by setting the lock's result after
calling the inner function.
It has the same semantics used when creating a caption (if no formatted
caption is provided, assume a provided caption is markdown and use that
as the formatted caption).
This PR makes audio, file, image and video messages be editable so that
the timeline signals when it is possible to use #4277/#4300 for editing
captions.
The (matrix-sdk-)common crate used the (matrix-sdk-)test crate only to
benefit from the `async_test` proc macro, which is conveniently defined
in another crate.
My goal is to make `EventFactory`, at this point in the commit history,
defined in the main SDK crate, available in the test crate.
`EventFactory` makes use of some types defined in common, so there's a
circular dependency at the moment.
To split this circular dependency, I've changed the common crate to
depend on the test-macro crate directly; now the test crate can depend
on the common crate, and everybody's happy.
test(timeline): add an integration test for sending an attachment
test(timeline): add tests for multiple caption edits and local reaction to a media upload
There has been a recent change on `Decryptor::decrypt_event_impl` causing
the function to return an TimelineEvent of kind unable to decrypt
instead of failing with an error.
The `late_decrypt` detection code was not changed, causing any retry to
mark UTDs as late decrypt.
This patch introduces a struct that normalizes and sanitizes display
names. Display names can be a source of abuse and can contain characters
which might make it hard to distinguish one display name from the other.
This struct attempts to make it easier to protect against such abuse.
Changelog: Introduce a DisplayName struct which normalizes and sanitizes
display names.
Co-authored-by: Denis Kasak <dkasak@termina.org.uk>
There was an implicit relationship that the `being_sent` lock needed to
be taken in order to do non-atomic state store operations. With the
change from this commit, the relationship is now more explicit: to get a
handle to the state store, or being_sent, you have to obtain a
`StoreLockGuard` by locking against the store itself. The `WeakClient`
isn't stored in the QueueStorage data structure itself, so it's the only
way to get a `dyn StateStore` from the `QueueStorage`.
Prior to this patch, the send queue would not maintain the ordering of
sending a media *then* a text, because it would push back a dependent
request graduating into a queued request.
The solution implemented here consists in adding a new priority column
to the send queue, defaulting to 0 for existing events, and use higher
priorities for the media uploads, so they're considered before other
requests.
A high priority is also used for aggregation events that are sent late,
so they're sent as soon as possible, before other subsequent events.
Add the `markdown` feature to the SDK crate, otherwise we can't use `FormattedBody::markdown`.
Refactor the pattern matching into an if, add tests to check its behaviour.
Changelog: For `Timeline::send_*` fns, treat the passed `caption` parameter as markdown and use the HTML generated from it as the `formatted_caption` if there is none.
Changelog: This patch introduces a mechanism similar to
`Client::add_event_handler` and `Client::add_room_event_handler`
but with a reactive programming pattern. This patch adds
`Client::observe_events` and `Client::observe_room_events`.
```rust
// Get an observer.
let observer =
client.observe_events::<SyncRoomMessageEvent, (Room, Vec<Action>)>();
// Subscribe to the observer.
let mut subscriber = observer.subscribe();
// Use the subscriber as a `Stream`.
let (message_event, (room, push_actions)) = subscriber.next().await.unwrap();
```
When calling `observe_events`, one has to specify the type of event
(in the example, `SyncRoomMessageEvent`) and a context (in the example,
`(Room, Vec<Action>)`, respectively for the room and the push actions).
This patch updates `eyeball-im` and `eyeball-im-util` to integrate
https://github.com/jplatte/eyeball/pull/63/. With this new feature, we
can have a single implementation of `ObservableMap` (instead of 2: one
for all targets, one for `wasm32-u-u`). It makes it possible to get
`Client::rooms_stream` available on all targets now.
The caption and filenames were weirdly duplicated in each media content,
when the expected behavior is well defined:
- if there's both a caption and a filename, body := caption, filename is
its own field.
- if there's only a filename, body := filename.
We can remove all duplicated fields, knowing this, and reconstruct the
body based on that information. This should make it clearer to FFI users
which is what, and provide a clearer API when creating the caption and
so on.
This patch continues to simplification of the `matrix_sdk_ffi::Client`.
The constructor can receive a `enable_oidc_refresh_lock: bool` instead
of `cross_process_refresh_lock_id: Option<String>`, which was a copy of
`matrix_sdk::Client::cross_process_store_locks_holder_name`.
Now there is a single boolean to indicate whether
`Oidc::enable_cross_process_refresh_lock` should be called
or not. If it has to be called, it is possible to re-use
`matrix_sdk::Client::cross_process_store_locks_holder_name`. Once
again, there is a single place to read this data, it's not copied over
different semantics.
This patch simplifies a little the `ClientBuilder` API:
* `enable_cross_process_refresh_lock` is removed
* `enable_oidc_refresh_crypto_lock` + `set_session_delegate` must be
used instead.
This patch removes the `process_id` argument from
`EncryptionSyncService::new()` and replaces it by
`Client::cross_process_store_locks_holder_name`. The “process ID” is
set when the `Client` is converted into another `Client` tailore for
notification in `NotificationClient` with `Client::notification_client`
which now has a new `cross_process_store_locks_holder_name` argument.
See the Changelog Section to get the details.
Changelog: `Client::cross_process_store_locks_holder_name` is used everywhere:
- `StoreConfig::new()` now takes a
`cross_process_store_locks_holder_name` argument.
- `StoreConfig` no longer implements `Default`.
- `BaseClient::new()` has been removed.
- `BaseClient::clone_with_in_memory_state_store()` now takes a
`cross_process_store_locks_holder_name` argument.
- `BaseClient` no longer implements `Default`.
- `EventCacheStoreLock::new()` no longer takes a `key` argument.
- `BuilderStoreConfig` no longer has
`cross_process_store_locks_holder_name` field for `Sqlite` and
`IndexedDb`.
This patch adds `ClientInner::cross_process_store_locks_holder_name` and
its public method `Client::cross_process_store_locks_holder_name`. This
patch also adds `ClientBuilder::cross_process_store_locks_holider_name`
to configure this value.
Changelog: Implement proper redact handling in the widget driver.
This allows the Rust SDK widget driver to support widgets that
rely on redacting.
Co-authored-by: Damir Jelić <poljar@termina.org.uk>
Previously this only used the Ruma checks, which only handled the initial `#` char and the domain part. With these changes, the name part is also validated, checking it's lowercase, with no whitespaces and containing only allowed chars, similar to what `DisplayName::to_room_alias_name` does.
Moved the code to the SDK crate so it can be properly tested.
This one is used when caching a thumbnail everywhere, and when
attempting to retrieve it; it gives us a single place where to
coordinate the default `MediaThumbnailSettings` parameters.
The previous values would lead to super large memory allocations, as
observed with `valgrind --tool=massive` on the tiny test added in this
commit:
- for 400 rooms each having 100 events, this led to 540MB of
allocations.
- for 1000 rooms each having 100 events, this led to 1.5GB of
allocations.
This is not acceptable for any kind of devices, especially for mobile
devices which may be more constrained on memory. The bloom filter is an
optimisation to avoid going through events in the room's event list, so
it shouldn't cause a big toll like that; instead, we can reduce the
parameters values given when creating the filters.
With the given parameters, 1000 rooms each having 100 events leads to
1.2MB of allocations.
With this, we get notified of the current verification state almost immediately.
Without it, you may either call it too soon and receive an `Unknown` state or you might have to call `Encryption::wait_for_e2ee_initialization_tasks()` and wait until it's finished to request a valid state value.
This patch introduces `EmptyChunk`, a new enum used to represent whether
empty chunks must be removed/unlink or kept from the `LinkedChunk`. It
is used by `LinkedChunk::remove_item_at`.
Why is it important? For example, imagine the following situation:
- one inserts a single event in a new chunk (possible if a (sliding)
sync is done with `timeline_limit=1`),
- one inserts many events at the position of the previous event,
with one of the new events being a duplicate of the first event
(possible if a (sliding) sync is done with `timeline_limit=10` this
time),
- prior to this patch, the older event was removed, resulting in an
empty chunk, which was removed from the `LinkedChunk`, invalidating
the insertion position!
So, with this patch:
- `RoomEvents::remove_events` does remove empty chunks, but
- `RoomEvents::remove_events_and_update_insert_position` does NOT remove
empty chunks, they are kept in case the position wants to insert in this
same chunk.
This fixes a problem when doing an incremental sync at launch,
where `NotLoaded` event would not be dispatched until data became
available or timeout is reached, leading to app waiting for it.
Changelog: the parameter `room_info_notable_update_receiver` was removed
from `RoomList::entries_with_dynamic_adapters`, since it could be
inferred internally instead.
Notably, make it super clear what parameters are required to create the
attachment type, since the function doesn't consume the whole
`AttachmentConfig` for realz.
Changelog: The send queue will now store a serialized
`QueuedRequestKind` instead of a raw event, which breaks the format.
As a result, all send queues have been emptied.
This makes it possible to have different kinds of *parent key*, to
update a dependent request. A dependent request waits for the parent key
to be set, before it can be acted upon; before, it could only be an
event id, because a dependent request would only wait for an event to be
sent. In a soon future, we're going to support uploading medias as
requests, and some subsequent requests will depend on this, but won't be
able to rely on an event id (since an upload doesn't return an
event/event id).
Since this changes the format of `DependentQueuedRequest`, which is
directly serialized into the state stores, I've also cleared the table,
to not have to migrate the data in there. Dependent requests are
supposed to be transient anyways, so it would be a bug if they were many
of them in the queue.
Since a migration was needed anyways, I've also removed the `rename`
annotations (that supported a previous format) for the
`DependentQueuedRequestKind` enum.
Breaking: `ffi::Client::resolve_room_alias` now returns `Result<Option<ResolvedRoomAlias>, ClientError>` instead of `Result<ResolvedRoomAlias, ClientError>`. This allows the client to match the 3 possible cases:
- The room alias exists.
- The room alias does not exist.
- The function failed internally.
This should take care of a bug that caused pinned events to be incorrectly removed when the new pinned event ids list was based on an empty one if the required state of the room didn't contain any pinned events info
This patch removes `RoomListService::new_with_encryption`. This feature
is not used, not useful since it's best to use `EncryptionSyncService`,
and it can be racy depending on how it's used. To avoid potential errors
and bugs, it's preferable to remove this code.
Changelog: a new optional `via_server` parameter was added to `sdk::RoomDirectorySearch::search`, to specify which homeserver to use for searching rooms. In the FFI layer, this parameter is called `via_server_name`.
This patch uses the new `Deduplicator` type, along with
`LinkeChunk::remove_item_at` to remove duplicated events. When a new
event is received, the older one is removed.
This patch introduces `Deduplicator`, an efficient type to detect
duplicated events in the event cache. It uses a bloom filter, and
decorates a collection of events with `Decoration`, which an enum that
marks whether an event is unique, duplicated or invalid.
Changelog: Renamed all the send-queue related "events" to "requests", so
as to generalize usage of the send queue to not-events (e.g. medias,
redactions, etc.).
In a next commit, the `QueuedEvent` will be renamed to `QueuedRequest`.
This specifies which kind of request we want to send with the send
queue; for now, it can only be an event.
This patch fixes a bug in an optimisation inside `UpdateToVectorDiff`
when an `Update::PushItems` is handled. It can sometimes create
`VectorDiff::Append` instead of a `VectorDiff::Insert`. The tests will
be part of the next patch.
This patch adds an `Event` type alias to `SyncTimelineEvent` to (i) make
the code shorter, (ii) remove some cognitive effort, (iii) make things
more convenient.
This method will return a `RoomPreview` for the provided room id.
Also added `fn RoomPreview::leave()` action to be able to decline
invites or cancel knocks, since there wasn't a
`Client::leave_room_by_id` counterpart as there is for join.
The PR also deprecates `RoomListItem::invited_room`, since we have a
better alternative now.
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
Changelog: Support for preallocated media content URI has been added in
`Media::create_content_uri()`, and uploading the content for such a
preallocated URI is possible with `Media::upload_preallocated()`.
Modify the SendQueue in order to persist the error that cause the event
to fail to send as a `QueueWedgeError`. The `QueueWedgeError` is not a
1:1 mapping for all kinds of errors, but holds variant and information
that the client can react to in order to propose "quick fixes"/solution
before retrying to send.
Fixes https://github.com/matrix-org/matrix-rust-sdk/issues/3973
Also fixes https://github.com/element-hq/element-x-ios/issues/3287
because when a timeline reset occurs the fail to send reason is also
lost.
This PR starts with a refactoring commit
https://github.com/matrix-org/matrix-rust-sdk/commit/e7696003e846b64c41761109f326fd37c4506040
to introduce the new `QueueWedgedError` and move the logic that was in
the ffi layer to convert api errors to SendState error variant. This
`QueueWedgedError` can be directly use in the `SendingFailed` variant
and expose to ffi.
Second commit
https://github.com/matrix-org/matrix-rust-sdk/commit/109c1337465ba7965825fec858fd2a4b8b954611
adds the persistence, `QueuedEvent` now have an optional error field
instead of a `is_weged` boolean. Same for LocalEchoContent::Event.
Adds also Migration for sqlite and indexeddb
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
Changelog: We now persist the error that caused an event to fail to send. The error `QueueWedgeError` contains info that client can use to try to resolve the problem when the error is not automatically retry-able. Some breaking changes occurred in the FFI layer for `timeline::EventSendState`, `SendingFailed` now directly contains the wedge reason enum; use it in place of the removed variant of `EventSendState`.
Before we do any more work here, give this variant a better name
Breaking-Change: `matrix_sdk_crypto::type::events::UtdCause::Membership` has
been renamed to `...::SentBeforeWeJoined`.
The FFI will request a scaled version of the thumbnail by default; let's
use the same cache key when caching the thumbnail after an upload.
Thanks @zecakeh for flagging the issue.
This patch removes the `settings` argument of
`RoomListService::subscribe_to_rooms`. The settings were mostly composed
of:
* `required_state`: now shared with `all_rooms`, so that we are
sure they are synced; except that `m.room.create` is added for
subscriptions.
* `timeline_limit`: now defaults to 20.
This patch thus creates the `DEFAULT_REQUIRED_STATE` and
`DEFAULT_ROOM_SUBSCRIPTION_TIMELINE_LIMIT` constants.
Finally, this patch updates the tests, and updates all usages of
`subscribe_to_rooms`.
This patch adds the `m.room.topic` and `m.room.pinned_events` state
events in the `required_state` of the `all_rooms` sliding sync list of
`RoomListService`.
We can't know which key is going to be used precisely for the thumbnail,
so assume non-animated cropped same-size thumbnail media request.
Changelog: when `SendAttachment::store_in_cache()` is set, the thumbnail
is also cached with a sensible default media request (not animated,
cropped, same dimensions as the uploaded thumbnail).
Make the tests behave the same way as the network code, by returning UTDs
as `TimelineEventKind::UnableToDecrypt` instead of `TimelineEventKind::PlainText`.
There's no need for this API anymore.
Changelog: `Timeline::get_event_timeline_item_by_transaction_id` has
been removed. There's no API that makes use of an `EventTimelineItem`
now, those APIs are using a `TimelineEventItemId` instead.
I feel like the ability to convert straight from a `Raw<AnySyncTimelineEvent>>`
into a `SyncTimelineEvent` is somewhat over-simplified: the two are only
occasionally equivalent, and it's better to be explicit.
Changelog: `SyncTimelineEvent` no longer implements `From<Raw<AnySyncTimelineEvent>>`.
The event cache stores its events in a linked chunk. The linked chunk
supports updates (`ObservableUpdates`) via `LinkedChunk::updates()`.
This `ObservableUpdates` receives all updates that are happening inside
the `LinkedChunk`. An `ObservableUpdates` wraps `UpdatesInner`, which
is the real logic to handle multiple update readers. Each reader has a
unique `ReaderToken`. `UpdatesInner` has a garbage collector that drops
all updates that are read by all readers. And here comes the problem.
A category of readers are `UpdatesSubscriber`, returned by
`ObservableUpdates::subscribe()`. When an `UpdatesSubscriber` is
dropped, its reader token was still alive, thus preventing the garbage
collector to clear all its pending updates: they were kept in memory
for the eternity.
This patch implements `Drop` for `UpdatesSubscriber` to correctly remove
its `ReaderToken` from `UpdatesInner`. This patch also adds a test that
runs multiple subscribers, and when one is dropped, its pending updates
are collected by the garbage collector.
The name wasn't very descriptive, and it's tweaking the content, so
let's do that in place, instead of deferring to another method somewhere
else in the codebase.
The tails of the prepare_attachment_message and
prepare_encrypted_attachment_message were almost the same, with the one
different that they were using different ctors for the `EventContent`
types. In fact, all these `EventContent` types also expose a plain `new`
function that can take in either an encrypted or a plain media source,
so we can commonize the code there.
This allows clients to set custom join rules for a room, as would be needed for the knock-only rooms, or restricted rooms (those that can only be joined if the user is part of some other room or space).
See previous commit for explanations. This makes for a simpler API
anyways.
Changelog: `Timeline::edit` doesn't return a bool anymore to indicate it
couldn't manage the edit in some cases, but will return errors
indicating what the cause of the error is.
I think this can't happen, but the send queue can return an error if a
local echo identified by a transaction id doesn't exist anymore in the
database. The only reason the latter could happen is because the local
echo has been sent, in which case an update to the timeline would be
dispatched, and the timeline item would have morphed into a remote echo
in the meantime. So it's really rare that this would happen, and the
`Timeline::redact()` method doesn't have to return a boolean to indicate
success in general.
Changelog: `Timeline::redact()` doesn't return a boolean; previously, it
would only return false if the internal state was invalid, so a new
error `RedactError::InvalidLocalEchoState` has been introduced to
represent that.
This maintains functionality we had prior to the previous commit: if an
event's missing from the timeline (e.g. timeline's been cleared after a
gappy sync response), then still allow editing it.
In particular, this means that trying to edit an event that's not
present anymore in a timeline (e.g. after a timeline reset) will fail,
while it worked before.
Changelog: `Timeline::edit_by_id` has been fused into `Timeline::edit`,
which now takes a `TimelineEventItemId` as the identifier for the local
or remote item to edit. This also means that editing an event that's not
in the timeline anymore will now fail. Callers should manually create
the edit event's content, and then send it via the send queue; which the
FFI function `Room::edit` does.
Changelog: `Timeline::redact_by_id` has been fused into
`Timeline::redact`, which now takes a `TimelineEventItemId` as an
identifier of the item (local or remote) to redact.
We can now use this type instead of passing a string, which means
there's no way to confuse oneself in methods like
`toggle_reaction_local`.
Changelog: Introduced `TimelineUniqueId`, returned by
`TimelineItem::unique_id()` and serving as an opaque identifier to use
in other methods modifying the timeline item (e.g. `toggle_reaction`).
This would not report the `txn_id` field because of the `skip_all`. It's
actually interesting to also get the error, so I'm only skipping self
from now on.
I suppose these were useful at the FFI layer at some point, but they
aren't anymore, so they could be removed.
Changelog: Got rid of `From<String/&str>` for `TimelineEventItemId`.
This would have avoided a few hours of debugging where we thought there
was an issue with multiple timelines spawned at the same time, and then
realized it was expected because of the existence of the pinned timeline
in EX apps.
Older versions of Git require --interactive when we supply --autosquash,
and it's also probably a good idea generally.
See https://stackoverflow.com/a/77663575/22610 for more info.
Changelog: `Client::url_for_oidc_login` is now `Client::url_for_oidc` with an additional `OidcPrompt` parameter. `abort_oidc_login` has been renamed to `abort_oidc_auth`.
This allows clients to directly open the web page they want: the login one, the registration one, consent, etc. It should improve the UX in the registration flow since we can now skip the login one.
We depend on the `futures_util::steam_select` macro since 9b36a04b. This
macro requires the async-await-macros and std feature of futures-util.
These features are the default features so let's just stop disabling the
default features for futures-util.
Signed-off-by: Damir Jelić <poljar@termina.org.uk>
Co-authored-by: Jonas Platte <jplatte@matrix.org>
This patch updates the `required_state` of `all_rooms` inside the
`RoomListService` to add `m.room.name`. Apparently, Synapse doesn't
always update the `response.rooms.*.avatar` field when the avatar is
updated. It's being investigated, but it doesn't hurt to ensure we get
it from the state events.
To fix the `test_room_avatar_group_conversation`, we need to ask for the
`m.room.avatar` state event from `required_state`. The rest of the patch
rewrites the test a little bit to make it more Rust idiomatic.
The `response.rooms.*.avatar` field from sliding sync should contain the
new avatar, but for the moment, it doesn't. It seems to be a bug.
To fix the `test_left_room`, we need to ask for the `m.room.member`
state event from `required_state`. The rest of the patch rewrites the
test a little bit to make it more Rust idiomatic.
Sliding sync expects all state events to be in `required_state`. State
events in `timeline` **must be ignored**. However, in sync v2, state
events in `timeline` **must be handled**.
In the sync response flow, both sliding sync and sync v2 uses the same
`handle_timeline` method. This patch adds an argument to ignore state
events. This is not ideal, but it's a temporary solution as a first
step. The next step is to refactor this code, but let's start easy.
The rest of the patch updates the tests accordingly.
With sliding sync, we must handle state events from `required_state`
only, not from `timeline`, this is a mistake as they might be incomplete
or _staled_.
I'm going to be replacing the inner structure of `TimelineEvent` with an
implementation that holds a `Raw<AnySyncTimelineEvent>`, rather than a
`Raw<AnyTimelineEvent>`. Prepare for that by changing the accessors to return
`Raw<AnySyncTimelineEvent>`.
... and add accessors instead.
Give `TimelineEvent` the same treatment we just gave `SyncTimelineEvent`: make
the fields private, and use accessors where we previously used direct access.
... and add accessors instead.
I'm going to change the inner structure of `SyncTimelineEvent`, meaning that
access will have to be via an accessor in future. Let's start by making the
fields private, and use accessors where we previously used direct access.
I'm going to change the internal structure of `SyncTimelineEvent`, and since
it implements `Deserialize`, we need to not break it. Let's add a test for the
current format.
`bootstrap_cross_signing` holds a lock on the private identity. In case
a new identity is created, it will try to acquire a lock on `account`.
The latter is locked by `sync`, which tries to acquire a lock on the private identity.
Note that the `bootstrap_cross_signing` call is executed in a separate
task e.g. in `restore_session`. In particular, this task and `sync` both
race to acquire locks described above.
Signed-off-by: boxdot <d@zerovolt.org>
This is useful for the knocking feature since we'll be able to differentiate between rooms that you were just invited from rooms that you knocked and then were granted access, or rooms that you left and rooms where your knocking attempt was rejected.
The `mark_room_as_*` functions have been updated so they reuse the same `set_state` function underneath, which only updates the previous state if the new one doesn't match.
Was added post-merge to the MSC for servers that support
authenticated media but do not support all of Matrix 1.11 yet.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch changes the behaviour of `timeline_limit` in sliding sync
requests. It previously was sticky, but since it's now mandatory
with MSC4186, it's preferable it to be non-sticky, otherwise in
some scenarios it might default to 0 (its default value). How?
If the server doesn't reply with our `txn_id` (because it doesn't
support sticky parameters or because it misses a `txn_id`), the
next request will be built with a default `timeline_limit` value,
which is zero, and won't get updated to the `timeline_limit` value
from `SlidingSyncListStickyParameters`. This is not good. Instead,
we must consider `timeline_limit` as non-sticky, and moves it from
`SlidingSyncListStickyParameters` to `SlidingSyncListInner`. This is
what this patch does.
The content of a poll timeline item goes to the content directory. The
data structure handling pending poll events goes into state.
No functional changes, only code motion.
It can occur that the data contains rooms that were forgotten.
Knowing that we now update that data after every sync, that creates
a lot of noise in the logs.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
The `handle_reaction_redaction` method is called by `handle_redaction`
for every single redaction event that we receive as a first step to
check if the redaction matches a reaction.
It means that not finding a reaction to redact is perfectly fine and is not worthy of a warning.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This old method was checking invariants that were
spooky-action-at-a-distance: these invariants have changed since then,
so this would panic instead of returning a proper error to the caller.
Signed-off-by: oliverw@element.io
---------
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
This was because we used a `BTreeSet`, which doesn't make sense anymore
since the data part of the key got mangled with some value unrelated to
the key itself.
This seems to be the only way to make the log rotation fix work and avoid build warnings like:
```
warning: patch for the non root package will be ignored, specify patch at the workspace root:
package: matrix-rust-sdk/bindings/matrix-sdk-ffi/Cargo.toml
workspace: = matrix-rust-sdk/Cargo.toml
Finished `dev` profile [unoptimized] target(s) in 0.30s
```
This improves parsing times in mobile Clients. On Android, this means a 5-10x faster parsing of timeline events.
To do that I had to:
- Make functions like `edit/redact/forward` take an identifier (EventId/TransactionId) instead of the actual event. This id will be used to look for the actual SDK timeline event in the timeline. This change will make these functions a bit less performant.
- Make `InReplyToDetails` an object instead since a record can't recursively contain itself.
- Turn `EventTimelineItem` into a record type. Do the same with `Message`, which is now `MessageContent`.
These tests were failing since the migration from the sliding sync proxy
to Synapse.
Since the previous fixes to re-enable other tests, these 2 passes for
free.
This test was failing since the migration from the sliding sync proxy
to Synapse.
This patch fixes the test. The failing parts were:
1. The `timeline_limit` wasn't set, so Synapse was returning an error,
2. The `unread_notifications` was set to 0 and could not be set to 1
because that's an encrypted room.
The fact `timeline_limit` is now mandatory has been mentioned in the MSC:
https://github.com/matrix-org/matrix-spec-proposals/pull/4186/files#r1775138458
A patch in Ruma has been created. The previous patch in this repository
also contains the fix for the SDK side.
The assertions around `unread_notifications` have been removed. We no
longer use this API anymore (and it should be deprecated by the way).
Since MSC4186, the `timeline_limit` value is required.
This patch uses 1 as the default value for `timeline_limit`, and forces
the `timeline_limit` to be defined everywhere.
When Bob receives the invite, the room has the correct name. Bob to sync
more to receive the new name. This is not a bug.
This patch updates the `CreateRoomRequest` to set the correct name
immediately.
This test was failing since the migration from the sliding sync proxy
to Synapse.
This patch fixes the test. The failing part was:
```rust
assert_eq!(notification.joined_members_count, 1);
```
This patch changes the value from 1 to 0. Indeed, Synapse doesn't share
this data for the sake of privacy because the room is not joined.
A comment has been made on MSC4186 to precise this behaviour:
https://github.com/matrix-org/matrix-spec-proposals/pull/4186#discussion_r1774775560.
Moreover, this test was asserting a bug (which is alright), but now
a bug report has been made. The patch contains the link to this bug
report.
The code has been a bit rewritten to make it simpler, and more comments
have been added.
Instead of forcing the room encryption to be known when the timeline is created and failing if it's not known, take the latest room encryption info as a base value and update it when processing timeline events.
At the time of writing this commit, the encryption info is only used to decide whether shields should be calculated for timeline items or not.
- explain what these tests are
- mention that it's sometimes needed to rebuild the synapse image
---------
Signed-off-by: Benjamin Bouvier <benjamin@bouvier.cc>
Co-authored-by: Ivan Enderlin <ivan@mnt.io>
This patch updates `merge_stream_and_receiver` to display an `error!`
when the room info receiver reads an error, like `Closed` or `Lagged`.
This is helpful when debugging.
The NotificationClient, responsible for handling, fetching, and
potentially decrypting events received via push notifications, creates a
copy of the main Client object.
During this process, the Client object is adjusted to use an in-memory
state store to prevent concurrency issues from multiple sync loops
attempting to write to the same database.
This copying unintentionally recreated the OlmMachine with fresh data
loaded from the database. If both Client instances were used for syncing
without proper cross-process locking, forks of the vodozemac Account and
Olm Sessions could be created and later persisted to the database.
This behavior can lead to the duplication of one-time keys, cause
sessions to lose their ability to decrypt messages, and result in the
generation of undecryptable messages on the recipient’s side.
This patch adds a new `cancel_in_flight_request` argument to
`SlidingSync::subscribe_to_rooms`, which tells the method cancel the
in-flight request if any.
This patch also updates `RoomListService::subscribe_to_rooms` to turn
this new argument to `true` if the state machine isn't in a “starting”
state.
The problem it's solving is the following:
* some apps starts the room list service
* a first request is sent with `pos = None`
* the server calculates a new session (which can be expensive)
* the app subscribes to a set of rooms
* a second request is immediately sent with `pos = None` again
* the server does possibly NOT cancel its previous calculations, but
starts a new session and its calculations
This is pretty expensive for the server. This patch makes so that the
immediate room subscriptions will be part of the second request, with
the first request not being cancelled.
This method is now private inside `matrix_sdk_ui` and removed
from `matrix_sdk_ffi`. This method is returning a stream of rooms,
but updates on rooms won't update the stream (only new rooms
will be seen on the stream). Nobody uses it as far as I know, and
`entries_with_dynamic_adapters` is the real only API we want people
to use.
This field is helpful as it tells us the sequence number of the message in the
megolm session, which gives us a clue about how long it will have been since
the session should have been shared with us.
This fn will loop until it finds an at least partially synced room with the given id. It uses the `ClientInner::sync_beat` listener to wait until the next check is needed.
This patch adds `m.room.canonical_name` in the `required_state` of the
`all_rooms` list defined by `RoomListService`.
This is useful to better compute the room name in a more robust way.
## Changes
Takes care of [this
TODO](https://github.com/matrix-org/matrix-rust-sdk/blob/9df1c480795c42afcee39e4c7e553d5a927a2680/crates/matrix-sdk-ui/src/timeline/mod.rs#L520).
- sdk & sdk-ui: unify `Timeline::edit` and `Timeline::edit_polls`, the
new fn takes an `EditedContent` parameter now, which includes a
`PollStart` case too.
- ffi: also unify the FFI fns there, using `PollStart` and a new
`EditContent` enum that must be passed from the clients like:
```kotlin
val messageContent = MessageEventContent.from(...)
timeline.edit(event, EditedContent.RoomMessage(messageContent))
```
Since the is mainly about changing the fns signatures I've reused the
existing tests, including one that used `edit_poll` that now uses the
new fn.
---------
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
There is a non-negligible difference MSC3575 and MSC4186 in how the
`e2ee` extension works. When the client sends a request with no `pos`:
* MSC3575 returns all device lists updates since the last request
from the device that asked for device lists (this works similarly to
to-device message handling),
* MSC4186 returns no device lists updates, as it only returns changes
since the provided `pos` (which is `null` in this case); this is in
line with sync v2.
Therefore, with MSC4186, the device list cache must be marked as to be
re-downloaded if the `since` token is `None`, otherwise it's easy to
miss device lists updates that happened between the previous request and
the new “initial” request.
This patch adds the `OldMachine::mark_all_tracked_users_as_dirty`.
This patch rewrites a bit `OlmMachine::new_helper` by extracting some
piece of it inside `OlmMachine::new_helper_prelude`. With that, we
can rewrite `OlmMachine::migration_post_verified_latch_support` to use
`IdentityManager::mark_all_tracked_users_as_dirty`.
This latter is the shared implementation with
`OlmMachine::mark_all_tracked_users_as_dirty`.
This patch adds a test for `OlmMachine:mark_all_tracked_users_as_dirty`.
This patch improves `Client::available_sliding_sync_versions` when
trying to detect the sliding sync proxy. Previously, we were relying
on the `Client::server` to send the `discover_homeserver::Request`.
Sadly, this value is an `Option<_>`, meaning it's not always defined
(it depends how the `Client` has been built with `HomeserverConfig`:
sometimes the homeserver URL is passed directly, so the server cannot
be known).
This patch tries to find to discover the homeserver by using
`Client::server` if it exists, like before, but it also tries by using
`Client::user_id`. Another problem arises then: the user ID indeed
contains a server name, but we don't know whether it's behind HTTPS or
HTTP. Thus, this patch tries both: it starts by testing with `https://`
and then fallbacks to `http://`.
A test has been added accordingly.
The term session is usually only used in the crypto crate to reference a
Megolm session, the rest of the SDK uses the name from the event and the
Matrix spec, this should lower the amount of confusion since the main
crate has already a session concept and its unrelated to end-to-end
encryption.
My theory is that the intermittent failure depends on the ordering of
the requests, and if the /keys/upload request happened before the key
backup request, then after failing the next key backup request wouldn't
run.
This is likely a small typo that the key upload returns a 404 error
instead of a 200, let's see if this improves the situation.
It does not make much sense to create an UI client that does not support
end-to-end encryption, besides disabling the feature was broken for
quite some time.
* Not verifying the remote device/user is normal: log it at debug rather than
info.
* On the other hand, if we do verify something, let's log that at info rather
than trace.
Also fix a comment, while we're here.
* Upgrade the log when we get the "reciprocate" message (which tells us the
other side has scanned our QR code) to debug, instead of trace.
* Warn if we get a reciprocate we don't understand
* Log when the user confirms that the other side has scanned successfully.
This is the message that tells us whether the other side wants to do QR code or
SAS (emoji) verification. Knowing which they have chosen is really helpful for
following the flow!
Attach the flow_id (the transaction ID or message ID from the `request`
message) to the span, so that it is displayed alongside loglines that happen
when processing the request.
Take the logging that happens when a QR code verification is added to the
`verification cache`, and push it down to the `VerificationCache` itself. Doing
so means that we will log when we *show* a QR code as well as when we scan it.
I would have found this helpful when trying to debug a verification flow this
week.
For debugging, it's useful to have a record of what we believe our own public
cross-signing keys to be. Currently, we log the keys at startup if we restore
them from the database, but if we subsequently create, or download, a set of
keys, they aren't logged.
When we receive an `/keys/query` response, look for existing
inboundgroupsessions created by updated devices, and see if we can update any
of their senderdata settings.
This module has a number of useful types (in particular, error types). Rather
than addding even more types to the top level module, let's export the
`sender_data_finder` module as a whole.
Turns out that most of the places we call `find_using_device_keys`, we already
have a DeviceData. So we might as well pass that in directly, rather than
extracting the device keys and then rebuilding a DeviceData.
This patch renames `HomeserverConfig::Url` to `HomeserverUrl`, and
`HomeserverConfig::ServerNameOrUrl` to `ServerNameOrHomeserverUrl`.
Funnily, the methods on `ClientBuilder` doesn't need to be renamed to
match the new naming since they already use this naming!
When subscribing to an already subscribed room, the subscription state
was reset, the subscription was resent by cancelling the in-flight
request. All this is useless and can create a feeling of lag.
This patch checks if a room is subscribed first. If it is, nothing
happens. If it is not, the room subscription is created, and the
in-flight request is cancelled.
Tests are updated to reflect this new change.
Note: room subscription settings are not taken into account in this
“presence” pattern. It means that if we subscribe to an already
subscribed room but with different settings, nothing will happen. The
server will ignore the new settings anywhere for the moment.
This patch moves `client/builder.rs` into `client/builder/mod.rs`.
This patch also moves the `HomeserverConfig` type and its siblings
(`discover_homeserver`, `UrlScheme` etc.) into its own module `client/
builder/homeserver_config.rs`.
This is purely for cleaning up.
This patch adds `Client::server`: the URL of the server.
Not to be confused with the `Client::homeserver`. The `server`
holds some information, like the `.well-known` file, to discover the
homeserver. The homeserver is the client-server Matrix API.
`server` is usually the server part in a user ID, e.g. with
`@mnt_io:matrix.org`, here `matrix.org` is the server, whilst
`matrix-client.matrix.org` is the homeserver.
This patch also moves the code about homeserver discovery in the
`HomeserverConfig::discover` new method. A new struct is introduced to
hold the result, to replace a 4-tuples.
`Client::server` is also now a `Option<_>` because in the case of
`HomeserverConfig::Url`, the server cannot be known.
This patch also removes several clones here and there.
Finally, this patch updates a test to quickly test the new behaviour. A
next patch will introduce proper tests.
Especially for remote items, it should be in sync with `should_add` as
it's used in this method, otherwise read receipt tracking will not work
correctly.
This will only apply to `async_test` functions, but I think this is a
win:
1. for consistency within the codebase, since I've started doing so in
many places,
2. because these function names will clearly identify these functions as
tests, in the call tree interfaces, when rendered using the LSP
show-callers/show-callees functionality.
Including non-live timelines (pinned event timelines and permalinked
timelines). This makes it possible to see that you're adding a reaction
etc. in real time, while it wasn't the case anymore.
Fixes#3906.
This was copied from SqliteStateStore, but the reason
that they are separated there is because some migrations
require the store cipher.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Setting the version number only when all migrations are done
means that the version will be wrong if a migration fails.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Otherwise, this would mean that logged out clients would infinitely
repeat network requests failing in the background.
Without this fix, the added test will time out, endlessly reattempting
network requests.
This patch adds bindings to `Client::available_sliding_sync_versions`
to `matrix-sdk-ffi`.
This patch also moves `Client::sliding_sync_version` from “private” to
“public” FFI API, in thee sense that this method is now exported with
UniFFI.
Previous patches have unified all sliding sync versions behind a
single type: `Version`. More recent previous patches have introduced
`VersionBuilder` so that a `ClientBuilder` can use them to coerce or
find the best `Version` possible. This patch implements a last missing
piece: `Client::available_sliding_sync_versions` will report all
available `Version`s at a given time. This is useful when a `Client`
is already built, and a session has been opened/a user is logged,
but someone has to take the decision whether it's useful to switch to
another sliding sync version or not.
This patch adds a builder for `sliding_sync::Version`. It is a similar
enum except that it has `DiscoverProxy` and `DiscoverNative` to
automatically configure `Version::Proxy` or `Version::Native`.
This patch changes the type of
`discover_homeserver_from_server_name_or_url`. It now returns a `Url`
instead of a `String` for the homeserver URL. It also returns an
`Option<get_supported_versions::Response>` in addition to the other
values.
The change from `String` to `Url` is necessary to avoid a
double-parsing. It was parsed in `build()` but previously in
`discover_homeserver_from_server_name_or_url` in the last branch.
The addition of `get_supported_versions::Response` is necessary for the
next patch. It's going to be helpful to auto-discover sliding sync in
Synapse. The change happens here because
`get_supported_versions::Response` is already received in
`discover_homeserver_from_server_name_or_url`. This patch makes it easy
to re-use it so that the request is sent only once.
This patch therefore changes `check_is_homeserver` a little bit to
become `get_supported_versions`, and inlines its previous call inside
`discover_homeserver_from_server_name_or_url`.
This patch replaces all the API using simplified sliding sync, or
sliding sync proxy, by a unified `sliding_sync::Version` type!
This patch disables auto-discovery for the moment. It will be re-enable
with the next patches.
If we're building for the wasm architecture, jump through the hoops to
tell wasm_bindgen about `ShieldStateCode`. This solves the need to
declare an identical copy of `ShieldStateCode` in the wasm bindings.
This is part of https://github.com/matrix-org/matrix-rust-sdk/pull/3662,
pulled out to into a separate PR. Recent changes in `main` made it
pretty much impossible to merge this section of code from `main` into
that PR, and Rich wanted to see the refactoring bits separate from the
behavioural changes. So I've re-written the refactoring.
Pulls the `match` on `sharing_strategy` outside of the `for` loop, and
moves any code that is specific to one strategy into the appropriate
branch.
This commit contains a new `assert_next_matches_with_timeout!` macro that will take a `Stream` and wait for a little while until its next item is ready, then match it to a pattern.
It was found that it's making the whole build slower because it's
hitting a pathologically slow path in type-checking. Considering that it
doesn't do much, let's get rid of it and inline it instead.
After this, compiles times are reduced from 30 seconds to 22 seconds on
my machine
This patch uses the new `RoomSubscriptionState` enum to filter
room subscriptions that have already been sent, when building a
`http::Request` with the sticky parameters.
By default, to start, a `http::request::RoomSubscription` is in the
state `Pending`, i.e. it's not sent yet. Once the sticky parameters are
committed, the state is updated to `Applied`. When the sticky parameters
are applied, only the `Pending` room subscriptions are added.
This patch contains one test to specifically assert this behaviour.
This patch adds the `commit` method on the `StickyData` trait. It is
called by `SlidingSyncStickyManager::maybe_commit` when we are sure the
data can be validated because of a valid response to the sent request.
## Changes
- Creates a separate `AllEventsCache` struct holding both the actual all
events cache map and the relationship one. This new cache wrapper also
holds a single `RwLock` to both inner caches, as they are independent
from each other.
- When a new event is saved either from `/sync` or another HS request,
it'll be saved to both the 'all events' cache and the relationship map.
- Add tests for the relationship cache.
Allows to save media in a different path than the state store.
This adds a "last_access" field to the SQLite implementation, to prepare
for future work on a media retention policy.
This removes the IndexedDB media cache implementation, because as far as
I know it is currently unused, and I have no idea how to implement
efficiently the planned media retention policy with a key-value store.
Closes#1810.
- [x] Public API changes documented in changelogs (optional)
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch clears all sliding sync room subscriptions when a session
expires. Indeed, we might not want to request all room subscriptions
when the session restarts. Imagine if the client has subscribed to 400
rooms and the session expires: once the session restarts, it will ask
for 400 room subscriptions, which is a lot and will result in a quite
slow response.
We're finding ourselves in the situation in which we can't interact with
invites through normal Room APIs as `full_room`s can't be build from the
RoomListItem. Full rooms require the timeline to be configured before
use and the timeline can't be configured because encryption cannot be
fetched for invited rooms on homeservers that have previews disabled
(see #3848 and #3850)
In response we now expose the room's membership directly from the
`RoomListItem` so that the final client can chose which of the 2 rooms
types (invited or full) to ask for before using aforementioned APIs.
Powers https://github.com/element-hq/element-x-ios/pull/3189
This patch updates when sliding sync requests have a `timeout`.
Prior to this patch, all requests had a `timeout` query, set to the
`poll_timeout` duration value. However it means: if there is no data
to return, wait `timeout` milliseconds for new data before returning.
This definition is correct. Problem: if the current range of a list
has no data, the server will wait! It means that, in a situation where
there is no update at all, but the client is fetching all rooms batch by
batch, it will wait `poll_timeout` for each batch!
The behaviour described above is absolutely correct. Some server
implementations are less strict though, and we didn't realise our code
was doing that, because the server had some optimisations to ignore the
timeout if the range wasn't covering all the rooms. Nonetheless, a new
server implementation (namely Synapse) is strict, and it confirms we
have a bug here.
This patch then configures a `timeout` if all lists require a timeout,
otherwise there is no `timeout`, which is equivalent to `timeout=0`.
- this prevents issues where spoofing the sender field is enough to spoof and edit and display wrong decorations in the app
- fixes matrix-org/internal-config/issues/1549
Most times we pulled an InboundGroupSession from the sqlite DB, we were
overriding whatever value for `backed_up` was stored inside the pickled
value, and using the value stored in the SQL column.
But when we pulled a single InboundGroupSession from the DB by ID, we
did not override it.
I am fairly sure this was an accidental oversight, so this change
corrects it, and unifies the code with other places we create these
objects.
It seems that the fact that matrix-rust-sdk contains `olm-rs` in its
`Cargo.lock` sparked panic, so let's attempt to fend off future concerns by
adding a comment.
This patch fixes a bug in the tracing system. It introduces one
fields formatter _per layer_ to force the fields to be recorded in
different span extensions, and thus to remove the duplicated fields in
`FormattedFields`.
The patch contains links to the bug report in `tokio-rs/tracing`. This
patch is a workaround.
This patch removes `NotificationSettings::subscribe_to_changes` because
it's not used anywhere in our code except in tests. It is indeed part of
the public API but I'm not aware of anyone using it for the moment. It
only adds complexity in the code.
This patch replaces `RoomInfo::user_defined_notification_mode` by
its cached variant: `cached_user_defined_notification_mode`, and
call `Room::cached_user_defined_notification_mode` which will boost
performance when computing a new `RoomInfo`.
This patch splits/copies the
`test_room_info_deserialization_without_optional_items` test into the
same test + `test_room_info_deserialization`.
It appears that the _without optional items_ part has been forgotten.
In the past, the test has been updated to test optional items. The
initial idea (based on my understanding of the comments) is to test
potentially old `RoomInfo` can still be deserialized today. So this test
must never be changed, except if a non-optional field is added.
For this reason, this patch removes from this test the assertions about
optional fields. A new test, named `test_room_info_deserialization` is
created, and tests all the fields, including the optional ones.
One important thing:
`test_room_info_deserialization_without_optional_items` now runs
even if the `experimental-sliding-sync` feature is absent. It was
required only because of `latest_event`, but that's an optional
field! However, the `test_room_info_deserialization` requires the
`experimental-sliding-sync` feature, as it tests `latest_event` but
also `recency_stamp`.
Finally, `test_room_info_deserialization` tests
`cached_user_defined_notification_mode`.
This patch updates `matrix_sdk::Room::user_defined_notification_mode`
to cache its result if some mode has been found. The cached result can
be retrieve with
`matrix_sdk_base::Room::cached_user_defined_notification_mode`.
Use a for loop rather than `partition_map`. We're about to add a third list, so
partition_map won't work.
(partition_map ends up using Vec::push under the hood, so this is pretty much
equivalent.)
Update Ruma dependency to expect call membership state events with state
keys that are arbitrary strings, not just pure MXIDs.
When a call membership state key does not exactly match the format of an
MXID, treat it as a valid state key if it starts with an MXID followed
by an underscore, with that MXID designating the owner of the event.
(The state key may also be optionally prefixed with an underscore, which
is permitted as a way to bypass pre-MSC3757 authorization rules against
sending state events with state keys that do not exactly match the
sender's MXID.)
---------
Signed-off-by: Andrew Ferrazzutti <andrewf@element.io>
Co-authored-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <benjamin@bouvier.cc>
The timeline already listens to changes to the pinned events list (via a
stream), so there's no need to fully reload all the pinned events every
time we receive a new event that's pinned. Technically it may avoid one
or a few lookups, but this is cheap and a subsequent commit/PR will
merge the pinned event cache into the event cache.
This patch restricts the call to `compute_limited` to the sliding sync
proxy implementation (aka MCS3575). It is not necessary for the sliding
sync native implementation (aka Simplified MSC3575). The proxy doesn't
implement the `limited` flag, contrary to Synapse. Let's not run
workarounds when we don't need them.
The patch https://github.com/matrix-org/matrix-rust-sdk/pull/2395 has
introduced `SlidingSyncInner::past_positions` as a mechanism to filter
duplicated responses. It was a problem because the sliding sync `ops`
could easily create corrupted states if they were applied more than
once.
Since https://github.com/matrix-org/matrix-rust-sdk/pull/3664/, `ops`
are ignored.
Now, `past_positions` create a problem with the sliding sync native
implementation inside Synapse because `pos` can stay the same between
multiple responses.
While `past_positions` was helpful to fix bugs in the past, it's no
longer necessary today. Moreover, it breaks an invariant about `pos`: we
must consider it as a blackbox. It means we must ignore if a `pos` value
has been received in the past or not. This invariant has been broken for
good reasons, but it now creates new issues.
This patch removes `past_positions`, along with the associated code
(like `Error::ResponseAlreadyReceived` for example).
This patch changes the visibility of
`SlidingSyncList::invalidate_sticky_data` from `pub` to `pub(super)`.
This is the only place where it must be accessible from.
This patch asserts that when subscribing to a new room, the old room
subscriptions are still present. Is it the behaviour we want? Probably
not, but this is the standard behaviour right now, and we need to assert
it.
We had *two* copies of `response_from_file`, and all calls to them were always
immediately followed by an operation to parse the response as a Ruma response
object.
We can save a whole lot of boilerplate with a generic function that wraps the
json into an HTTP response *and* parses it into a Ruma object.
We weren't updating the database schema version immediately after the v10 -> v11
migration. This was fine in practice, because (a) for now, there is no v12
migration so we ended up setting the schema version immediately anyway; (b) the
migration is idempotent.
However, it's inconsistent with the other migrations and confusing, and is
about to make my test fail, so let's clean it up.
It's better to have fewer public APIs, especially when there's little
annoyance to have it. We could use a request builder that converts into
a Future, too, but considering there's only a single optional parameter,
it's fine to include it in the function's signature.
We should only show the spinner if the *first* sliding sync request is
taking a while. If we have received some data and the second request
takes a while, that is OK.
For the state transition of `Init -> SettingUp` this is handled
correctly, however for `Terminated -> Recovering -> Running` we waited
until the second request returned before hiding the sync spinner. This
meant that if the first request returned quickly the app would show new
data and *then* the sync spinner would show (if the second request took
time).
This situation occurs frequently with the new SSS API, where if all the
new data was returned in the first sync then the second sync would
block waiting for new data, triggering the sync spinner.
This patch changes the `SlidingSync::subscribe_to_room` method to
`subscribe_to_rooms`. Note the plural form. It's now mandatory to
subscribe to a set of rooms. The idea is to avoid calling this method
repeatedly. Why? Because each time the method is called, it sends a
`SlidingSyncInternalMessage` of kind `SyncLoopSkipOverCurrentIteration`,
i.e. it cancels the in-flight sliding sync request, to start over with
a new one (with the new room subscription). A problem arises when the
async runtime (here, Tokio) is busy: in this case, the internal message
channel can be filled pretty easily because its size is 8. Messages
are not consumed as fast as they are inserted. By changing this API:
subscribing to multiple rooms will result in a single internal message,
instead of one per room.
Consequently, the rest of the patch moves the `subscribe` method of
`room_list_service::Room` to `room_list_service::RoomListService`
because it now concerns multiple rooms instead of a single one.
Our CI used to be quite slow and used up a lot of CI time, so as an
optimization we disabled CI for draft PRs.
This is a bit annoying since people want to open draft PRs to check if
CI passes, but without triggering any review requests.
Since our CI is nowadays a bit more efficient let's see if we can enable
it for draft PRs.
This method works the same as `Room::event` but you can provide a custom `RequestConfig` to it.
It's especially useful for the pinned events timeline, since we need a max number of retries and a max number of concurrent requests. With this we can remove some unnecessary complexity.
This way it matches the rest of timelines in the SDK, I reversed it here because I didn't realise most clients just do this reversal of ordering themselves. As they do, they need the same order for this timeline too to be able to reuse their existing logic.
This ensures the cache keeps the events even when the associated `Room` is dropped, which is what we want when using it to cache the pinned events for rooms in the client.
Add `fn Client::pinned_event_cached()` to get a reference to it.
In `matrix.rs` we add methods to interact with the matrix homeserver. And in `machine/mod.rs` we implement the widgetMachine cases for handling (checking capabilities and using the matrixDriver) delayed events.
Add `PinnedEventsLoader` to encapsulate this logic, being able to provide a max number of events to load and how many can be loaded at the same time. Also implement an event cache for it.
Add `PinnedEventsRoom` trait to use it in the same way as `PaginableRoom`, only for pinned events.
The code used to only add new targets to rooms
but never remove the ones that are not in the event anymore.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch ensures that we retain the master key for a given UserIdentityData object, even when a new and different identity arrives via the `/keys/query` endpoint. This concept, called pinning, is similar to certificate pinning in web browsers.
Retaining the master key allows us to detect changes and notify the user accordingly.
This tiny change allows one to easily name the return type of
`EventTimelineItem::reactions()` such that you can use it in a function
signature. Same goes for the `ReactionInfo` type.
I think this was likely just a small oversight in recent changes to the
reactions API, so hopefully it's not controversial.
Without this, it's impossible to write a function that uses
`ReactionsByKeyBySender` or `ReactionInfo` in the signature.
Signed-off-by: Kevin Boos <kevinaboos@gmail.com>
- disable backups and recovery before requesting the reset handle
- attempt device key upload
- re-enable backups both when UIAA is required and when not
This means we have all the information inside SenderData to populate
VerificationStatus and DeviceId for EncryptionInfo, so we can share the
code between SenderDataFinder and get_verification_state.
This makes it impossible to represent states like "there's a local *and*
a remote echo for the same sender for a given reaction", or multiple
reactions from the same sender to the same event, and so on.
Remove `Result` where it is not needed, and switch to `CryptoStoreError`
instead of `OlmError` where possible. Soon, this will allow us to call
some of these methods from places that don't know about `OlmError`.
These calls are equivalent because the old code called
`self.get_user_devices` with a `timeout` of `None`, which meant the call
to `wait_if_user_pending` inside was a no-op.
As it says on the tin. Needed the functions but they were missing. They are analogous to the GlobalAccountData setters.
Signed-off-by: Benjamin Kampmann <ben@acter.global>
The sliding sync proxy has a bug: despite the presence of
`m.room.create` in `bump_event_types`, an invite with an `m.room.create`
event will not have a `timestamp`. Thus, such a room cannot be sorted
reliably.
This patch fixes this problem with a terrible hack, where it tries to
find an `origin_server_ts` value from within the `invite_state` state
events.
Please read the comment to learn more.
The `UserIdentity::is_verified()` method in the matrix-sdk-crypto crate
before version 0.7.2 doesn't take into account the verification status
of the user's own identity while performing the check and may as a result
return a value contrary to what is implied by its name and documentation.
This patch fixes this and adds a regression test.
The method itself is not used internally and as such has not a larger
impact.
Co-authored-by: Denis Kasak <dkasak@termina.org.uk>
Signed-off-by: Damir Jelić <poljar@termina.org.uk>
The `RoomList` provides a `Stream<Item = Vec<VectorDiff<Room>>>`.
This `Stream` receives updates from 2 sources: `RoomList::entries`,
and `Receiver<RoomInfoNotableUpdate>`. When a `RoomInfo` is
updated, a notable update is emitted and broadcasted. The
`RoomList` was filtering these notable updates by _reasons_ (namely
`RoomInfoNotableUpdateReasons`).
This is great and it's a good idea since we can filter which
`RoomInfoNotableUpdate` will trigger a `RoomList` update. However, too
many _reasons_ were hidden/implicit, and it creates several regressions
because (i) these _reasons_ were implicit, (ii) since the business rules
are not defined, there is no tests for that (not in this SDK, not in
apps like ElementX). It means we discover missing _reasons_ bug after
bug. It's not pleasant.
The reality is: we are in the middle of big changes, mostly with room
list client-side sorting and simplified sliding sync. We want to relax
a little bit. This patch then disable the feature _filter updates by
reasons_. The `RoomList` will update to all `RoomInfoNotableUpdate` for
the moment. We will get back to this optimisation later.
The `UserIdentity::is_verified()` method in the matrix-sdk-crypto crate
before version 0.7.2 doesn't take into account the verification status
of the user's own identity while performing the check and may as a result
return a value contrary to what is implied by its name and documentation.
This patch fixes this and adds a regression test.
The method itself is not used internally and as such has not a larger
impact.
Co-authored-by: Denis Kasak <dkasak@termina.org.uk>
Signed-off-by: Damir Jelić <poljar@termina.org.uk>
Simplified sliding sync no longer has `sort` or `bump_event_types`
because they are static values on the implementation/server side. This
patch removes them here.
This patch renames “recency timestamp” to “recency stamp”. It prepares
the fact that simplified sliding sync has a `bump_stamp` instead of a
`timestamp`. The notion of _timestamp_ must be removed.
This patch extracts most of `SlidingSync::sync_once` into a
method named `SlidingSync::send_sync_request`. The name mimics
the `SlidingSync::generate_sync_request` similar method: first we
_generate_, then we _send_.
The `SlidingSync::send_sync_request` is generic over the `Request` and
`::sync_once` passes the correct type depending of whether Simplified
MSC3575 is enabled.
This patch is the first over two patches to change the support
of Simplified MSC3575 from a compiler feature flag to a runtime
configuration. This configuration is held by the `Client`.
By default, it's turned off inside `matrix-sdk-ffi` and
`matrix-sdk-integration-testing`, otherwise it's turned on.
Currently on Element X if you receive a sticker in a room, the room list
will show the room as updated but it will show the latest event that is
not a sticker. This change fixes that.
Signed-off-by:
Marco Antonio Alvarez <surakin@gmail.com>
---------
Signed-off-by: Marco Antonio Alvarez <surakin@gmail.com>
Signed-off-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <public@benj.me>
New feature from Matrix 1.11.
- [x] Public API changes documented in changelogs (optional)
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
There were some leftovers from the rename of the ReadOnlyDevice and
identity structs. The store still referenced them.
This gets rid of all the mentions and improves the documentation of the
store methods for devices and identities.
This reduces the work we do to calculate changed devices etc. when DEBUG
logging is not enabled, but more importantly (to me) it makes clear
that this code is only used for logging.
It is possible to remove the m.call capability because we now have
merged and released a version that does not request it anymore on
call.element.io
---------
Signed-off-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <public@benj.me>
This patch rewrites the `test_delayed_decryption_latest_event` test a
little bit. It does exactly the same things, but in a simpler way: it
removes multiple `sleep` and remove 2 sliding sync loops.
First off, the `SyncService` already starts the `RoomListService and the
`EncryptionSync` service. Both of them have their own sliding sync
loop. The test doesn't need other sliding sync loops in their own tasks,
this is not necessary at all: it's just pretty confusing and doesn't
reflect the reality, i.e. how these API are supposed to be used.
Second, it also tests the room for Bob is seen as encrypted.
Third, the `VectorDiff::Reset` is tested before the event from Bob
is sent. It's not only for clarity: it makes the test more robust for
future modifications.
Fourth, instead of waiting with a `sleep` for the event from Bob to be
received by Alice, we instead wait on the room list's stream of Alice to
receive an update. It's more robust this way and reflects the real usage
of this API. It also helps to remove an intermediate `assert_pending!`
that is no longer necessary because we are waiting on the stream just
after.
Finally, just like for the previous modification, this patch removes
another `sleep` for the to-device event from Bob to be received by
Alice, and instead wait on the room list's stream to receive an update.
It's again more robust and reflects the real usage of this API. Plus, it
makes the last `assert_pending!` macro to not be flaky.
This patch fixes a bug where rooms stored in the sliding sync cache
aren't restored by the `SlidingSyncBuilder`.
This patch also removes `SlidingSyncBuilder::rooms` fields which was
used but never modified. It's dead code.
Again, a transaction id received from the remote flow doesn't mean it
corresponds to a local echo sent *this particular session*, so no need
to warn about it.
This log can happen when an event is received, has a transaction id (in
the data received from sync), and doesn't have a corresponding timeline
item (be it local or remote). There's no reason to warn about this,
because this would happen in most cases, for new incoming events coming
from sync, and this pollutes the logs of rageshakes.
This patch removes the `test_room_info_notable_update_deduplication`
test. First off, it's flaky because sometimes Synapse lags, or sends
another events, which makes the test to fail. Second, the same feature
is tested inside the `matrix_sdk_ui::room_list_service` test suite,
with `test_room_sorting` and `test_room_latest_event`, and inside the
`matrix_sdk_base::sliding_sync` test suite, with a better granularity.
And lastly, this test doesn't test what it says: there is no room info
notable update deduplication whatsoever. I personally don't believe it
has ever existed. This test isn't necessary.
ReadOnlyDevice is not particularly useful as a description of why we
have two device types. This commit renames it into DeviceData, as this
struct is used to hold the device keys and additional local device data.
I'm not quite sure why it took me so long to come up with a better name.
Please forgive me past readers.
The test database was created using a slightly modified `oidc-cli`
example, to turn of the database encryption, on commit
d6dca91df86413b0cbf193a4be191835dd81862e
Since it's only used for remote events, and a local echo couldn't figure
that out anyways (since it's not sent yet, and that information makes
sense after sending).
This is needed to prevent the race condition where the invite request finished, the `/sync` one didn't fetch the new membership event yet and we send a message in the room. This message won't be encrypted for the newly invited user and will result in an UTD.
I added a new integration test and I can confirm this [complement-crypto test](https://github.com/matrix-org/complement-crypto/pull/98) now passes instead of being skipped.
Fixes#3622.
---
* fix(sdk): force room member reload after inviting a user
This is needed to prevent the race condition where the invite request finished, the `/sync` one didn't fetch the new membership event yet and we send a message in the room. This message won't be encrypted for the newly invited user and will result in an UTD.
* Use `room.mark_members_missing()` instead, add integration test
* Abort syncing before the test ends
* Resolve nit: else after a return
* Fix race condition where bob may try to join the room before the invite is received
* Remove double sync
This patch removes everything related to the computation of `ops`
from a sliding sync response. With the recent `RoomList`'s client-side
sorting project, we no longer need to handle these `ops`. Moreover, the
simplified sliding sync specification that is coming removes the `ops`.
A `SlidingSyncList` was containing a `rooms` field. It's removed by
this patch. Consequently, all the `SlidingSyncList::room_list` and
`::room_list_stream` methods are also removed.
A `FrozenSlidingSyncList` was containing the `FrozenSlidingSyncRoom`.
This patch moves the `FrozenSlidingSyncRoom`s inside
`FrozenSlidingSync`. Why is it still correct? We only want to keep the
`SlidingSyncRoom::timeline_queue` in the cache for the moment (until
the `EventCache` has a persistent storage). Since a `SlidingSyncList`
no longer holds any information about the rooms, and since `SlidingSync`
itself has all the `SlidingSyncRoom`, this move is natural and still
valid.
Bye bye all this code :'-).
The only thing is that our client builder allows setting only the server
versions, so this means the new field has to contain two `Option`al
fields. This causes a bit of churn, but isn't too bad in the end.
I've been debugging a cause of flakey complement-crypto tests for
about a month now. I was pretty convinced it was deadlocking
somewhere in the memory store `save_changes` code. With additional
logging, it's now clear that the there is an ABBA style deadlock
when `save_changes` is called at the same time as `get_state_events`.
I've also adjusted code for `get_user_ids` as it has a very similar
pattern and also acquires locks in reverse order to `save_changes`,
so is potentially vulnerable to this.
This patch avoids to emit a
`RoomInfoNotableUpdateReasons::RECENCY_TIMESTAMP` for rooms that are new.
Otherwise the entries in `matrix_sdk_ui::room_list_service::RoomList`
receive a `VectorDiff` because a new room is inserted, and then a
`VectorDiff` because the recency timestamp is updated. The second
`VectorDiff` is useless in this case.
First off, this patch removes the
`RoomInfoNotableUpdate::trigger_room_list_update` field. It is replaced
by a `reasons: RoomInfoNotableUpdateReasons` 8-bit unsigned integer. It
addresses the following issues:
1. When a subscriber receives a `RoomInfoNotableUpdate`, they have no
idea what has triggered this update.
2. In
`matrix_sdk_base::sliding_sync::BaseClient::process_sliding_sync_e2ee`,
we were triggering an update even if the latest event wasn't
modified: it is a false-positive, it was a bug and a waste of
resources. Now it's more refined, see the why below.
Second, this patch removes the second `trigger_room_list_update`
argument of `matrix_sdk_base::BaseClient::apply_changes`. This method
now knows where to find the reasons for the room info notable updates,
see next point.
Third, this patch adds a new
`matrix_sdk::StateChanges::room_info_notable_updates` field which is a
B-tree map between an `OwnedRoomId` and a
`RoomInfoNotableUpdateReasons`. The idea is that all places that receive
a `StateChanges` can also create a room info notable update with a
specific reason. This is a finer grained mechanism, and it avoids to
pass new arguments everywhere. It's cleaner.
Finally, it's easier than ever to add a new reason and to propagate it
to subscribers.
The patch renames `RoomInfoUpdate` to `RoomInfoNotableUpdate`.
The functions, methods or variables whose names start with
`roominfo_update(.*)`` are renamed `room_info_notable_update$1`.
* ffi: expose methods for SSO login
Signed-off-by: Johannes Marbach <n0-0ne+github@mailbox.org>
* Update bindings/matrix-sdk-ffi/Cargo.toml
Co-authored-by: Ivan Enderlin <ivan@mnt.io>
Signed-off-by: Johannes Marbach <n0-0ne+github@mailbox.org>
* Refactor code to use a separate object to complete the SSO login
* Remove superfluous .workspace
* Remove superfluous field name
* Fix formatting with nightly toolchain
* Fix clippy errors using nightly toolchain
* Move SSO methods over into Client as AuthenticationService will go away very soon
* Add login tests
* Assign tokio runtime and url getter
* Reformat
* Relocate parts of the code as per review comments
* Add url to debugging representation of SsoHandler
* Add example for login_with_sso_callback
* Use unwrap_or_default to avoid creating empty string
* Remove leftover dependencies
---------
Signed-off-by: Johannes Marbach <n0-0ne+github@mailbox.org>
Co-authored-by: Ivan Enderlin <ivan@mnt.io>
When a request fails because of the exponential backoff, it won't be
re-logged again. It would be useful, for the purposes of the send queue
notably, to see when a request is re-attempted.
Synapse returns a bare `{ "membership": "leave" }` as the content of a
room membership event (for leave membership changes and likely others).
In this case, it'd still be nice to have some kind of display
name/avatar URL to show in UIs; it's possible to reuse information from
the previous member event, if available.
Add a new `max_concurrent_requests` parameter in the `RequestConfig` limits the number of http(s) requests the internal sdk client issues concurrently (if > 0). The default behavior is the same as before: there is no limit on concurrent requests issued.
This is especially useful for resource constrained platforms (e.g. mobile platforms), and if your pattern might lead to issuing many requests at the same time (like downloading and caching all avatars at startup).
- [x] Public API changes documented in changelogs (optional)
Signed-off-by: Benjamin Kampmann <ben.kampmann@gmail.com>
It turns out this will cause a network request if the encryption info hasn't been loaded before, which is the case for opening a client in offline mode. It will slow down displaying the room list or loading the room info in general.
Turns out legacy unencrypted objects can take many forms, and we need to be
tolerant of them.
Also expose `deserialize_legacy_value` as a separate function, because we're
going to need to special-case it.
This patch revisits the need to trigger a room list update for
all changes of `RoomInfo`. For the moment, it reduces the scope to
`recency_timestamp` update.
This patch comes with a test to ensure things work as expected.
... and make `deserialize_value` handle both the old and new formats.
`maybe_encrypt_value` uses a much more efficient representation, so let's
migrate to that.
It's currently possible for integ test results to leak from one test run to the
next (for example, the indexeddb stores hang around in the browser), causing
bad test results.
Extend the test setup routine to clear out the store before the test starts.
This patch removes the `LatestEvent::cached_event_origin_ts`.
It's no longer necessary to cache this value as the
`matrix_sdk_ui::room_list_service::sorter::recency` sorter no longer
uses it.
This patch adds a new field in `RoomInfo`: `recency_timestamp:
Option<MilliSecondsSinceUnixEpoch>>`. Its value comes from a Sliding
Sync Room response, that's why all this API is behind `cfg(feature =
"experimental-sliding-sync")`.
This allows seeing the events directly from the room event cache. I'm
hoping this will give us some interesting insights about duplicates,
among other things.
Most tests would randomize the username when creating a
`TestClientBuilder`; make it the default, since it's a sensible choice,
and avoids interference between different tests / test runs.
A single test required an actual non-randomized username, so a specific
way to opt out from this new default behavior has been introduced.
* Use `IndexeddbSerializer` more widely in test code
reuse `IndexeddbSerializer::maybe_encrypt_value` instead of re-inventing it.
* Rewrite `StoreCipher::decrypt_value_base64_typed`
Instead of un-base64-ing and calling `decrypt_value_typed` (which deserializes
the result`), call `decrypt_value_base64_data` (which un-base64s before
decrypting but does not deserialize the result), then deserialize.
This makes it more symmetrical with `encrypt_value_base64_typed`, and helps me
get rid of `decrypt_value_typed` (which is barely used.)
* Fix docs on `StoreCipher::encrypt_value_base64_typed`
looks like they got C&Ped from `encrypt_value_typed`.
* Inline `StoreCipher::{encrypt,decrypt}_value_typed`
Each of these are quite simple, are only used in two places, and their
existence melts my brain.
* Rewrite `IndexeddbSerializer::maybe_{encrypt,decrypt}_value`
... to use `en/decrypt_value_base64_data` instead of
`en/decrypt_value_base64_typed`.
We have to have the de/serialization code for the unencrypted case anyway, so
using the higher-level method isn't helping us much.
* Remove unused `StoreCipher::{en,de}crypt_value_base64_typed`
Outside of tests, these things are totally unused.
This patch removes the `RoomListService::rooms` cache, since now a
`Room` is pretty cheap to build.
This cache was also used to keep the `Timeline` alive, but it's now
recommended that the consumer of the `Room` keeps its own clone of the
`Timeline` somewhere. We may introduce a cache inside `RoomListService`
for the `Timeline` later.
This patch rewrites `merge_stream_and_receiver` to switch the order
of `roominfo_update_recv` and `raw_stream`. The idea is to give the
priority to `raw_stream` since it will necessarily trigger the room
items recomputation.
This patch also remove the `for` loop with `Iterator::enumerate`, to
simply use `Iterator::position`: it's more compact and it removes a
`break` (it makes the code simpler to understand).
Finally, this patch renames `merged_stream` into `merged_streams`.
Complement uses the FFI `RoomList` API. Since the patch set modifies
this API, Complement is broken. We disable it and will re-enable it once
we have updated Complement.
This patch adapts the `RoomList` FFI API to the recent changes to
suport a `Stream<Item = RoomListItem>` instead of a `Stream<Item =
RoomListEntry>`. Behind the scene, it supports client side sorting for
the rooms but this is transparent for this API.
This patch also removes the `RoomListInput` enum as no input is
supporter anymore.
The `entries` method no long returns a `RoomListEntriesResult` but
directly a `TaskHandle`. The given listener will receive the initial
entries as a `VectorDiff::Append`, which first is simpler but also fixe
a potential race condition bug.
This patch adds a new `cached_event_origin_server_ts` field on
`LatestEvent`, which is a copy of the `origin_server_ts` of the inner
`SyncTimelineEvent`.
This patch removes the `visible_rooms` sliding sync list from
`RoomListService`. As we are taking the path of doing client-side
sorting, the ordering of the server-side will most likely always
mismatch the ordering of the client-side, thus using `visible_rooms`
with room indices make no sense (indices from server-side won't map
indices on the client-side, so room ranges from client-side won't map
what the server knows).
We used to use `visible_rooms` to “preload” the timeline of rooms in
the user app viewport, with a `timeline_limit` of 20. This should be
replaced by room subscriptions starting from now. For the moment, the
user of `RoomListService` is responsible to do that manually. Maybe
`RoomListService` will handle that automatically in the future.
There were two disconnected sources of truth for the state of event to
be sent:
- it can or cannot be in the in-memory `being_sent` map
- it can or cannot be in the database
Unfortunately, this led to subtle race conditions when it comes to
editing/aborting. The following sequence of operations was possible:
- try to send an event: a local echo is added to storage, but it's not
marked as being sent yet
- the task wakes up, finds the local echo in the storage,...
- try to edit/abort the event: the event is not marked as being sent
yet, so we think we can edit/abort it
- ... having found the local echo, it is marked as being sent.
This would result in the event misleadlingly not being aborted/edited,
while it should have been.
Now, there's already a lock on the `being_sent` map, so we can hold onto
it while we're touching storage, making sure that there aren't two
callers trying to manipulate storage *and* `being_sent` at the same
time.
This is pretty tricky to test properly, since this requires super
precise timing control over the state store, so there's no test for
this. I can confirm this avoids some weirdness I observed with
`multiverse` though.
Using `SendHandle::abort()` after the event has been sent would look
like a successful abort of the event, while it's not the case; this
fixes this by having the state store backends return whether they've
touched an entry in the database.
It could be that the last event in a room's send queue has been marked
as wedged. In that case, the task will sleep until it's notified again.
If the event is being edited, then nothing would wake up the task; a
manual wakeup might be required in that case.
The new integration test shows the issue; the last `assert_update` would
fail with a timeout before this patch.
Commit d41af396c implemented MSC4147, which puts the device keys into a
Olm message. It failed to adhere to the unstable prefix proposed in the
MSC:
> Until this MSC is accepted, the new property should be named
> org.matrix.msc4147.device_keys.
This patch fixes this.
This patch switches the way we update the recovery state upon changes in
the backup state. Previously two places updated the recovery state after
the backup state changed:
1. A method living in the recovery subsystem that the backup
subsystem itself calls.
2. An event handler which is called when we receive a m.secret.send
event.
The first method is a hack because it introduces a circular dependency
between the recovery and backup subsystems.
More importantly, the second method can miss updates, because the backup
subsystem has a similar event handler which then processes the secret we
received and if the secret was a backup recovery key, enables backups.
Depending on the order these event handlers are called, the recovery
subsystem might update the recovery state before the secret has been
handled.
The backup subsystem provides an async stream which broadcasts updates
to the backup state, letting the recovery subsystem listen to this
stream and update its state if we notice such updates fixes both
problems we listed above. The method in the first bullet point was
completely removed, the event handler is kept for other secret types but
we don't rely on it for the backup state anymore.
Where a string is clearly documented as a room ID, event ID or mxc URI,
use OwnedRoomId, OwnedEventId and OwnedMxcURI, respectively.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
We tend to use fixup commits quite a bit, and in the heat of the moment
we sometimes forget to squash them before the final merge.
This should prevent us from doing so.
This will still disable the room's send queue, but the embedder may then
decide whether to re-enable the queue or not, based on network
connectivity. Ideally, we'd implement some retry mechanism here too, but
since we're at a different layer than the HTTP one, we can't get that
"for free", so let the embedder decide what to do here.
This patch is quite big… `RoomList::entries*` now returns `Room`s
instead of `RoomListEntry`s. This patch consequently updates all the
filters to manipulate `Room` instead of `RoomListEntry`. No more
`Client` is needed in the filters.
This patch also disables the `RoomList` integration test suite in order
to keep this patch “small”.
For each recipient device, we keep an "Olm wedging index" that indicates (approximately) how many times they tried to unwedge the Olm session. When we share a Megolm session, we store the Olm wedging index. This allows us to tell whether the Olm session was unwedged since we shared the Megolm session, so that we know if we should re-share it.
We detect an attempt to unwedge the Olm session by checking if the Olm message that we decrypted is from a brand new Olm session. This is not entirely accurate, since this could happen, for example, if Alice and Bob create Olm sessions at the same time. This may result in some Megolm sessions being re-shared unnecessarily, but should not make a huge difference.
It also resets the wedged state, so that the queue will retry to send
this event later. This will show useful in the following case: when an
event is too big, we can now retry to send it, even if it was blocked,
by splitting the event instead of copy/abort/recreate it.
It turns out that so as to be able to read the room ids, they need to be
*values*, not only *keys* (since keys are one-way hashed). This means we
need to duplicate the room_id field in indexeddb/sqlite, so each entry
contains both the room_id as a key (for queries) and as a value (to
return it).
Since there's no meaningful migration we can apply, the way to go is to
drop the pending events table and recreate it from the ground up. It is
assumed that no one has used the store on indexeddb; otherwise the
workaround would be to drop and recreate it.
A few were missing in the public API like SendEventError and RedactEventError and their dependencies.
This uses a wildcard because it should be rare to not want to expose an error type.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
A user of the library needs to add this crate as a dependency just to be able to match on `VectorDiff`.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
The issue here seems to be that
the `panic!` and `unreachable!()` macros used in the tests return `!`.
In the future, `!` will not fallback to `()`, which is what `dependency_on_unit_never_type_fallback` checks.
`add_event_handler` expects a function that returns an `EventHandlerResult`, but it is only implemented for `()`, not for `!`.
A solution could be to implement that trait for `!` but it is an unstable feature right now.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
It is only needed with the experimental-oidc and e2e-encryption features.
The former is less likely to be enabled so use it to enable the dependency.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch set does two things:
1. Extracted the logic to collect the devices that should receive a room key.
2. Introduce a new CollectStrategy enum which defines which rules are
used to collect recipient devices for a room key. Currently only the
existing rules have beenmoved under this enum.
This can happen if there's a load-balancer or any modification of the
response by a reverse proxy (e.g. rewrite 5XX errors into 429, to not
let a reverse proxy mark the upstream server as being down, as
Cloudflare seems to do).
As a result, such requests will be retried in multiple places, including
when sending something with the send queue. Also, the send queue will
mark these errors as recoverable instead of unrecoverable.
No test, because the change really is trivial and a regression test
didn't seem worth it, for once.
RoomInfo::handle_state_event(...) will check this condition so we don't have to later ask the HS whether the room is encrypted or not through room::is_encrypted().
Also, as we need some way to get this info in the room list in our clients, two ways of doing this were added the FFI crate, one through RoomInfo and another one through RoomListItem.
`RoomInfo::handle_state_event` will check this condition so we don't have to later ask the HS whether the room is encrypted or not through `room::is_encrypted()`.
This adds support to input your recovery key to the OIDC example which
will allow the OIDC example client to be verified and have access to all
the secrets (cross-signing keys and the backup recovery key).
Not particularly useful right now, but once the OIDC example is able to
log in other devices via a QR code it becomes necessary to have access
to all the secrets.
The workspace root enables some features which are required to compile
the benchmarks, but if you decide to just compile the benchmarks these
features won't be enabled since they aren't specified in the Cargo.toml
file of the benchmarks.
Let's define all the required features so compilation works in both
cases.
Fixes#3538
The current implementation for send_reply and edit only work with timeline items that have already been paginated.
However given the fact that by restoring drafts, we may restore a reply to an event for timeline where such event has not been paginated, sending such reply would fail (same for the edit event).
So I reworked a bit the code here to use. only the event id, and reuse the existing timeline if available, otherwise we can fetch the event and synthethise the content and still be able to successfully send the event.
This is the third part of the breakdown of the following PR: https://github.com/matrix-org/matrix-rust-sdk/pull/3439
Change this so that it fires off a task for each UTD event, rather than each
megolm session. This is a step towards considering the message index when
deciding whether to carry on with the download.
Not doing this leads to an invalid ordering of events, as shown in the
test: if we increase the timeline limit of a room using sliding sync,
we'll receive a duplicate event, and if we ignore it, it'll be in an
invalid position. The solution is to keep the most recent event (and
remove the old one from event_meta).
This PR makes 2 changes:
- Updates the storage of room heroes to be a single array containing the user's complete profile.
- Exposes these to the FFI so that client apps can use these for avatar colours/clusters.
Closes#2702 (again, now with avatars 🖼️)
---
* rooms: Store heroes as a complete user profile.
* ffi: Expose room heroes on Room and RoomInfo.
* chore: Remove TODO comment.
* Update crates/matrix-sdk-base/src/rooms/normal.rs
Signed-off-by: Benjamin Bouvier <public@benj.me>
---------
Signed-off-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <public@benj.me>
retry kind: make `characterize_retry_kind` a method of `HttpError`
---
retry kind: mark all network errors as transient
This should cause more backoff in general, if the client is disconnected
of the grid, or the remote server is, which is good behavior to have in
general.
---
http error: introduce another rustfmt-formated impl block for HttpError
The previous `impl` block was marked as skipping rustfmt, which led to
weird formatting behavior.
Changes are formatting only.
* sdk: Integ tests: factor out some helper methods
A few common methods for logic shared between the tests. This serves two
purposes:
* It reduces duplication, this making it easier to maintain the tests by
reducing the number of places that we have to change things.
* It makes the tests easier to read. By factoring out discrete steps, it's
much easier to get an overview of what a test is doing than when it's all in
one big function.
* sdk: Integ tests: timeout if nothing happens
If the test is going to fail, let's have it do so properly rather than
mysteriously getting stuck.
This patch creates an `ObservableMap` for `wasm32-unknown-unknown` which
simply wraps a `BTreeMap`, and without the `stream` method. Indeed, the
first implementation uses `eyeball_im::ObservableVector` which requires
`Send` and `Sync` on its values, which cannot compile to Wasm.
This patch implements `Client::rooms_stream` by forwarding the
result of `matrix_sdk_base::Client::rooms_stream`, and mapping the
`matrix_sdk_base::Room` to `matrix_sdk::Room`.
Running `cargo test -p matrix-sdk-base --features e2e-encryption`
makes the code to fail to compile because `crate::latest_event` isn't
defined. And indeed, it must be defined if `experimental-sliding-sync`
is enabled, but also if `e2e-encryption` is enabled.
This patch updates `Store::rooms` from a
`Arc<StdRwLock<BTreeMap<OwnedRoomId, Room>>>` to a
`Arc<StdRwLock<Rooms>>` where `Rooms` is a new type that mimics a
`BTreeMap` but that is observable. It uses an `ObservableVector`
for saving us the costs of writing a new `ObservableMap` type in
`eyeball-im`. It would have too much implications that are clearly
not necessary. The major one being that an `ObservableMap` must emit
`MapDiff`, but we expect `VectorDiff` everywhere in the SDK where rooms
are observable. It would break too many API and too many projects.
The benchmark is coming, but here are the results (for 10'000 rooms,
extreme case):
* `get_rooms` and `get_rooms_filtered` are twice faster (in practise, it
means 650µs is saved per call),
* `get_room` and `get_or_create_room` are 10% slower (in practise, it
means 8-10ns is lost per call).
Overall, I believe these results are acceptable, and even an improvement
for the first one.
This patch removes the useless `SlidingSyncList::get_room_id` method as
I don't see how it can be useful for anybody. It's mostly dead public
code, let's clean that up.
This might be controversial, but I do strongly prefer explicit over
implicit, in this case: it helps looking at the use cases, when using
the LSP's goto_references().
The HttpError variant was misplaced, as it was always used when the
`user_id()` is missing, which is causing `Error::AuthenticationRequired`
errors everywhere else in the code base. This patch streamlines usage of
`Error::AuthenticationRequired`, and gets rid of the `HttpError`
variant.
This patch renames `Client::get_rooms`, `::get_rooms_filtered` and
`::get_room` to respectively `::rooms`, `::rooms_filtered` and `::room`.
This `get_` prefix isn't really Rust idiomatic.
This patch renames `Store::get_rooms`, `::get_rooms_filtered` and
`::get_room` to respectively `::rooms`, `::rooms_filtered` and `::room`.
This `get_` prefix isn't really Rust idiomatic.
`Store::get_rooms` worked as follows: For each room ID, call
* Take the read lock for `rooms`,
* For each entry in `rooms`, take the room ID (the key), then call `Store::get_room`, which…
* Take the read lock for `rooms`,
* Based on the room ID (the key), read the room (the value) from `rooms`.
So calling `get_rooms` calls `get_room` for each room, which takes the lock every time.
This patch modifies that by fetching the values (the rooms) directly
from `rooms`, without calling `get_room`.
In my benchmark, it took 1.2ms to read 10'000 rooms. Now it takes 542µs.
Performance time has improved by -55.8%.
This patch replaces the `RwLock` by a `Mutex` in
`RoomListService::rooms` because:
1. it removes a race condition,
2. `RwLock` was used because the lock could have been taken for a long
time due to the previous `.await` point. It's no longer the case, so we
can blindly use a `Mutex` here.
This patch makes `RoomListService::room` synchronous. It no longer reads
a `SlidingSyncRoom` from `SlidingSync`, then it not needs to be async
anymore. This patch replaces the `RwLock` of `RoomListService::rooms`
from `tokio::sync` to `std::sync`.
The patch updates all calls to `RoomListService::room` to remove the
`.await` point.
This patch updates `room_list_service::Room::new` to take a `&Client`
and a `&RoomId` instead of a `SlidingSyncRoom`. The `SlidingSyncRoom` is
only used in `Room::default_room_timeline_builder` and is fetched there
from the `SlidingSync` instance lazily. It confines the
`SlidingSyncRoom` to one single method for `Room` now.
Using the resolved homeserver URL causes problems if we need to inspect
the well-known configuration of the homeserver, for example, if the
server name is matrix.org, but the homeserver URL is server.matrix.org,
the well-known might be only available for the former.
This is why we also need to receive the former, i.e. the server name in
the QR code data.
This patch removes `RoomInfo::latest_event`. To get the latest event,
one has to use `RoomListItem::latest_event` because it supports
local events and remote events. It was supposed to be the case of
`Room::room_info` too, but only when the method was called: Once the
`RoomInfo` was created, the latest event inside `RoomInfo`
was static, not dynamic. Also, this code wasn't tested
contrary to `RoomListItem::latest_event` which uses
`matrix_sdk_ui::room_list_service::Room::latest_event` which is itself
tested.
This patch changes the behaviour of the `Timeline` when a duplicated
event is received. Prior to this patch, the old event was removed, and
the one was added. With this patch, the old event is kept, and the new
one is skipped.
This is important for https://github.com/matrix-org/matrix-rust-sdk/pull/3512
where events are mapped to the timeline item indices. When an event is
removed, it becomes difficult to keep the mapping correct.
This patch also adds duplication detection for pagination (with
`TimelineItemPosition::Start`).
This patch removes the `matrix_sdk_ui::timeline::SlidingSyncRoomExt`
trait as it's no longer necessary since `room_list::Room::latest_event`
no longer uses `SlidingSyncRoom` to fetch the latest event.
This patch updates `matrix_sdk_ui::room_list::Room::latest_event`
to fetch the latest event from `matrix_sdk::Room` instead of
`matrix_sdk::sliding_sync::SlidingSyncRoom`. It removes one dependency
to `SlidingSyncRoom` and it simplifies the code.
This avoids a race condition where the caller hasn't set up the initial
items or the listener, and the listener is called *before* the initial
items have been used.
second part of the #3439 breakdown
The context for this API, is for the composer preview an reply without the need of it being actively in the timeline.
This PR is the first piece of the breakdown of the following PR: https://github.com/matrix-org/matrix-rust-sdk/pull/3439 where only the core functionalities of the feature are implemented, without addressing the issue of editing and replying to events that have not yet been paginated.
The test looks at updates of `RoomInfo`s, but these depends on external
factors like the server sending them in one part or multiple ones.
Instead of trying to figure out which partial updates the server sent,
wait for the stream of `RoomInfo`s to stabilize by trying to get the
latest item from the stream, then wait up to N seconds for another item
to show up, and continue as long as items come in.
This should allow us to get rid of some code that was required to
prevent flakey updates.
The previous code would use `ACTIVE` which means "either joined or
invited" members, but the code thereafter considered this to be the
number of joined members.
Also replaced a call to `self.members()` (which does a lot of work under
the hood) with a call to `self.joined_user_ids()`, since we only
retrieve the `.len()` of the resulting vector.
- Change incorrect "key sharing" references to "key forwarding".
- Explain that this is a feature that needs to be enabled.
- Regenerate the key forwarding algorithm diagram.
- Provide a spec link and mention alternative names.
Use a Bloom filter to keep track of which events we have already reported to the parent UTD hook.
We load the data from database on startup, and flush it out immediately on every update.
This patch renames `TimelineEventHandler::add` to `add_item`.
This patch also removes the `should_add` argument. The `add_item`
is called conditionally everytime. I believe this is much cleaner.
Otherwise the method should have been called `maybe_add_item` with a
return type or something that indicates whether the item is added. This
patch makes it clear and remove one possible state in `add_item`.
This patch updates `TimelineEventHandler` to re-use the public API and
avoid using `TimelineInnerMeta::next_internal_id`. The consequence is
that `next_internal_id` can now be private instead of public. It's
better to have isolated API in this code.
This patch removes a useless clone of `TimelineInner::items`. This clone
was technically cheap because it's a `Vector` behind the scene, which is
cheap to clone, but still, it's not necessary at all to clone it.
Because of the task cancellation that can happen at any place in the
code base, it's really hard to predict in which situation a
day-divider-adjuster should have run or not, so just demote the assert
to an error. The intent is that, if we see errors with day dividers in
the future, they'll be reported along rageshakes and we'll notice this
in the logs.
Using async when not required will increase compile times, and propagate async-ness to the callers, transitively.
See also the [lint description](https://rust-lang.github.io/rust-clippy/master/#/unused_async).
Since we only had a few false positives, I've enabled it by default.
Also reunify two methods that did the same thing, with slightly
different semantics, and test the one that wasn't tested at all.
Note that `is_editable()` was already exposed to the FFI and used in EX
apps.
It was used after a previous local echo couldn't be sent (i.e. the
sending queue failed to send it). However, it was slightly incorrect to
mark those as cancelled, since those local echoes would still have
corresponding items in the sending queue, and would be retried as soon
as the sending queue was reenabled and could send events.
Instead, this is removed: it means that previously cancelled events
would be in the NotSentYet state, which is correct. (At the limit, we
could even get rid of the SendingFailed variant, since the sending queue
will automatically retry as soon as possible.)
The test relied on the fact that sending an event from a given timeline
is not observable from another timeline. Indeed, it sent a message using
a first timeline object, then constructed a second timeline object and
expected only the remote event to be in there.
Now, the sending queue is shared across all instances of a Room, thus
all instances of a Timeline, and the second timeline can see the local
echo for the message sent by the first timeline.
The "fix" is thus in the test structure itself: when waiting for the
remote echo to be there, check that the timeline item doesn't pertain to
a local echo, i.e. is a remote echo.
Technically, the test already passed before the change in the builder,
because `TimelineEventHandler::handle_event` already filters out local
events on non-live timelines, but it's a waste of resources to even
spawn the local echo listener task in that case, so let's not do it.
By keeping a reference to the FFI Client, we make sure that the SDK
Client is dropped while in a tokio runtime context. This makes it
possible for an `Encryption` (FFI) object to outlive the `Client` (FFI)
object without crashing (because deadpool requires to be in a tokio
runtime context when dropping).
Following https://github.com/matrix-org/matrix-rust-sdk/pull/3473, we
made it so that if there's a base-path but no username, then we'll
create a sqlite database.
Unfortunately, this introduced a bad side-effect: for the temporary
client used during homeserver resolution, this would create a temporary
sqlite database.
The "fix" is to not set the base path for the temporary client, and only
for the other caller of `new_client_builder()`, manually. The long term
fix is to get rid of the `AuthenticationService` so we can test it
properly.
This patch updates Ruma to 75e8829 so that `RoomSummary::heroes` is now
a `Vec<OwnedUserId>` instead of a `Vec<String>`. This patch updates our
code accordingly by removing the parsing of heroes as `OwnedUserId`.
Without this, the configs from `.cargo/config.toml` were not read in CI
tasks, causing false positives when running Clippy on CI (i.e. there
were issues observed when compiling locally that were not found when
compiling remotely).
Not entirely sure why it's needed, because I'm seeing the issues when
I'm using `cargo xtask ci clippy` locally, with nothing changed.
It turns out that the failure came when using the known room path in the
room preview: Alice knows about the room, but for some reason the client
didn't retrieve all the state events from the sliding sync proxy yet.
Before, the sync would be considered stable after 2 seconds. This is too
little, considering that events come from the proxy that listens to
events from synapse. Raising this threshold to 15 seconds should help
getting all the room information from the proxy, and thus get all the
information we expected in the client.
While this mutex is taken, in `oldest_token()` we can get a hold of the
RoomEvent mutex, making it so that the `waited_...` token covers the
critical region of room events.
Unfortunately, in `clear()`, we're taking these two locks in the
opposite order, implying that the room events critical region now
overlaps that of the `waited_...` lock.
The way to break the cycle is to keep the `waited_` token for as short
as possible.
So as to not use sync mutexes across await points, we have to use an
async mutex, BUT it can't be immediately called in a wiremock responder,
so we need to shoe-horn a bit: create a new tokio runtime, which can
only be called from another running thread, etc. It's a bit ugly, but I
couldn't find another mechanism to block the responder from returning a
Response immediately; happy to change that anytime if there's a simpler
way.
A client would require this to know that the sending queue has been
globally disabled, across all the rooms, otherwise they couldn't know
that it needs to be re-enabled later on.
This implements the following features:
- enabling/disabling the sending queue at a client-wide granularity,
- one sending queue per room, that lives until the client is shutting
down,
- retrying sending an event three times, if it failed in the past, with
some exponential backoff to attempt to re-send,
- allowing to cancel sending an event we haven't sent yet.
fixup sdk
This can be equivalently retrieved using:
`get_timeline_event_by_event_id(event_id).content().as_message()?.content()`
As per the commit date, this is used in one place in both EXA and EXI,
so it seems fine to change the caller's call sites.
And this avoids one explicit call site of
`Timeline::item_by_event_id()`.
If an event cannot be decrypted after a grace period, it is reported to the application (and thence Posthog) as a UTD event. Currently, if it is then successfully decrypted, the event is then re-reported as a "late decryption".
This does not match the expected behaviour in Posthog - an event is *either* a UTD, or a late decryption; it makes no sense for it to be both. This PR fixes the problem.
I've attempted to make the commits sensible, but I'm not entirely sure I've succeeded. The tests do pass after each commit, though. The interesting change itself is somewhere in the middle; there is non-functional groundwork before and cleanup afterwards.
---
* ui: Factor out UTD report code to a closure
For now, this doesn't help much, but in future there will be more logic here,
and it helps reduce the repetition between the delay and no-delay cases.
* ui: Convert UtdHookManager::pending_delayed to a HashMap
* ui: Store decryption time in `UtdHookManager::pending_delayed`
This is a step on getting rid of `known_utds`
* ui: Fix re-reporting of late decryptions
This fixes the problem where a message that was previously reported as a UTD,
and was then subsequently successfully decrypted, is then re-reported as a late
decryption. This artificially inflated the UTD metrics.
We do this by checking the `pending_delayed` list in `on_late_decrypt`, instead
of the `known_utds` list. There is some associated reordering of code to get
the locking right.
* ui: Remove unused "utd report time" from `UtdHookManager::known_utds`
* ui: Replace `UtdHookManager::known_utds` with `reported_utds`
Keep a list of the UTDs we've actually reported, rather than the union of those
we've reported together with those we might report in a while.
I find this much easier to reason about.
* Address minor review comments
* Reinstate assertion in UTD hook tests
* Reinstate `known_utds`
To differentiate the SAS state between the party
that sent the verification start and the party that received it.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch adds a new argument to the `until` argument closure of
`RoomPagination::run_backwards`: `timeline_has_been_reset`, which
designates when the timeline has been reset.
A test has been updated accordingly.
* chore(sdk): Rename `RoomEventCacheUpdate::UpdateReadMarker`.
This patch renames `RoomEventCacheUpdate::UpdateReadMarker` to
`ReadMarker`. The `Update` prefix is already part of the enum name.
* feat(sdk): Rename `RoomEventCacheUpdate::ReadMarker::event_id` to `move_to`.
This patch renames `RoomEventCacheUpdate::ReadMarker::event_id` to
`ReadMarker::move_to` as I feel like it conveys a better semantics.
* feat(sdk): Extract `RoomEventCacheUpdate::Append::ambiguity_changes`.
This patch extracts `RoomEventCacheUpdate::Append::ambiguity_changes`
into a new variant `RoomEventCacheUpdate::Members { ambiguity_changes }
`.
This patch also creates a new private
`RoomEventCacheInner::send_grouped_updates_for_events` method to ensure
the updates are sent in a particular order.
* feat(sdk): Rename `RoomEventCacheUpdate::Append`.
This patch renames `RoomEventCacheUpdate::Append` to `SyncEvents`. The
field `events` is renamed `timeline`.
This patch also renames some variables to clarify the code and to match
the renamings in `RoomEventCacheUpdate`.
* feat(sdk): Rename `RoomEventCacheUpdate::ReadMarker` and `Members`.
This patch renames `ReadMarker { move_to: … }` to `MoveReaderMarkerTo
{ event_id: … }`.
This patch also renames `Members` to `UpdateMembers`.
Finally, this patch renames some other variables to avoid clashes with
terminology in `matrix_sdk_ui`.
* feat(sdk): Split `RoomEventCacheUpdate::SyncEvents`.
This patch splits `RoomEventCacheUpdate::SyncEvents` into 2 new variants:
`AddTimelineEvents` and `AddEphemeralEvents`.
This patch takes this opportunity to update `matrix_sdk_ui::timeline`
a little bit too. `handle_sync_events` is renamed
`handle_ephemeral_events`, and the `SyncTimelineEvent` argument is
removed: it's possible to use `add_events_at` directly to handle the
`SyncTimelineEvent`s.
* fix(sdk): Do not send `RoomEventCacheUpdate` if values are empty.
This patch prevents sending useless `RoomEventCacheUdpate` if their
respective values are empty.
* chore(ui): Update a log message.
* Apply suggestions from code review
Signed-off-by: Benjamin Bouvier <public@benj.me>
---------
Signed-off-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <public@benj.me>
This patch adds a new argument to `RoomPagination::run_backwards`:
`until`. It becomes:
pub async fn run_backwards<F, B, Fut>(&self,
batch_size: u16,
mut until: F
) -> Result<B>
where
F: FnMut(BackPaginationOutcome) -> Fut,
Fut: Future<Output = ControlFlow<B, ()>>,
The idea behind `until` is to run pagination _until_ `until` returns
`ControlFlow::Break`, otherwise it continues paginating.
This is useful is many scenearii (cf. the documentation). This is
also and primarily the first step to stop adding events directly from
the pagination, and starts adding events only and strictly only from
`event_cache::RoomEventCacheUpdate` (again, see the `TODO` in the
documentation). This is not done in this patch for the sake of ease
of review.
This is the right goal, and Ruma will be updated to reflect that the
`heroes` field should contain `OwnedUserId` too in [1].
This simplifies the code a bit, and avoids a round-trip encoding
user-ids into a string then decoding them from a string later, in the
case of sliding sync and room name computation.
[1] https://github.com/ruma/ruma/pull/1822
Since rendering is sync, and I want the computed display name (which
retrieval is async), I have to fetch the display name early, just after
getting the room. Oh well :)
This patch does 3 things:
1. It updates Ruma to the latest revision at the time of writing,
2. It updates `matrix-sdk-ffi` to provide the
`RoomSubscription::include_heroes` field,
3. It updates `matrix-sdk-base` so that SlidingSync consumes `heroes`
from the response and update the `RoomSummary` accordingly.
A test has been added to ensure the `RoomSummary` is updated as expected.
It doesn't make sense to expose it for a focused timeline.
Note there's no way for timeline to change focus, at the moment; this
would require special handling here, because one could subscribe to the
back pagination status on a live timeline first, then switch the
timeline to the focus mode. The stream wrapper could then observe other
states that aren't expected by the converter method.
This is actually required for one use case: if the event cache triggers
back-pagination in the background *before* a timeline is opened, it's
possible the timeline observes pagination is running at the beginning,
and doesn't know if more back-pagination is required or not.
This wraps the `Paginator::status` stream by reading the
hit_timeline_start() from the room pagination and adding it to the
`Idle` state.
This whole method is a bit broken. It assumes that, when we did an import from
backup, the backup that they came from is the same as the "current" version.
We *could* add another argument, but to be honest I find the whole method a bit
pointless. It's not particularly tied to the `BackupMachine` abstraction. Let's
instead expose `Store::import_room_keys` and
`ExportedRooomKey::from_backed_up_room_key` publicly, and tell callers to use
that directly.
When we are importing a batch of room keys, use the newly-added
`CryptoStore::save_inbound_group_sessions` method instead of
`CryptoStore::save_changes`.
To do this, we need to pass the backup version into `Store::import_room_keys`
instead of just `from_backup` flag.
When we add a batch of inbound group sessions to the store, if they came from a
backup, we need to record which backup version they came from.
`CryptoStore::save_changes` doesn't give us an easy way to set the backup
version (we could add it to the `Changes` struct, but ugh).
So, we add a new method `save_inbound_group_sessions` which takes the backup
version as a separate param. This works fine, because whenever we import
sessions there are no other changes to store.
The new method isn't used outside of tests yet -- that comes in a follow-up
commit.
This moves code around, and tweaks the behavior:
- `WeakRoom::get()` returns an `Option`, that will be None if the client
is missing or the room is missing.
- `PaginableRoom` for `WeakRoom` will now return a default response if
the room could not be reconstructed from the weak room. It's fine to do
so because we're in a shutdown context.
It's possible the timeline starts with a read marker, in which case the
first item won't be a day divider; in that case, check for the next
item, if present.
test_start_with_read_marker
This patch adds another check to ensure `AsVector` generates
`VectorDiff`s that, once combined, produce an expected `Vector`. It
avoids errors when unit testing `VectorDiff` alone.
This patch ensures that an underflow is not possible when the length
of `chunks` is 0. In practise, it's not possible because there is
_always_ one chunk inside `LinkedChunk`, but it's better to have good
code habits.
This patch removes the `unsafe` part of `as_vector`. The idea is to
pass `Iter` (the forward iterator of `Chunk`) to `AsVector` so that it
internally computes `initial_chunk_lengths`. The shape of this data must
no longer be guaranteed by the caller.
This patch goes a bit further: `UpdateToVectorDiff` has
a new constructor which consumes this `Iter` and builds
`initial_chunk_lengths` itself. Even better!
Finally, `Updates::as_vector` is removed. It's clearly no longer
necessary and it was creating borrowing issues anyway with the new code
structure.
Allow applications to skip the PBKDF2 operation if they already have a cryptographically secure key,
instead using a simple HKDF to derive a key.
In order to maintain compatibility for existing element-web sessions, if we discover that we have an
existing store that was encrypted with a key derived from PBKDF2, then we reconstruct what
element-web used to do: specifically, we base64-encode the key to obtain the "passphrase" that
was previously passed in. If that matches, we know we've got the right key, and can update the
meta store accordingly.
Part of a resolution to element-hq/element-web#26821.
Signed-off-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Co-authored-by: Damir Jelić <poljar@termina.org.uk>
This patch renames `ReattachItems` to `StartReattachItems` and
`ReattachItemsDone` to `EndReattachItems`. This naming conveys better
the idea of a _state transition_.
Add a CI step running complement crypto, automatically matching the complement-crypto branch name based on the current branch name, if needs be.
Signed-off-by: Kegan Dougal <7190048+kegsay@users.noreply.github.com>
We could end up, like in the regression test, with a sequence of
operations like that:
- remove day divider @ i+1 (because it's redundant with one @ i)
- remove day divider @ i (because it's useless, since the event before
the day divider and after the day divider use the same date).
In that case, it would break the non-decreasing invariant: we'd apply an
operation on the array @ i+1, then @ i, which troubles the offset
computation.
Instead, when doing an operation based on the "prev_item" (now with a
small helper struct, to facilitate understanding of each field), we also
record the insertion order for the operation itself: it's always "at the
end of the operation order, at the time we're looking at it", so
equivalent to a "push_back" if there's no operation in between; but that
ensures that we'll do the operation in a non-decreasing order. For
instance in the above test case, the Remove(i) is now inserted before
the Remove(i+1), instead of after.
* sdk: Return a Thumbnail from generate_image_thumbnail
We have already all the data for it.
Also fixes an error where the thumbnail format was assumed to always be
JPEG.
* sdk: Allow to select the format of the generated thumbnail
Sending an attachment could often fail if the image crate
cannot encode the thumbnail to the same format as the original.
This allows to select a known supported format to always
be able to generate a thumbnail.
* sdk: Do not return error of thumbnail generation for SendAttachment
Since the thumbnail is optional, failing to generate it should not
stop us from sending the attachment.
* Apply code review fixes
* sdk: Split attachment tests in separate file
* sdk: Add integration tests for generating thumbnails
* Revert wiremock debug log level
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
It's always provided in the `avatar` response in the field, which
sliding sync's using by default, so there's no need to request the
`m.room.avatar` event too.
If we want to be able to note the absence of a room name, we shouldn't
take the name returned by the server (which may be computed, and thus
always be non-null). This way, we can expose both the name contained in
the m.room.name event as well as the computed name everywhere.
A consequence of this is that the room list service must now ask for the
m.room.name event as part of the required state for every room.
With a regression test that didn't pass on main, and would incorrectly
remove the read marker, because it tried to replace the day divider at
(4), that was just scheduled for deletion in the previous loop
iteration.
This patch removes the possibility to have `LinkedChunk` as an
`ObservableVector` via a `Stream`, which was initially the goal of
`AsVectorSubscriber`. Having a `Stream` was more complex than required
for the integration inside `EventCache`. This patch keeps the same code
but renames `AsVectorSubscriber` into `AsVector`. A new internal type,
`UpdateToVectorDiff`, applies the algorithm itself and maintains its
own state, making it possible to re-introduce a `Stream` later if we
want to.
* event cache: reuse the paginator internally
Fixes#3355.
* event cache: move the `pagination_token_notifier` into the `RoomPaginationData` as well
* event cache: introduce a `RoomPagination` API object and move code around
Only code motion. No changes in functionality.
* event cache: remove "paginate" (et al.) in `RoomPagination` method names
No changes in functionality, just renamings.
* event_cache/timeline: have the event cache handle restarting a back-pagination that failed under our feet
When a timeline reset happens while we're back-paginating, the event
cache method to run back pagination would return an success result
indicating that the pagination token disappeared. After thinking about
it, it's not the best API in the world; ideally, the backpagination
mechanism would restart automatically.
Now, this was handled in the timeline before, and the reason it was
handled there was because it was possible to back-paginate and ask for a
certain number of events. I've removed that feature, so that
back-pagination on a live timeline matches the capabilities of a
focused-timeline back-pagination: one can only ask for a given number of
*events*, not timeline items.
As a matter of fact, this simplifies the code a lot by removing many
data structures, that were also exposed (and unused, since recent
changes) in the FFI layer.
* Address review comments
This patch adds 2 new updates: `ReattachItems` and `ReattachItemsDone`.
It's useful to optimise insertion, esp. in `AsVectorSubscriber` but
almost maybe in database.
This patch renames `TruncateItems` to `DetachLastItems`. It's
technically the same things, but the fields are different: `chunk:
ChunkIdentifier` and `length: usize` become a unique `at: Position`. The
spirit is the same but the semantics is a bit different.
This patch, with this renaming, also prepares the next commits.
First off, this patch renames `Update::InsertItems` to
`Update::PushItems` because all items insertions happen at the end of
a chunk. Let's reflect that. `InsertItems` would have meant that it's
possible to insert at any position, but it's not what happens by design.
Second, this patch renames `Update::PushItems::at` to
`Update::PushItems::position_hint`. Indeed, the `at` field would have
meant that the position is specific whilst it's not. It's just a hint to
make the life of update readers simpler. It's also a good way to improve
performance in some cases: since items are pushed, to know the new
position of the items, one has to read the position of the previous last
item and compute the new position from there. Having `position_hint`
help to prevent this dance and save the cost of a reading. That's also
why the field has been renamed to `position_hint`, it's a hint.
This patch updates the constructor of `AsVectorSubscriber` to receive
the initial chunk lengths so that this type doesn't have to guess what
to do when receiving an unknown chunk identifier. `AsVectorSubscriber`
is de facto synchronized with its source `LinkedChunk` when created.
I found the previous API confusing. Firstly, the parameter name was "filename" when we want a file path.
Secondly, we take a `String` when we want a `Path`.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Cargo nightly now checks whether a cfg option exists.
We need to declare the custom
options we use
and fix those that are wrong.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This makes the public key fields private to ensure that we don't
accidentally swap them out. It also moves the construction of the
subkeys into the master key type.
Since the master/self-signing/user-signing public key types are used for
public user identities as well as for the private key type we have, and
we'd like to sign the public key types it makes sense that the types
itself aren't using an Arc.
Let's instead put the Arc inside the user identity structs.
This will allow us later on to more easily sign the public key types.
This patch removes the notion of `take` vs. `peek` from
`LinkedChunkUpdatesInner` and widens the problem to a more general
approach: `LinkedChunkUpdatesInner` must support multiple readers, not
only two (`take` was the first reader, `peek` was the second reader,
kind of).
Why do we need multiple readers? `LinkedChunkUpdates::take` is
clearly the first reader, it's part of the public API. But the private
API needs to read the updates too, without consuming them, like
`LinkedChunkUpdatesSubscriber`. `peek` was nice for that, but it's
possible to have multiple `LinkedChunkUpdatesSubscriber` at the same
time! Hence the need to widen the approach from 2 readers to many
readers.
This patch introduces a `ReaderToken` to identify readers. The last
indexes are now all stored in a `HashMap<ReaderToken, usize>`. The rest
of the modifications are the consequence of that.
The test `test_updates_take_and_peek` has been entirely rewritten to be
`test_updates_take_and_garbage_collector` where it tests 2 readers and
see how the garbage collector reacts to that.
This patch implements `LinkedChunkUpdates::subscribe` and
`LinkedChunkUpdateSubscriber`, which itself implements `Stream`.
This patch splits `LinkedChunkUpdates` into `LinkedChunkUpdatesInner`,
so that the latter can be shared with `LinkedChunkUpdatesSubscriber`.
This patch adds the `LinkedChunkUpdates::peek` method. This is
a new channel to read the updates without consuming them, like
`LinkedChunkUpdates::take` does.
The complexity is: when do we clear/drop the updates then? We don't
want to keep them in memory forever. Initially `take` was clearing
the updates, but now that we can read them with `peek` too, who's
responsible to clear them? Enter `garbage_collect`. First off,
we already need to maintain 2 index, resp. `last_taken_index` and
`last_peeked_index` so that `take` and `peek` don't return already
returned updates. They respectively know the index of the last update
that has been read. We can use this information to know which updates
must be garbage collected: that's all updates below the two index.
Tadaa. Simple. The only _drawback_ (if it can be considered as such)
is that the garbage collection happens on the next call to `take` or
`peek` (because of the borrow checker). That's not really a big deal in
practise. We could make it happens immediately when calling `take` or
`peek` but it needs more pointer arithmetic and a less straighforward
code.
This patch updates the `LinkedChunk::update_history` field from
a simple `Vec<LinkedChunkUpdate<Item, Gap>>` type to the new
`LinkedChunkUpdates<Item, Gap>` type (note the plural).
This is going to be helpul for the next patches.
This patch removes support for OpenTelemetry because it's not used
anymore by anybody. It also adds multiple duplicated dependencies (like
`reqwest`). Anyway. Farewell.
Preferring to use "reject" wording rather than "failed to
create/update". The latter can be easily misinterpreted as a failure of
the local client to create an entirely new device from scratch, rather
than refusal to instantiate a new local device representation of an
(invalid) device definition received from the server.
This patch makes the `LinkedChunk::update_history` field optional,
so that it doesn't require the user to drain it to avoid eating the
universe.
The `new` constructor disabled the update history, the
`new_with_update_history` enables it.
This patch is a turn around about the `LinkedChunkListener`. Many
patches have been removed because `LinkedChunkListener` needed to
support I/O, so errors and async code. The whole code was affected by
that, resulting in a complex API. The idea of this patch is to decoupled
this. Here is how.
First off, `LinkedChunkListener` is removed. So it's one less generic
parameter on `LinkedChunk`. It's also one less trait, so less
implementations.
Second, now `LinkedChunk` accumulates/collects all “updates” under the
form of a new enum `LinkedChunkUpdate`. These updates can be read with
`LinkedChunk::updates(&mut self) -> &Vec<LinkedChunkUpdate>`. The reader
can simply read them, or even drain them. The reader is responsible
to handle these updates and to dispatch them in a storage or whatever.
`LinkedChunk` is no longer responsible to do that, removing the need to
support errors and to be async.
Third, the simplification has led to an optimisation by introducing a
new type `LinkedChunkLinks`. The documentation explains what it does
and why it was needed. The benefit of this type is: it doesn't increase
the size of `LinkedChunk`, but it simplifies the code: no more `Arc`,
no more `Mutex` (it was required because with I/O and async), no more
borrow checker trick, and the code stays as safe as before.
This patch removes the `displaydoc` dependency. Why?
1. It creates a warning in rustc nightly:
```
warning: non-local `impl` definition, they should be avoided as they go against expectation
--> crates/matrix-sdk-store-encryption/src/lib.rs:49:17
|
49 | #[derive(Debug, Display, thiserror::Error)]
| ^^^^^^^
|
= help: move this `impl` block outside the of the current constant `_DERIVE_Display_FOR_Error`
= note: an `impl` definition is non-local if it is nested inside an item and may impact type checking outside of that item. This can be the case if neither the trait or the self type are at the same nesting level as the `impl`
= note: one exception to the rule are anon-const (`const _: () = { ... }`) at top-level module and anon-const at the same nesting as the trait or type
= note: this lint may become deny-by-default in the edition 2024 and higher, see the tracking issue <https://github.com/rust-lang/rust/issues/120363>
= note: the derive macro `Display` may come from an old version of the `displaydoc` crate, try updating your dependency with `cargo update -p displaydoc`
= note: `#[warn(non_local_definitions)]` on by default
= note: this warning originates in the derive macro `Display` (in Nightly builds, run with -Z macro-backtrace for more info)
```
2. `thiserror` is already used, which seems to provide a similar issue.
3. That's less dependency, and less proc-macro, which will improve the
compilation time in general.
Also:
- rename `display_name` to `computed_display_name` in several places,
and reflect that change into a few callers
- simplify slightly the `computed_display_name()` method
This is in line with what the other method using sliding sync does. This
wasn't tested before, because this required `filter_by_push_rules()` to
be enabled in the notification client; now that it's the default, the
test revealed the bug, and so it could be fixed.
The builder had only one meaningful method, `filter_by_push_rules`,
which was always called by the applications — and in fact should always
be true. It was designed as an extra method because it was experimental
at the time, but it's stabilized sufficiently that we can enable this
behavior by default now, considering that a notification that is not
wanted by the user shouldn't be kept, to respect their intent. (This is
in the UI crate, which is opinionated, so it's fine to assume such
intents by design.)
They need to be updated together
because the latters depend on the former.
matrix-authentication-service is still using http 0.2
so we need to add a conversion layer between both major versions
for OIDC requests.
We need to update vodozemac too because of a dependency resolution issue.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This makes all the requests (/context and /messages) cancellation-safe
by making two changes:
- first, use sync locks instead of async locks for the prev/next batch
tokens. This ensures that we can't get cancelled after receiving a
response and managing internal state, i.e. we keep run-to-completion
semantics until the end of the function, once we got a response.
- second, introduce a RAII guard that will reset the state to a given
value when dropped. Then use this to reset the state to Initial (resp.
Idle) in /context (resp. /messages) when the async call is aborted. We
use `mem::forget` once the response has been returned, so as to not call
the `Drop` implementation of the guard later on.
A regression test has also been introduced.
This patch changes the signature of `LinkedChunk<Item, Gap, const
CHUNK_CAPACITY = usize>` to `LinkedChunk<const CHUNK_CAPACITY = usize,
Item, Gap>`. It allows to add more generic parameters if needed, without
conflicting with generic constants.
The clippy website explains that `Box::new(Default::default())` can be
shortened to `Box::default()` and that it's more readable. That's true…
when you don't have a generic parameter in the mix (which may be
required if rustc can't infer the type of the boxee), in which case it
just looks bad, e.g. `Box::<MyType>::default()` instead of
`Box::new(MyType::default())`.
I do strongly prefer the latter, and propose to get rid of the lint, as
a result.
Version 0.7.5 triggers a warning in
our version of rust nightly in CI,
but we can't update it to a recent
version because it triggers a warning
caused by displaydoc
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This introduces the `TimelineFocus`, a new enum to declare if the
timeline is "live" aka looking at events from sync and displaying them
as they come in, or focused on an event (e.g. after clicking a
permalink).
When in the second mode, the timeline can paginate forwards and
backwards, without interacting with the event cache (as this would
require some complicated reconciliation of known events with events
received from pagination, with no guarantee that those events are event
connected in whatever way).
An event-focused timeline will also show edits/reactions/redactions in
real-time (as the events are received from the sync), but will not show
new timeline items, be they for local echoes or events received from the
sync.
`Room::invite_details()` can return an error if the room member event
(the invite) is missing from the store. Usually that would be a sign
that the state is semi-broken, since there should always be such an
event for an invited room. But if it's missing, it shouldn't break the
creation of a `RoomInfo`, and just be missing from the struct.
In https://github.com/matrix-org/matrix-rust-sdk/pull/2691, I suppose
the way `add_mentions` is computed is… wrong. `AddMentions` is used to
automatically infer the `m.mentions` of the reply event based on the
replied event. The way it was computed was based on the reply event
`mentions`, which seems wrong: if the reply contains mentions, then the
sender should be part of it? Nah. That's a bug. We want the reply event
to automatically mention the sender of the replied event if and only
if it's not the same as the current user, i.e. the sender of the reply
event.
This patch fixes the `add_mentions` calculation. This patch also updates
a test and adds another test to ensure that `m.mentions` is correctly
defined when replying to an event.
The `state` lock was taken at the top level of this function, and
indirectly implicitly in the `set_fully_read_event` function. This fixes
it, and adds a regression test.
Fixes#3311.
When the timeline is lagging behind the event cache, it should not only
clear what it contains (because it may be lagging some information
coming from the event cache), it should also retrieve the events the
cache knows about, and adds them as if they were "initial" events.
This makes sure that the event cache's events and the timeline's events
are always the same, and that, in the case of a lag, there won't be any
missing chunks (caused by the event cache back-pagination being further
away in the past, than what's displayed in the timeline).
The lag behind a bit too hard to reproduce, I've not included a test
here; but I think this should be the right move.
Redaction was requiring the `RoomRedactionEventContent`, which was
eventually unused in all the code paths; this removes it.
Also adds lots of documentation for the different types of reaction that
can happen in the timeline.
This introduces a new helper object to run arbitrary pagination requests, backwards- or forward-. At the moment they're disconnected from the event cache, although I've put the files there for future convenience, since at some point we'll want to merge the retrieved events with the cache (? maybe).
This little state machine makes it possible to retrieve the initial data, given an initial event id, using the /context endpoint; then allow stateful pagination using a paginator kind of API. Paginating in the timeline indicates whether we've reached the start/end of the timeline.
The test for the state subscription is quite extensive and makes sure the basic functionality works as intended.
Some testing helpers have been (re)introduced in the SDK crate, simplifying the code, and introducing a better `EventFactory` / `EventBuilder` pattern than the existing one in the `matrix-sdk-test` crate. In particular, this can make use of some types in `matrix-sdk`, notably `SyncTimelineEvent` and `TimelineEvent`, and I've found the API to be simpler to use as well.
Part of #3234.
This protects against the sliding sync proxy (or synapse) sending lots
of duplicate fully read marker events for the same room in case of
reset. This would lead to forwarding lots and lots of
`RoomEventCacheUpdate`s to the timeline, cluttering the channel, and
resulting in a lag. This lag would then clear the timeline, seemingly
cause missing chunks from history.
https://github.com/matrix-org/matrix-rust-sdk/pull/3312 is likely the
long term fix (after a clear(), the timeline should retrieve all the
memoized events from the event cache), but this should prevent the root
cause of the spamming in the first place, getting us back to a safe
state.
We do see lots of "broken channel" log lines for these two log
statements, and since I'm unclear why they happened, I'd like to add a
bit more logging to those.
Also makes the log level consistent, both are set to "warn" instead of
one warn and one error. Usually it's not a big deal because the only
error that may happen is that the channel is broken, indicating the task
died before, so there's no need to stop it manually.
Those two issues really aren't errors we should be too scared of:
- on the one hand, they don't prevent correct usage of the timeline, but
slightly decrease the UX's quality
- on the other hand, they may indicate that read receipts haven't been
received yet, OR some events are unknown (can be happening if the
previous read receipt refers to an event the timeline doesn't know
about).
So I lowered their severity to debug instead of error, since only
outstanding issues should errors or warnings in my opinion.
`force_update_sender_profiles` goes through all the timeline items, even
though the `ambiguity_changes` map should be empty, leading to wasted
work. It might not be a big deal, but it's nice to avoid awaiting locks
when we don't have to.
First, add a log line, since this is a pretty big deal if we start
lagging behind, and we should understand this clearly in rageshakes.
Second: don't remove existing `RoomEventCache`s, but instead clear them:
we shouldn't re-create new `RoomEventCache`s for the same room id,
otherwise a previous subscriber to updates to `RoomEventCache` would not
get newer updates coming later on.
Third: let consumers know that we cleared the events from the
`RoomEventCaches`.
* ffi: Use async functions in AuthenticationService
* Fix swift tests
* Set async_runtime for uniffi::export attribute
* Rename RwLocks
* Get rid of unnecessary map_err
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Before this patch, only local echoes that were messages not sent or
messages succesfully sent (but not ack'd yet by sync) would be pinned to
the bottom.
This fixes it by also pinning events that failed to send.
Fixes#3287.
The algorithm works on the basis that we remove a previous day divider
if it was spurious. But we'd never do this, for a final item that would
be a day divider! So chase these explicitly, also ignore read markers
that would be in the way.
The `MemberInfo` struct only existed to regroup parameters, so let's
pass individual parameters instead. One less data structure is good for
the cognitive load.
This patch fixes a bug when inserting items at the end of the
current items of a chunk. The condition was: `if item_index >=
current_items_length`, which is too restrictive. The bug was “hidden”
so far because it's not possible to get the position of “after an item”
with the current API. However, the bug arises if the items are empty and
new items are inserted the position index 0 (index = 0, length = 0).
The fix consists of relaxing the condition, and introducing an
optimisation. If we insert at the end, we do a simple push (like an
append). If we insert at another position, we split, then push the new
items, and finally push the detached items (as it was the case before).
The tests have been updated accordingly.
Prior to this patch, in `RoomEventCacheInner::backpaginate`, when the
`token` validity was checked, and it was invalid:
* before calling `/messages`, `Err(EventCacheError::UnknownBackpaginationToken)` was returned,
* after calling `/messages`, `Ok(BackPaginationOutput::UnknownBackpaginationToken)` was returned.
This patch tries to uniformize this by only returning
`Ok(BackPaginationOutput::UnknownBackpaginationToken)`.
That's a tradeoff. It will probably be refactor later.
The idea is also to call `/messages` **before** taking the write-lock
of `RoomEvents`, otherwise it can keep the lock for up to 30secs in
this case. Also, checking the validity of the `token` **before** and
**after** `/messages` is not necessary: it can be done only after.
This patch ensures that operations on `RoomEvents` happen in one block,
by sharing the same lock.
2 new methods are created: `replace_all_events_by` and
`append_new_events`.
The test `test_reset_while_backpaginating` was expecting a
race-condition, which no longer exists. It first initially tried to
assert a workaround about this race-condition. It doesn't hold anymore.
Rewrite the test to assert the (correct) new behaviour.
This patch implements the following wrapper methods (over
`LinkedChunk`): `push_gap`, `replace_gap_at` and `events`. This patch
also implements the `reset` method that clears/drops all chunks in the
`LinkedChunk`.
There was a message sent, *then* an attempt to wait for the remote echo later. It's not ideal, because if the time setting up the waiting is high enough, and the server is fast
enough, the remote echo could come *before* we started waiting for it, resulting in timeouts. This fixes it by spawning the waiting task first, and then only sending the message.
Let's see how this helps with this test.
This adds additional checks for each room updates, and works around a few race conditions, notably one where the server would send a remote echo for a message, but not update the
computed unread_notification_counts immediately. This tends to make the test more stable, in that each response is well known and now properly tested.
It's not worth panic'ing the whole timeline because we removed the wrong item; worst case, users will complain and can send a rageshake that contains all the information that's
needed to debug what went wrong.
There could be situations where we have a day divider, then a read marker, then an event. In that case, when looking at the event, if the previous day divider is wrong and needs
to be removed, we would assume the "previous item" (= the day divider) was at position i-1, which could be that of the read marker, and we'd remove the read marker instead of the
day divider.
Closes#3265.
There is currently no way to get the URL of a homeserver's sliding sync proxy before logging in on the homeserver.
I suggest exposing the URL via the `HomeserverLoginDetail` struct after configuring the homeserver (through `configure_homeserver`).
Since the homeserver may not declare a corresponding sliding sync proxy, this value is an `Optional`. It is used later in `configure_homeserver` to check if a sliding sync proxy exists and throws an error otherwise. Previously, this check was done against the client's `discovered_sliding_sync_proxy` function.
- [ ] Public API changes documented in changelogs (optional)
Signed-off-by: Thomas Völkl <thomas@vollkorntomate.de>
Signed-off-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <public@benj.me>
Local echoes (which haven't received a remote echo yet) can have no event id, so when computing the report, don't unwrap the event id but use a sensible
default instead.
Also tweaks comments from a previous version of another PR. And rename `DayDividerAdjuster::maybe_adjust_day_dividers` to `run`.
The previous API relied on the callers not forgetting about adjusting day dividers after handling an event.
This makes it statically impossible, by requiring that `TimelineEventHandler` takes a `&mut DayDividerAdjuster` when operating, that it marks as not "consumed". Later, the caller must call `DayDividerAdjuster::maybe_adjust_day_dividers()`, to mark it as consumed. When dropping, we check that it's been consumed, otherwise we panic — as it's a developer error to not call `maybe_adjust_day_dividers()`.
Notably, split the code into smaller functions, and revamp the high-level signatures so the individual handle_ functions don't take a bazillion arguments.
The reports now include the final state as well as the set of operations to run, so we can really debug all the steps just from looking at a rageshake.
When we're removing or inserting any day divider, we're also updating the offset, so that subsequent operations happen at the right positions.
The previous code made the error to clamp the offset when assigning it, instead of letting it be "out of bounds" and clamping the uses, which is
the correct way to implement this.
The test has been failing with a timeout recently, several time. Let's see if it was a fluke caused by the low threshold (because the server might be
busy handling other requests from other tests), or an actual issue.
It appears that tarpaulin complains if the symbol information is stripped from
the binary, and as of Rust 1.77, `debug=0` causes Cargo to strip all debug
info.
To fix this, set `debug=1`.
This patch renames `ChunkIdentifierGenerator::generate_next` to `next.
This patch also simplifies a `.saturating_add(1)` to a simple `+ 1`,
which is fine because we have checked for overflow just before.
Up until uniffi 0.26 it was not possible to send objects
across the boundary unless they were wrapped in an `Arc<>`,
see https://github.com/mozilla/uniffi-rs/pull/1672
The bindings generator used in complement-crypto only supports
up to uniffi 0.25, meaning having a function which returns objects
ends up erroring with:
```
error[E0277]: the trait bound `TaskHandle: LowerReturn<UniFfiTag>` is not satisfied
--> bindings/matrix-sdk-ffi/src/room_directory_search.rs:109:10
|
109 | ) -> TaskHandle {
| ^^^^^^^^^^ the trait `LowerReturn<UniFfiTag>` is not implemented for `TaskHandle`
|
= help: the following other types implement trait `LowerReturn<UT>`:
<bool as LowerReturn<UT>>
<i8 as LowerReturn<UT>>
<i16 as LowerReturn<UT>>
<i32 as LowerReturn<UT>>
<i64 as LowerReturn<UT>>
<u8 as LowerReturn<UT>>
<u16 as LowerReturn<UT>>
<u32 as LowerReturn<UT>>
and 133 others
error[E0277]: the trait bound `TaskHandle: LowerReturn<_>` is not satisfied
--> bindings/matrix-sdk-ffi/src/room_directory_search.rs:82:1
|
82 | #[uniffi::export(async_runtime = "tokio")]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `LowerReturn<_>` is not implemented for `TaskHandle`
|
= help: the following other types implement trait `LowerReturn<UT>`:
<bool as LowerReturn<UT>>
<i8 as LowerReturn<UT>>
<i16 as LowerReturn<UT>>
<i32 as LowerReturn<UT>>
<i64 as LowerReturn<UT>>
<u8 as LowerReturn<UT>>
<u16 as LowerReturn<UT>>
<u32 as LowerReturn<UT>>
and 133 others
```
This PR wraps the offending function in an `Arc<>` to make it uniffi 0.25 compatible,
which unbreaks complement crypto.
`TryFrom` and `TryInto` are imported redundantly. They are already
defined in `core::prelude::rust_2021` which is automatically imported.
This is generating warnings on my side. This patch fixes that.
This patch makes `ChunkIdentifierGenerator::generate_next` to panic
if there is no more identifiers available. It was previously returning
a `Result` but we were doing nothing with this `Result` except
`unwrap`ping it. To simplify the API: let's panic.
As suggested in https://github.com/matrix-org/matrix-rust-sdk/
pull/3251#discussion_r1532103818 by Poljar, it is possible that the
value of the atomic changes between the `fetch_add` and the `load` (if
and only if it is used in a concurrency model, which is not the case
right now, but anyway… better being correct now!). The idea is not
`load` but repeat the addition manually to compute the “current” value.
Imagine we have this linked chunk:
```rust
assert_items_eq!(linked_chunk, ['a'] [-] ['b', 'c'] ['d', 'e', 'f']);
```
Before the patch, when we were running:
```rust
let position_of_d = linked_chunk.item_position(|item| *item == 'd').unwrap();
linked_chunk.insert_gap_at((), position_of_d)?;
```
it was taking `['d', 'e', 'f']` to split it at index 0, so `[]` + `['d',
'e', 'f']`, and then was inserting a gap in between, thus resulting
into:
```rust
assert_items_eq!(linked_chunk, ['a'] [-] ['b', 'c'] [] [-] ['d', 'e', 'f']);
```
The problem is that it creates a useless empty chunk. It's a waste of
space, and rapidly, of computation. When used with `EventCache`, this
problem occurs every time a backpagination occurs (because it executes
a `replace_gap_at` to insert the new item, and then executes a
`insert_gap_at` if the backpagination contains another `prev_batch`
token).
With this patch, the result of the operation is now:
```rust
assert_items_eq!(linked_chunk, ['a'] [-] ['b', 'c'] [-] ['d', 'e', 'f']);
```
The `assert_items_eq!` macro has been updated to be able to assert
empty chunks. The `test_insert_gap_at` has been updated to test all
edge cases.
A couple of small changes that allow us to drop locks that would
otherwise persist when running the tests over the MemoryStore, and some
documentation comments for assertions to make it easier to spot which
assertion failed, even though we are inside a macro.
Backing up room keys isn't part of the outgoing requests processing
loop, instead the user is supposed to have a separate loop calling
`BackupMachine::backup()`.
We don't need the `hyper` feature as we use our own HTTP client,
and the `keystore` will not be used in most cases.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Fallback keys until now have been rotated on the basis that the
homeserver tells us that a fallback key has been used.
Now this leads to various problems if the server tells us too often that
it has been used, i.e. if we receive the same sync response too often.
It leaves us also open to the homeserver being dishonest and never
telling us that the fallback key has been used.
Let's avoid all these problems by just rotating the fallback key in a
time-based manner.
Signed-off-by: Damir Jelić <poljar@termina.org.uk>
Co-authored-by: Andy Balaam <andy.balaam@matrix.org>
Now that there is some support for [MSC2530](https://github.com/matrix-org/matrix-spec-proposals/pull/2530), I gave adding sending captions a try. ( This is my first time with Rust 😄 )
I tried it on Element X with a hardcoded caption and it seems to work well
(It even got forwarded through mautrix-whatsapp and the caption was visible on the Whatsapp side)
---
* ffi: Expose filename and formatted body fields for media captions
In relevance to MSC2530
* MSC2530: added the ability to send media with captions
Signed-off-by: Marco Antonio Alvarez <surakin@gmail.com>
* signoff
Signed-off-by: Marco Antonio Alvarez <surakin@gmail.com>
* fixing the import messup
* fix missing parameters in documentation
* fix formatting
* move optional parameters to the end
* more formatting fixes
* more formatting fixes
* rename url parameter to filename in send_attachment and helpers
* fix send_attachment documentation example
* move caption and formatted_caption into attachmentconfig
* fix formatting
* fix formatting
* fix formatting (hopefully the last one)
* updated stale comments
* simplify attachment message comments
---------
Signed-off-by: Marco Antonio Alvarez <surakin@gmail.com>
Co-authored-by: SpiritCroc <dev@spiritcroc.de>
Previously, in a chunk with items `a`, `b` and `c`, their indices were
2, 1 and 0. It creates a problem when we push new items: all indices
must be shifted to the left inside the same chunk. That's not optimised.
This patch reverses the order, thus now `a` has index 0, `b` has index 1
and `c` has index 2.
It also removes a possible bug where we use `item_index` without
“reversing” it. This is now obvious that we don't need to re-compute the
`item_index`, we can use it directly.
This patch optimises `LinkedChunk::rchunks` and `chunks` by _not_ using
`rchunks_from` and `chunks_from`. Indeed, it's faster to not call the
inner `chunk` method + `unwrap`ping the result and so on. Just a tiny
optimisation.
This patch also uses the new `Chunk::last_item_position` method for
`LinkedChunk::items`. Abstraction for the win!
This patch implements `Chunk::first_item_position` and
`Chunk::last_item_position` as a way to _position_ the
“cursor” (`ItemPosition`) in the correct way when we want to do some
particular insertion (at the beginning or at the end of a chunk).
This patch rewrites `LinkedChunk::replace_gap_at`. Instead of inserting
new items on the _previous_ chunk of the gap and doing all the stuff
here, a new items chunk is created _after_ the gap (where items are
pushed), to finally unlink and remove the gap.
First off, it removes an `unwrap`. Second, if the previous chunk was
an items chunk, and had free space, the items would have been added in
there, which is not the intended behaviour. The tests have been updated
accordingly.
This patch implements `From<ChunkIdentifier>` for `ItemPosition`.
It's useful when we get a `ChunkIdentifier` and we need to `insert_…
_at(item_position)`.
This patch updates `ChunkContent::Gap` to hold a content `U`. Thus,
`Chunk` and LinkedChunk` both get a new generic parameter `U`. Some
methods like `new_gap` or `insert_gap_at` take a new `content: U`
parameter.
This type `RoomEvents` (that uses `LinkedChunk`) is also updated
accordingly.
The small first commit reintroduces `sanitize_server_name` in the public API. In Fractal, we use it to validate the string in the input before allowing the user to trigger the discovery.
The main commit changes a bit the discovery process: before, server names like `example.org` would only be checked for the presence of a well-known, and only URLs like `https://example.org` would be checked as a homeserver. Now, providing `example.org` will also check if `https://example.org` is the URL of a homeserver.
Sadly I don't think it's possible to add tests for it as it would require to have a homeserver accessible via HTTPS.
---
* sdk: Restore sanitize_server_name in the public API
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
* sdk: Check if a provided server name points to a homeserver during discovery
Before, only URLs like `https://example.org` would be checked as a homeserver.
Providing `example.org` will check if `https://example.org` is the URL of a homeserver.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
* Refactor to avoid duplication
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Before this patch, the meta field would be mutated, even when the transaction would be aborted. This changes the update scheme to meta
with the following:
- when creating the transaction, clone the meta (but keep the pointer location to the previous one)
- all the transaction's methods operate on the WIP meta
- when committing, replace the previous meta with the current one
This patch is a work-in-progress. It explores an experimental data
structure to store events in an efficient way.
Note: in this comment, I will use the term _store_ to mean _database_
or _storage_.
The biggest constraint is the following: events can be ordered in
multiple ways, either topological order, or sync order. The problem is
that, when syncing events (with `/sync`), or when fetching events (with
`/messages`), we **don't know** how to order the newly received events
compared to the already downloaded events. A reconciliation algorithm
must be written (see #3058). However, from the “storage” point of view,
events must be read, written and re-ordered efficiently.
The simplest approach would be to use an `order_index` for example.
Every time a new event is inserted, it uses the position of the last
event, increments it by one, and done.
However, inserting a new event in _the middle_ of existing events would
shift all events on one side of the insertion point: given `a`, `b`,
`c`, `d`, `e`, `f` with `f` being the most recent event, if `g` needs
to be inserted between `b` and `c`, then `c`, `d`, `e`, `f`'s ordering
positions need to be shifted. That's not optimal at all as it would
imply a lot of updates in the store.
Example of a relational database:
| ordering_index | event |
|----------------|-------|
| 0 | `a` |
| 1 | `b` |
| 2 | `g` |
| 3 | `c` |
| … | … |
An insertion can be O(n), and it can happen more frequently than one
can think of. Let's imagine a permalink to an old message: the user
opens it, a couple of events are fetched (with `/messages`), and these
events must be inserted in the store, thus potentially shifting a lot of
existing events. Another example: Imagine the SDK has a search API for
events; as long as no search result is found, the SDK will back-paginate
until reaching the beginning of the room; every time there is a
back-pagination, a block of events will be inserted: there is more and
more events to shift at each back-pagination.
OK, let's forget the `order_index`. Let's use a linked list then? Each
event has a _link_ to the _previous_ and to the _next_ event.
Inserting an event would be at worst O(3) in this case: if the previous
event exists, it must be updated, if the next event exists, it must be
updated, finally, insert the new event.
Example with a relational database:
| previous | id | event | next |
|----------|---------|-------|---------|
| null | `id(a)` | `a` | `id(b)` |
| `id(a)` | `id(b)` | `b` | `id(c)` |
| `id(b)` | `id(c)` | `c` | null |
This approach ensures a fast _writing_, but a terribly slow _reading_.
Indeed, reading N events require N queries in the store. Events aren't
contiguous in the store, and cannot be ordered by the database engine
(e.g. with `ORDER BY` for SQL-based database). So it really requires one
query per event. That's a no-go.
In the two scenarios above, another problem arises. How to represent
a gap? Indeed, when new events are synced (via `/sync`), sometimes the
response contains a `limited` flag, which means that the results are
_partial_.
Let's take the following example: the store contains `a`, `b`, `c`.
After a long offline period (during which the room has been pretty
active), a sync is started, which provides the following events: `x`,
`y`, `z` + the _limited_ flag. The app is killed and reopened later.
The event cache store will contain `a`, `b`, `c`, `x`, `y`, `z`. How
do we know that there is a hole/a gap between `c` and `x`? This is an
important information! When `z`, `y` and `x` are displayed, and the user
would like to scroll up, the SDK must know that it must back-paginate
before providing `c`, `b` and `a`.
So the data structure we use must also represent gaps. This information
is also crucial for the events reconciliation algorithm.
What about a mix between the two? Here is _Linked Chunk_.
A _linked chunk_ is like a linked list, except that each node is either
a _Gap_ or an _Items_. A _Gap_ contains nothing, it's just a gap. An
_Items_ contains _several_ events. A node is called a _Chunk_. A _chunk_
has a maximum size, which is called a _capacity_. When a chunk is full,
a new chunk is created and linked appropriately. Inside a chunk, an
ordering index is used to order events. At this point, it becomes a
trade-off the find the appropriate chunk size to balance the performance
between reading and writing. Nonetheless, if the chunk size is 50, then
reading events is 50 times more efficient with a linked chunk than with
a linked list, and writing events is at worst O(49), compare to the O(n
- 1) of the ordering index.
Example with a relational database. First table is `events`, second
table is `chunks`.
| chunk id | index | event |
|----------|-------|-------|
| `$0` | 0 | `a` |
| `$0` | 1 | `b` |
| `$0` | 2 | `c` |
| `$0` | 3 | `d` |
| `$2` | 0 | `e` |
| `$2` | 1 | `f` |
| `$2` | 2 | `g` |
| `$2` | 3 | `h` |
| chunk id | type | previous | next |
|----------|-------|----------|------|
| `$0` | items | null | `$1` |
| `$1` | gap | `$0` | `$2` |
| `$2` | items | `$1` | null |
Reading the last chunk consists of reading all events where the
`chunk_id` is `$2` for example, and contains events `e`, `f`, `g` and
`h`. We can sort them easily by using the `event_index` column. The
previous chunk is a gap. The previous chunk contains events `a`, `b`,
`c` and `d`.
Being able to read events by chunk clearly limit the amount of reading
and writing in the store. It is also close to what will be really done
in real life with this store. It also allows to represent gaps. We can
replace a gap by new chunk pretty easily with few writings.
A summary:
| Data structure | Reading | Writing |
|----------------|-------------------|-----------------|
| Ordering index | “O(1)”[^1] (fast) | O(n - 1) (slow) |
| Linked list | O(n) (slow) | O(3) (fast) |
| Linked chunk | O(n / capacity) | O(capacity - 1) |
This patch contains a draft implementation of a linked chunk. It will
strictly only contain the required API for the `EventCache`, understand
it _is not_ designed as a generic data structure type.
[^1]: O(1) because it's simply one query to run; the database engine
does the sorting for us in a very efficient way, particularly if the
`ordering_index` is an unsigned integer.
This adds a new mechanism in the UI crate (since re-attempts to decrypt happen in the timeline, as of today — later that'll happen in the event cache) to notify whenever we run into a UTD (an event couldn't be decrypted) or a late-decryption event (after some time, a UTD could be decrypted).
This new hook will deduplicate pings for the same event (identifying events on their event id), and also implements an optional grace period. If an event was a UTD:
- if it's still a UTD after the grace period, then it's reported with a `None` `time_to_decrypt`,
- if it's not a UTD anymore (i.e. it's been decrypted in the meanwhile), then it's reported with a `time_to_decrypt` set to the time it took to decrypt the event (approximate, since it starts counting from the time the timeline receives it, not the time the SDK fails to decrypt it at first).
It's configurable as an optional hook on timeline builders. For the FFI, it's configurable at the sync service's level with a "delegate", and then the sync service will forward the hook to the timelines it creates, and the hook will forward the UTD info to the delegate.
Part of https://github.com/element-hq/element-meta/issues/2300.
---
* ui: add a new module and trait for subscribing to unable-to-decrypt notifications
and late decryptions (i.e. the key came in after the event that required it for decryption).
* timeline: hook in (!) the unable-to-decrypt hook
* timeline: prefix some test names with test_
* utd hook: delay reporting a UTDs
* ffi: add support for configuring the utd hook
* utd hook: switch strategy, have a single hook
And have the data structure contain extra information about late-decryption events.
* utd hook: rename `SmartUtdHook` to `UtdHookManager`
* ffi: configure the UTD hook with a grace period of 60 seconds
And ignore UTDs that have been late-decrypted in less than 4 seconds.
* utd hook: update documentation and satisfy the clippy gods
* ffi: introduce another UnableToDecryptInfo FFI struct that exposes simplified fields from the SDK's version
* review: introduce type alias for pending utd reports
* review: address other review comments
This adds support for back-pagination into the event cache, supporting enough features for integrating with the timeline (which is going to happen in a separate PR).
The idea is to provide two new primitives:
- one to get (or wait, if we don't have any handy) the latest pagination token received by the sync,
- one to run a single back-pagination, given a token (or not — which will backpaginate from the end of the room's timeline)
The timeline code can then use those two primitives in a loop to replicate the current behavior it has (next PR to be open Soon™).
The representation of events in the store is changed, so that a timeline can have *entries*, which are one of two things:
- either an event, as before
- or a gap, identified by a backpagination token (at the moment)
This allows us to avoid a lot of complexity from the back-pagination code in the timeline, where we'd attach the backpagination token to an event that only had an event_id. We don't have to do this here, and I suppose we could even attach the backpagination token to the next event itself.
This doesn't do reconciliation yet; the plan is to add it as a next step.
It's a bit unclear whether the crypto-store generation counter is doing the right thing
in terms of causing us to reload the OlmMachine. There is a suspicion that things
might be keeping hold of references to the old OlmMachine.
This PR attempts to add the generation number to the logging for any operations that
hold the cross-process lock. It's obviously not bulletproof: for example, it is possible
for the OlmMachine to be replaced without holding the lock; but hopefully this will
at least help us understand what's going on.
Wiremock doesn't allow overwriting of a mock, so if we want to mock
different sync response bodies for the same path we'll have to mount the
mock in a subscobe.
This also removes the need to access some internal OlmMachine state to
get us notified about changed devices.
When all room members are loaded, we do not need an incremental member update. We know that parsing the /members response will only lead to more ambiguous names, not less. And because /members returns the complete list, we can directly use that list as the disambiguation map.
This improves the performance in my emulator from 56s to 9s and on a less performant device from 11mins to 11s (Tested experimentally on Matrix HQ using log statements in element android. If I have time, I will write a proper benchmark tomorrow.
See also https://github.com/matrix-org/matrix-rust-sdk/pull/3184#issuecomment-1986170631 for a more detailed benchmark run.
---
* members: Simplify disambiguation logic
* members: Prevent api misuse for receive_members
* members: Benchmark receive_all_members performance
* sdk: remove unused import
* sdk-base: rename `ApiMisuse` error to `InvalidReceiveMembersParameters`
* benchmarks: extract the member loading benchmark to `room_bench.rs`
* benchmarks: remove wiremock
* sdk-base: fix format
* sdk-base: try fixing tests
* benchmark: Provide some data to the store so the search and disambiguation happen
* benchmark: fix clippy
* benchmark: use a constant for `MEMBERS_IN_ROOM`
* sdk(style): reduce indent in `receive_all_members`
---------
Co-authored-by: Jorge Martín <jorgem@element.io>
Co-authored-by: Benjamin Bouvier <public@benj.me>
The idea of this patch is to explore the possibility to unify the
`all_rooms` list with the `invites` list in `RoomListService`.
Since this is entirely experimental, it's behind a new feature
flag. The feature itself can be configured at runtime by using the
new `SyncServiceBuilder::with_unified_invites_in_room_list` builder
method, or directly with `RoomListService::new_with_unified_invites`
constructor.
This is needed to be able to diff between increases and decreases of power levels ("user Alice was promoted Admin", etc.).
---
* ffi: add `previous` power levels to `OtherState::RoomPowerLevels`
This is needed to be able to diff between increases and decreases of power levels.
* ffi: please clippy
* ffi: inline initialization of `previous` and `users`
Fixes https://github.com/matrix-org/matrix-rust-sdk/issues/3191
Allows support for fetching the unstable_features from `/_matrix/clients/versions`.
Specifically, to be used for checking the state of org.matrix.msc4028 through ffi to the clients.
---
* sdk: fetch unstable_features supported by homeserver
Signed-off-by: hanadi92 <hanadi.tamimi@gmail.com>
* ffi: add can_homeserver_push_encrypted_event_to_device method
Signed-off-by: hanadi92 <hanadi.tamimi@gmail.com>
* fix: use copied instead of dereferencing
Co-authored-by: Benjamin Bouvier <public@benj.me>
Signed-off-by: Hanadi <hanadi.tamimi@gmail.com>
* fix: move can_homeserver_push_encrypted_event_to_device logic to sdk
Signed-off-by: hanadi92 <hanadi.tamimi@gmail.com>
* fix: remove unused unstable features param in client builder
Signed-off-by: hanadi92 <hanadi.tamimi@gmail.com>
* fix: use assert instead of asserteq for bool check
Signed-off-by: hanadi92 <hanadi.tamimi@gmail.com>
* fix: documentation
Signed-off-by: hanadi92 <hanadi.tamimi@gmail.com>
* Apply suggestions from code review
Signed-off-by: Benjamin Bouvier <public@benj.me>
---------
Signed-off-by: hanadi92 <hanadi.tamimi@gmail.com>
Signed-off-by: Hanadi <hanadi.tamimi@gmail.com>
Signed-off-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <public@benj.me>
- `Room::build_power_level_changes_from_current()` was replaced by `Room::get_power_levels()`, which now returns an SDK/Ruma `RoomPowerLevels` value containing all the data we need to display these values in UI and not only the customised values.
- `Room::reset_power_levels()` was added to the FFI layer.
It maps user ids to users' power levels.
Also, make sure it just returns an empty map if this info is not available, instead of crashing.
Then use it in the FFI side to output updated data for the `RoomInfo`.
---
* sdk: create new `users_with_power_levels` fn which maps user ids to users' power levels
Also, make sure it just returns an empty map if this info is not available, instead of crashing.
* Update crates/matrix-sdk/src/room/mod.rs
Co-authored-by: Benjamin Bouvier <public@benj.me>
Signed-off-by: Jorge Martin Espinosa <angel.arasthel@gmail.com>
* Improve tests
---------
Signed-off-by: Jorge Martin Espinosa <angel.arasthel@gmail.com>
Co-authored-by: Benjamin Bouvier <public@benj.me>
Having `EventCache::for_room` return an `Option` avoids the need for the error. One caller can
safely unwrap it, and others can log and ignore rooms that have disappeared (like we do in `call_sync_response_handlers`).
This does two things:
- raise the timeout value for the `/members` queries, because they can super slow in Synapse because of nasal demons,
- and aggregate all the fields of `RoomMember` early, so users don't have to call getters to read each field.
---
* room_members: Export plain old data for room members to avoid FFI calls
Signed-off-by: Timo Kösters <timo@koesters.xyz>
* room_member: Respond to review feedback
* room_member: Remove previous RoomMember struct
* ffi: rename ExportedRoomMember to RoomMember
---------
Signed-off-by: Timo Kösters <timo@koesters.xyz>
Co-authored-by: Benjamin Bouvier <public@benj.me>
Changes:
- sdk: Add `get_user_power_level` and `get_suggested_user_role` functions so we don't need to load the whole room member list to know if a user has some power level/role.
- ffi: add an FFI fn for `get_suggested_user_role`.
- ffi: add `user_power_levels` to `RoomInfo`.
The goal of this PR is being able to fetch a user's power level or role almost immediately and avoid having to load and find the user in the room member list, which can be very slow to load (especially in EX Android).
---
* sdk: Add `get_user_power_level` and `get_suggested_user_role` functions to get the power level for a user without loading the room member list.
* ffi: add `suggested_role_for_user` fn, which calls the new `get_suggested_user_role` fn in Room
* sdk: add test
* ffi: add user power level info to `RoomInfo`
* Add changes to changelog
* sdk: Fix docs formatting
* sdk & ffi: use `&UserId` instead of `OwnedUserId`
Also, simplify error mapping.
* sdk: add extra test
* ffi: fix `OwnedUserId` -> `&UserId` conversion
* sdk: Replace `UserId::parse` with `user_id!` macro for literals in tests
* Update crates/matrix-sdk/tests/integration/room/joined.rs
Signed-off-by: Benjamin Bouvier <public@benj.me>
---------
Signed-off-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <public@benj.me>
Mostly an optimization that was also revealed by the previous version of the test failing, because it would receive an initial update with empty
events.
This adds a few basic tests for the event cache, notably one for the `add_initial_events`, for something I identified while working on the code, but
that could've caused bad issues:
The `test_add_initial_events` checks that even if we received updates with meaningful events for a room, the room's events are cleared before we add initial events (since
we have no ways to know where to insert the events, in this case). In the future, we can keep this test as a "smoke test" for basic functionality of
the event cache.
We're subscribing to settings updates *after* sending a request to change a setting. In an unlucky scenario, the following sequence of events could happen:
- sending request to change the settings
- response is received
- we set up the receiver to settings updates, but it's too late
The fix would then be to subscribe to the changes *before* we even send the request to update settings.
It means the `wiremock` dependency is not a `dev-dependency` anymore, but an optional `dependency` enabled only if `testing` is enabled.
It seems like a fine tradeoff to me.
To do so, we need to put the weak reference `EventCache -> Client` back into the `EventCache`.
Putting the `sdk::Room` in the `RoomEventCache` will be useful to run room queries like backpagination (aka call `Room::messages()`).
The ClientBuilder already exposes setting the proxy, but sadly the
AuthService doesn't let you configure the ClientBuilder directly (yet).
So logging in with a proxy wasn't supported until now.
There is a race condition in `EventCacheInner::for_room`. Before this
patch, it was possible for 2 concurrent execution to do the following:
* Execution 1 takes the read lock; there is no room in `by_room`, so it
creates the `RoomEventCache`, code is suspended while trying to get
the write lock.
* Execution 2 takes the read lock; there is no room in `by_room`, so it
creates the `RoomEventCache`, code is _not_ suspended because _insert
reason_, the write lock is acquired, and the `RoomEventCache` is saved
and a shallow clone is returned.
* Execution 1 resumes, the write lock is acquired, the `RoomEventCache`
overwrites the one from Execution 2, and a shallow clone is returned.
Now Execution 2 holds a `RoomEventCache` that isn't saved in `by_room`.
It's a ghost! Worst, because the store is cloned in both
`RoomEventCache`, 2 versions will overwrite themselves in the store
constantly.
This patch uses a single write lock, and uses the `BTreeMap::entry`
API instead. Thus, the race condition disappears.
This patch also changes the return type of `for_room` to make it
infallible. It was already the case: only the `Ok` value was returned.
This patch imports the `spawn` function from
`matrix_sdk_common::executor` instead of `tokio`.
`matrix_sdk_common::executor` adds support for WebAssembly.
* event cache: move it to the main SDK crate
* event cache: add requested Debug impl to `RoomEventCacheUpdate`
Somehow the compiler asked for it now...
* event cache: add missing copyright notice to store file
* event cache: use a weak reference to the client internally
This will make it possible to have the `Client` own an `EventCache` without a reference
cycle.
* event cache: move the spawned task to its own function
* event cache: move RwLock from EventCache::inner to the only mutable field inside EventCacheInner
* event cache: have the Client own *the* event cache
The goal is to have a unique EventCache instance overall, that's available from everywhere in
the SDK, notably when creating timelines for rooms.
Because the event cache only owns a weak reference to the client, it means the Client still
can be dropped, In turn, this will close its sender of `RoomUpdates`, which will gracefully
close the task spawned in `EventCache::new` after it's done handling the latest updates.
* event cache: process room updates one at a time
* timeline: use the client-wide event cache instead of spawning one per timeline
This now means that we're passing the "initial events" to the event cache just before initializing
the timeline. As a result, there might be previous events that the event cache saw (coming from
sync), but now we can't decide where to put them; drop previously known events in that case.
* event cache: hey, turns out we don't even need the weak back-link
Keeping it as a separate commit, to make it easier to revert later.
* event cache: remove unused errors
Keeping the error type and results, though, because we might have store errors soonish.
* fixup! event cache: move the spawned task to its own function
* event cache: manually subscribe to the event cache
It was a bad idea to have it enabled by default, since some users may not be interested in all
updates for all rooms (e.g. bots). Instead, we make it so that the event cache must be
explicitly subscribed to, and we do it in two cases:
- in the UI `TimelineBuilder::build` method, because we're interested in updates to the current
room,
- in the `RoomListService`, because we *will* be interested in updates to room derived data (e.g.
unread counts, read receipts, and so on).
This avoids a bit of fiddling when creating the event cache in the client.
This is resilient when a parent Client is forked into a child Client, because the child
`EventCache` share the same subscription as the parent's.
It doesn't make sense to set a thread identifier for the receipt of the latest event: the event might not belong to that thread, and the SDK would need to check that,
since the latest event isn't reachable from the outside world (the reason why `mark_as_read` had been introduced). So let's remove it for now, and
add comments related to threaded receipts.
This slightly changes the API when interacting from the FFI layer:
- instead of `mark_as_unread` and `mark_as_read`, there's now a single method `set_unread_flag(bool)`, which callers may call with true (i.e. unread) or false (i.e. not unread).
- there's a new method `mark_as_read` which sends a read receipt to the latest event in the timeline, using other commits from the same PR,
- forcing a room as read requires calling first `set_unread_flag(false)` then `mark_as_read()`
It's wrong that the first client, used only to determine how to log in and find the user id, try to run the encryption initialization tasks.
In particular, it should not even try to bootstrap the account, as this may send OTKs to the server, which the client will forget about as soon
as it's respawned as a database-backed client.
I have also changed the priorities so that manual updates are preferred.
This means that duplicate updates do not happen if the room was previously unknown.
Signed-off-by: Timo Kösters <timo@koesters.xyz>
Vectors grow in powers of two while reading, which is doubly wasteful:
* causes every byte to be copied in average once
* leaves the vector in average 25% larger than it needs to be, wasting memory.
For example, adding this test to
`crates/matrix-sdk-crypto/src/file_encryption/attachments.rs`:
```
fn encrypt_decrypt_minimize_memory() {
let data = std::iter::repeat("abcdefg").take(10000).collect::<String>();
let mut cursor = Cursor::new(data.clone());
let mut encryptor = AttachmentEncryptor::new(&mut cursor);
let mut encrypted = Vec::new();
encryptor.read_to_end(&mut encrypted).unwrap();
let key = encryptor.finish();
assert_ne!(encrypted.as_slice(), data.as_bytes());
let mut cursor = Cursor::new(encrypted);
let mut decryptor = AttachmentDecryptor::new(&mut cursor, key).unwrap();
let mut decrypted_data = Vec::new();
decryptor.read_to_end(&mut decrypted_data).unwrap();
assert_eq!(
decrypted_data.len(),
decrypted_data.capacity(),
"{} bytes wasted by decrypted_data",
decrypted_data.capacity() - decrypted_data.len()
);
}
```
errors with:
```
assertion `left == right` failed: 61072 bytes wasted by decrypted_data
left: 70000
right: 131072
```
By initially setting this capacity, the vector should be slightly larger
than needed from the start, avoiding both copying and leftover capacity
after decryption is done.
`matrix_sdk_ui::timeline::Message::msgtype` returns `&MessageType`,
and variants of `MessageType` implement `MediaEventContent`, so it is
allowing references here avoids cloning message content.
This didn't show up, and it's quite useful to know for which room we're trying to send a receipt, without having to look up the room based on the
target event id.
This patch rewrites all the integration tests for the notable tags. The
tests were good, but we could merge some together. The names were too
long, they have been shortened.
The workflow inside `Client::receive_sync_response` is a little
bit fragile. When `Client::handle_room_account_data` is called, it
modifies `RoomInfo`. The problem is that a current `RoomInfo` is being
computed before `handle_room_account_data` is called, and saved a
couple lines after, resulting in overwriting the modification made by
`handle_room_account_data`.
This patch ensures that the new `RoomInfo` is saved before
`handle_room_account_data` is called.
Now that `NotableTags` has replaced `RoomNotableTags` entirely, this
patch can rename `NotableTags` to `RoomNotableTags` as it's a better
name for it.
Note that this type is private to `matrix_sdk_base`, so it's fine to
rename it.
The Matrix specification uses the `m.favourite` orthography. Let's use
the same in our code. So `set_is_favorite` becomes `set_is_favourite`.
This patch updates this in various places for the sake of consistency.
The previous patch has introduced the new `NotableTags` type to replace
the `RoomNotableTags`. This patch removes the latter.
This patch keeps the `Room::set_is_favorite` and `::set_is_low_priority`
methods. However, this patch adds the `Room::is_favourite` and
`::is_low_priority` methods, with the consequence of entirely hiding the
notable tags type from the public API.
This patch is the first step to remove the [`RoomNotableTags`] API. The
idea is to compute the notable tags as a single 8-bit unsigned integer
(`u8`) and to store it inside `RoomInfo`. The benefits are numerous:
1. The type is smaller that `RoomNotableTags`,
2. It's part of `RoomInfo` so:
- We don't need an async API to fetch the tags from the store and to
compute the notable tags,
- It can be put in the store,
- The observable/subscribe mechanism is supported by `RoomInfo` behind
observable instead of introducing a new subscribe mechanism.
This patch exposes the 1-to-1 encryption method that is usually used to share a room key with a device. Users might want to send encrypted custom to-device events to a device directly, so let's expose this functionality.
Co-authored-by: Damir Jelić <poljar@termina.org.uk>
We don't want the rest of the method to abort executing because
automatic backup creation failed.
We also move the infallible event handler registration at the top of the
method.
The logic to automatically create a backup uses the following logic:
1. We check if we need to create a backup, this includes a check if a
backup exists on the server.
2. We conclude that we should create a backup, no backup exists on the
server.
3. The method to create the backup checks again if a backup exists, this
was an API change because the method is public and users misused the
method.
4. Finally, we create the backup.
A race exists between the first time we check if a backup exists and the
second time, i.e. step 1 and 3.
It seems that some users create two Client objects which then make this
race a common occurrence. Clients should not do that, but at least we
don't error out too soon anymore if the automatic creation of the backup
fails.
Quick performance improvement on `save_change` for indexeddb.
All the serialization/encryption is now done outside the db transaction
---
* do serialization/encryption before db transaction
* clippy
* add changelog for indexeddb crate
* clean comments
* fix typo
* Review: Fix typo in changelog
* Review: refactor, rename DbOperation to PendingOperation
* Review: rename variant PutKeyVal to Put
* Review: fix doc typo
* Review: rename perfrom_operation to apply
* Review: remove unneeded isEmpty checks
* Review: refactor better API for PendingStoreChanges
* Review: Prefer BTreeMap to HashMap
* Refactor: rename IndexeddbChangesKeyValue
* Refactoring: get the list of affected store from PendingIndexeddbChanges
* cleaning
* Review: Better names and comments
* Review: use filter_map instead of filter then map
fix: Preserve the event of any settings that are changed back to the default level.
sdk: Rename get_room_power_levels (drop the get).
chore: Refactor with more sensible naming.
sdk: Clean up the RoomPowerLevelChanges API.
The IndexMap::remove method has been deprecated, the documentation[1] on
the method tells us that we can replace the usage of it with
IndexMap::swap_remove:
> NOTE: This is equivalent to .swap_remove(key), replacing this entry’s
> position with the last element, and it is deprecated in favor of
> calling that explicitly.
[1]: https://docs.rs/indexmap/2.2.2/indexmap/map/struct.IndexMap.html#method.remove
This patch implements the `category` room list filter. It introduces a
new type: `RoomCategory`, to ensure that “group” and “people” are
mutually exclusives.
This patch implements `Room::direct_targets_length`. It avoids to call
`Room::is_direct` if and only if we don't care about the room's state
and we don't want an async call, and if we don't want to pay the cost of
`Room::direct_targets` which clones the `HashSet` as an alternative way
to get a similar information than `Room::is_direct`.
This patch implements the `all` and `any` filters in `matrix-sdk-ffi`.
The `not` filter cannot be implemented because recursive enum isn't
supported by UniFFI (see https://github.com/mozilla/uniffi-rs/issues/396).
This patch introduces the
`matrix_sdk_ui::room_list_service::filters::Filter` trait alias.
This patch also cleans up a little bit the filters by renaming some
methods for the sake of consistency across all the existing filters.
A decryption failure would have prevented the recording of the session
otherwise. We are mostly interested in the state of the session if a
decryption failure happens.
This patch inlines `SlidingSyncRoom::latest_event` into its unique call
site: `SlidingSyncRoomExt::latest_timeline_item`.
`SlidingSyncRoom::latest_event` is not using any data from
`SlidingSyncRoom`, except the `Client`. So it can easily live somewhere
else. Our goal is to clean up `SlidingSyncRoom` as much as possible, and
to remove any kind of logic from it.
Since my investigation found that it significantly speeds up deletion of
a store on both Firefox and Chromium if you clear() it first, do that in
our migration code.
This patch removes the need to store and to update the
`SlidingSyncRoom::inner` field (of type `v4::SlidingSyncRoom`). Now,
we just need to handle the `prev_batch`, which is the only data we care
about.
Bonus: it reduces the size of the frozen sliding sync room quite much.
This patch removes the `SlidingSyncRoom::is_dm` method. It's not used
anywhere in the SDK, neither by the FFI bindings. Since this code is
still experimental, it's fine to remove public API.
This patch removes the `SlidingSyncRoom::is_initial_response` method.
It's not used anywhere in the SDK, neither by the FFI bindings. Since
this code is still experimental, it's fine to remove public API.
I think the reasoning behind using a RwLock for sync, and notably allowing multiple concurrent `.read()` lock taking was flawed: since there's no way
to identify which subpiece of the store we're reading and writing, there could be two concurrent requests to the write to the same thing at the same
time. To get rid of this possibility, this commit changes the lock to a single access only Mutex lock.
This patch removes the `room_list::Room::has_unread_notifications` and
the `::unread_notifications` methods. Since the previous commits,
the respective `SlidingSyncRoom` methods have been removed too. It's
better to use the `Deref` implementation of `room_list::Room` to
`matrix_sdk::Room` to use the correct methods.
This patch removes the `SlidingSyncRoom::has_unread_notifications`
and the `::unread_notifications` methods. It doesn't hold any relevant
information that `matrix_sdk::Room` can provide.
`SlidingSyncRoom` no longer has an `avatar_url` method.
`room_list::Room` no longer needs to check first in sliding sync then in
`Room` as a fallback.
This patch removes the `room_list::Room::avatar_url` method.
This patch also implements `Deref` for `room_list::Room` to
`matrix_sdk::Room`, to make our lifes easier.
With the previous commit, the avatar is properly synchronized with the
`Room`. The result is that `SlidingSyncRoom` no longer needs to hold
the `avatar_url`.
To update the avatar of a room, one has to look up in the state event.
That's the canonical way to do. For Sliding Sync, it means looking
inside the `required_state` field of the `v4::SlidingSyncRoom`. This
case already works and was tested.
However, a `v4::SlidingSyncRoom` comes with a direct `avatar` field.
It's another way to know the avatar URL of the room. This case was
handled and tested in `matrix_sdk::sliding_sync::SlidingSyncRoom`, but
it was never propagated into the proper sync mechanism. So when the
`avatar` field was set up, `matrix_sdk::sliding_sync::SlidingSyncRoom`
was holding this information, and the `avatar` wasn't defined in the
proper `Room`: `SlidingSyncRoom` has to look up inside the `Room` as
a fallback.
This patch is the first one to fix this “fallback” mechanism.
The `avatar` field of a `v4::SlidingSyncRoom` now triggers an update to
the new `RoomInfo::update_avatar` method (à la `update_name`), via
`process_room_properties`.
It's only used in test code, so it's not worth exposing to the SlidingSyncRoom object (and the Room already has such a timeline function, if needs be).
Makes AmbiguityChange easier to use, especially outside of SyncResponse,
where we might not have the m.room.member event.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
The room sync could be over, and then the timeline items would not contain the updated item yet, if the timeline didn't get a chance to finish its
internal update. Fix this by:
1. limiting the sync time to 10 seconds
2. giving up to one second for the timeline to update internally, by watching its stream of events (we're ignoring stream updates just after this loop)
Adds a `TimelineEventTypeFilter` enum that either returns only events whose event_type is included in a set of allowed event types, or all events but those whose event types are in a list of excluded event types.
Also adds `TimelineEventTypeFilter` so the clients can use it to define those lists of event types, which are then converted to ruma `TimelineEventType` and used for filtering.
---
* matrix-sdk-ui: add `TimelineEventTypeFilter` to filter timeline events by either including only those of some event types or all but the ones that match those event types.
* ffi: add bindings to `TimelineEventTypeFilter` and `FilterTimelineEventType` so we can provide these event types from the FFI clients
* Fix format
* Fix tests
* Fix format again (using nightly toolchain)
* Remove `all_filter_...` functions as there is no right way to support it at the moment and they're just helpers
* Improve tests
* Make `TimelineEventFilterFn` public so it can be used in several layers.
* Make `TimelineEventTypeFilter` a struct in the FFI layer
* Add fns for creating a timeline with cache and event type filters
* Remove dead code
* Fix some review comments
* ffi: create new timeline initialization APIs, modify existing ones.
ui: make `Room::timeline()` return `None` if no timeline exists instead of lazily creating one.
More details:
- Added `init_timeline_with_builder` to `matrix_sdk_ui::room_list_service::Room` so a timeline can be initialized at will given a `TimelineBuilder`.
- Create `is_timeline_initialized()` fns in both the ui and ffi layers to check the status of the timeline.
- Make `matrix_sdk_ui::room_list_service::Room::timeline()` only return a timeline if it's already been initialized.
- Create FFI functions to expose these UI ones.
* Fix tests
* Fix some review comments
* Update bindings/matrix-sdk-ffi/src/room_list.rs
Signed-off-by: Benjamin Bouvier <public@benj.me>
---------
Signed-off-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <public@benj.me>
It's started failing with timeouts in multiple PRs, so this PR should help spot where the issues specifically are, since codecov test failures are so hard to reproduce locally.
Podman running unprivileged containers by default, using system folders
can result in permission issues inside the containers.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This fixes an issue with the sliding sync processing where room account data are not processed when the associated room has no other changes.
---
* sliding_sync : fix room account data not being processed when there is no other changes in the room
* sliding_sync : properly handle room_account_data from extension
* sliding_sync : add test for processing rooms_account_data
* sliding_sync : fix formatting
* Apply suggestions from code review
Co-authored-by: Ivan Enderlin <ivan@mnt.io>
Signed-off-by: ganfra <francois.ganard@gmail.com>
* sliding_sync : avoid cloning room account data events
Co-authored-by: Benjamin Bouvier <public@benj.me>
Signed-off-by: ganfra <francois.ganard@gmail.com>
---------
Signed-off-by: ganfra <francois.ganard@gmail.com>
Co-authored-by: Ivan Enderlin <ivan@mnt.io>
Co-authored-by: Benjamin Bouvier <public@benj.me>
A quick performance improvement to not do the deserialization/decryption inside the transaction.
---
* quick perf inbound_group_sessions_for_backup
* log on deserialize error
* Review: Unneeded use of :? for an object with Display trait
Co-authored-by: Benjamin Bouvier <public@benj.me>
Signed-off-by: Valere <bill.carson@valrsoft.com>
* Doc: fix capitalization
Co-authored-by: Benjamin Bouvier <public@benj.me>
Signed-off-by: Valere <bill.carson@valrsoft.com>
* Review: better naming
---------
Signed-off-by: Valere <bill.carson@valrsoft.com>
Co-authored-by: Benjamin Bouvier <public@benj.me>
This allows us to use the native `SystemCleaner` in Android 13+ which is a bit safer regarding deadlocks than the JNA alternative we'd use otherwise.
Note this breaks compatibility with pure JVM environments, meaning the library can only be used in Android.
Before, to avoid saving a room info if the read receipts `RoomInfo` field hadn't changed, we used explicit reporting to the caller that a field
changed value. This way was error-prone, and there's been bugs twice in this area: one fixed in a previous PR,
and there was another one in the existing implementation (when the pending list shrinked, it wouldn't tell the caller).
This commit changes the implementation so that it we snapshot the previous read receipts, and after processing unread counts, we compare the
new version with the snapshot, relying on `PartialEq` to do the leg work here. This is more future proof and less error prone.
Also `compute_unread_counts` isn't fallible, so doesn't need to return a Result.
This is renamed to `has_up_to_date_read_marker_item`, and inverted in meaning, compared to its previous meaning.
It's simpler to think of it as the presence of the read marker item: if it's not there, then that's the only case where we'd need to worry about adding one.
This patch updates `SecretStorageKey::check_zero_message` to assume
that a missing `iv` and/or `mac` is valid, instead of an error, as the
spec suggests.
The real reason why the test wasn't passing was the same root cause as #3031. The client's room member information was missing, meaning we couldn't compute a push context, so we couldn't compute notifications/mentions either. The fix is in the testing code itself, the sliding sync should request the `$ME` room member state event to work correctly and non-racily.
This was quite handy during development of the client-side computation of read-receipts, to analyze some bugs.
It might be not useful to have it checked in for long, but I would love to make sure it keeps on compiling
until we have a more stable handling of read receipts in general.
The default implementation of `RingBuffer` was setting a capacity of 0.
This was incorrect as it wasn't possible to insert any items. This patch
updates it to take a `NonZeroUsize` so that it's impossible to set a
negative or zero capacity.
We need to make sure we guard against servers downgrading the encryption
settings, or breaking them with unsupported algorithms. It's not *entirely*
clear what the expectation is here, but legacy crypto in matrix-js-sdk blocks
any changes to the settings at all, so we'll follow suit for now.
Without this, we wouldn't ever have the member information for the logged-in user, the first time they log in, if they don't happen to be the
last user sending a message in a room (a case covered by `$LAZY`).
Necessary for the `.m.rule.invite_for_me` rule that should only happen in invited rooms.
Requires to create a `Notification` type that accepts stripped state events. It is simpler than Ruma's type because it lacks fields with data that is made up or redundant.
Tested locally with Fractal.
Fixes#1912.
---
* Upgrade Ruma
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
* base-client: Support notifications in invited rooms
Necessary for the .m.rule.invite_for_me rule that should
only happen in invited rooms.
Requires to create a Notification type that accepts stripped state
events.
It also removes fields with data that is made up or redundant.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
* matrix-sdk-test: Add macros to construct raw sync or stripped state events
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
* client: Add tests for register_notification_handler
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
* Upgrade Ruma
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
* Fix base changelog
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
* Fix methods on Room
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
* Fix FFI
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
* Fix FFI RoomMember
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
* Simplify and_then chain
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Before, restoring a timeline from the cache, for which we had a receipt for the last element, would not mark that receipt as "interesting", thus
considering that all the events are new. This is incorrect, and another way to fix that would be to set `best_receipt` to the initial read receipt;
but having the information that we had a change is more useful in general, so the comparison is changed from strict to lax here.
When restoring a timeline from the disk cache, or when we get a timeline from a limited sync, we can have events in common to the two sets, messing
up with the count. In that case, forget about previous events, since we don't try to stitch timelines yet.
This is a workaround because the proxy may send the same event multiple times in a sync timeline. When that happens and we had the read receipt
on the duplicated event, it may cause some events to be counted twice, resulting in invalid counts. This patch fixes it by resetting the count
*every* time we see the matching event.
In existing code the bootstraping is done in different places:
- in `bootstrap_cross_signing_if_needed`
- or in `run_intialization_tasks`
And both are based on the same EncryptionSettings.
`bootstrap_cross_signing_if_needed` was done by `LoginBuilder` but `run_intialization_tasks` by `MatrixAuth`.
I propose to move all that under `run_intialization_tasks` that has already support to do it in background, and to call it in one place only: `MatrixAuth`
Also Fixes https://github.com/matrix-org/matrix-rust-sdk/issues/2763
## Notes
- Some tests have been updated to properly `wait_for_e2ee_initialization_tasks`
- `bootstrap_cross_signing_if_needed` might require re-authentication. Which I suppose was the original reason to have that in a seperate place. But given that the need to re-auth will soon be deprecated for bootstrap (when this [MSC](https://github.com/matrix-org/matrix-spec-proposals/pull/3967) will land); so for now we pass the authentication info to `run_intialization_tasks` if any.
---
* refactor(bootstrap): init cross-signing with other e2e initial task
* fix compilation warning for NoEncryption feature set
* Doc and Formatting: Improve doc and formatting
* Fix missing import with encryption feature flag
Replace `make_store_config` with a pair of funtions `open_stores_with_name` and
`open_state_store`. This allows us to remove a dependency on
`matrix-sdk-base/e2e-encryption`, and generally simplifies things.
Instead of needing three methods, we now only need one: since we use the
crypto crate enum directly, we no longer need to convert between the
crypto crate and main crate enums.
Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
Specifically, this removes the following redundant types in the main
crate:
- enum UserIdentities
- struct OwnUserIdentity
- struct OtherUserIdentity
This is possible because `OwnUserIdentity` and `OtherUserIdentity` are
exactly the same as their crypto crate counterparts, except they also
wrap a `Client`.
As a consequence, the main crate enum `UserIdentities` is isomorphic to
a struct containing:
1. the crypto crate `UserIdentities`
2. a `Client`
So this commit performs that transformation.
Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
Some sliding sync responses may include extension data that will cause an update to the room, but the room itself may not be in the set of
rooms as returned in the top-level `rooms` fields of the response. This may cause missed updates for rooms, since notifiers won't be notified
about those.
This fixes that, by making sure that a `RoomInfo`-only update will cause the room (and the lists that contain it) to be marked as updated.
When using the same pattern of `client.register(); client.login_*` that is used by tests in [our Acter app, strangely showed that more than one device/session had been opened](https://github.com/acterglobal/a3/issues/938).
Turns out, if the server is set up for it, according [to the spec](https://spec.matrix.org/v1.8/client-server-api/#post_matrixclientv3register) registration _is_ already a login and both device id and access token will be returned ... which are then straight-up ignored by the matrix-sdk Client and without any way of setting it from the outside, one _must_ login again, creating a second unnecessary device/session.
This PR adds an integration test case that checks that upon the minimal registration-login-flow only one device/session is found (as is to be expected from the outside), surfacing the error. It also contains a "somewhat fix" (as we need this in the app right now), but as it a) changes the behavior of the current API and b) isn't fully implementing the encryption-bootstrapping-pattern (and thus fails a different test), I'd leave it open to discussion whether that was appropriate way to go.
---
* Test showing registration-login-pattern leads to too many devices
* Fix too-many-devices-bug
* Do not panic on failure to set session, refactor and add docs
* refactor bootstrap crosssinging
* Fixup
* Use new post_login_cross_signing feature from Oidc, too
* unwrap at construction for less verbose code
* remove comment from test
* make new fn pub(crate) to fix build
… and specify a version such that publishing of the ui crate becomes possible.
We still use a git dependency for FFI crate builds because there were some
likely important breaking changes that haven't been released.
This is required to publish `matrix-sdk-ui`.
In migrate_data_for_v6, we incorrectly copied the keys in inbound_group_sessions verbatim into inbound_group_sessions2. What we should have done is re-encrypt them using the new table name, so we fix that up with a new migration here.
This caused the bug because we were looking for sessions to mark as backed up by calculating their key (from room_id and session_id) but that key did not exist, because the old sessions were stored under the incorrect keys. So no sessions were marked as backed up, and we repeatedly tried to re-mark them.
Before this patch, we needed to clone the inner `timeline_queue` and turn it into a concrete `Vec<SyncTimelineEvent>`, just to iterate on the elements,
and because returning an iterator from a trait method is impractical. This now changes it to return the actual concrete type of `timeline_queue`, so
we don't need the extra allocations.
Ideally, matrix-sdk and matrix-sdk-base would be merged, so we don't need to use a trait at all here.
* Bring back original (slower) build style under the `sequentially` flag to work around new build style hanging on older machines
* Apply suggestions from code review
Signed-off-by: Benjamin Bouvier <public@benj.me>
---------
Signed-off-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Benjamin Bouvier <public@benj.me>
The room_info variable in this function contains the latest_event, and later it's set as the Room's room info (`inner.inner`) field.
Calling `set_latest_event` manually here will cause an update of the `RoomInfo` subscriber, while we're not done processing the full
room, and a room info update will happen anyways later (when the entire room is processed), so this one is spurious and will only
show a partial update of the fields.
- We can look up the session meta only once, it's not useful to do it once per event as it's
loop-invariant.
- We can look at the events backwards and stop after the first room membership event we see
in that order.
* maybe trigger backup at end of sync
* add test for backup upload
* missing e2e cfg
* add sliding sync tests
* fix borrow
* fix indent
* fix naming from copy/paste
* Remove extracted function
* fix clippy
* style: inline a few variables only used once into their use sites
---------
Co-authored-by: Benjamin Bouvier <public@benj.me>
This patch improves `test_media_content` to ensure that, in case of
multiple medias, only the expected ones are removed. Previously, the
test wasn't testing _other_ medias that should be kept in case of
removals.
This patch continues to improve `test_media_content` to ensure that the
content of the media are the expected ones.
Finally, this patch updates the `MemoryStore` implementation to make
tests happy.
This patch rewrites `MemoryStore::add_media_content`,
`::get_media_content`, `::remove_media_content` and
`::remove_media_content_for_uri` to (i) work on `mxc://` URI instead
of “unique key”, and (ii) to handle removal correctly thanks to the new
`RingBuffer::remove` method.
If `use_cache` is true and the cache exists, let's return everything in
one go instead of declaring a `content` variable. It makes the code
easier to read and to understand.
This is a rather over simplistic media cache implementation for the
`MemoryStore`.
It's based on a `RingBuffer` of size 20.
`remove_media_content` pops all medias until the correct one is met (if
it exists). `remove_media_content_for_uri` removes all medias, it
ignores the URI.
A `From` implementation is part of the public API
and this conversion does not make sense outside of the module.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch updates `Timeline::send_image` and `Timeline::send_video` so
that `thumbnail_url` is now an `Option<String>`.
The idea is to allow sending an image or a video without a thumbnail.
This patch removes `RUNTIME.block_on` inside `Client::get_media_file`,
`::upload_media`, `::get_media_content` and `::get_media_thumbnail`, and
makes those methods async.
Useful if we want to know in what room
we just sent a VerificationRequest,
with UserIdentity::request_verification
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
… and fix build swift errors.
- The XCFramework was being built with dylibs which aren't supported on iOS
- The tests build was attempting to generate uniffi from a moved file
`notifications` were stored in the `StateChanges` struct, which made me think that they're then persisted in the database. It's not the case, they
were just stored there by convenience. This commit changes it to a parameter that's passed, as it's not too invasive and clearer that this is only
transient data.
`itertools::format` is known to cause issues when its display implementation is being
used multiple times, as it consumes the iterator it was given (and that can only happen once,
unless caching it). This is bad, as our production apps may have multiple subscribers they will
run into a panic.
The fix is to reify the debug string before it's logged, so the tracing consumer will only see
a string, and not the display implementation that would panic on the second use.
Currently, querying for inbound group sessions which need backing up is very
inefficient: we have to search through the whole list.
Here, we change the way they are stored so that we can maintain an index of the
ones that need a backup.
Fixes: https://github.com/vector-im/element-web/issues/26488
Fixes: https://github.com/matrix-org/matrix-rust-sdk/issues/2877
---
* indexeddb: Update storage for inbound_group_sessions
Currently, querying for inbound group sessions which need backing up is very
inefficient: we have to search through the whole list.
Here, we change the way they are stored so that we can maintain an index of the
ones that need a backup.
* Rename functions for clarity
* Remove spurious log line
This was a bit verbose
* Rename constants for i_g_s store names
* improve log messages
* add a warning
* Rename `InboundGroupSessionIndexedDbObject.data`
* formatting
Otherwise the matrix-sdk crate can be used with an older version of
ruma-client-api which uses different types for a sliding sync type.
The sliding sync breaking change was allowed to be part of a patch
release because sliding sync is an unstable feature in Ruma.
Somehow `fmt::pretty` manages to be both overcomplicated and not flexible
enough.
The driver here is that I want to put the event "fields" on a separate line to
the message.
This PR fixes the `EventTimelineItem.is_editable()` function for polls.
Before this changes it always returned false.
Now it consider poll's preconditions for editing:
- The poll has no votes yet
- The poll hasn't an end event
Sometimes, we called this `keys query`, sometimes `/keys/query`, and sometimes,
just for variety, `keys/query`. Generally in Matrix we talk about `/keys/query`
so let's standardise on that.
Until https://github.com/matrix-org/matrix-rust-sdk/pull/2862,
`GroupSessionManager::encrypt_session_for` called `Device::encrypt` (via
`Device::maybe_encrypt_room_key`.
`Device::encrypt` is a thin wrapper for `ReadOnlyDevice::encrypt`, but it also
has an `instrument` annotation.
https://github.com/matrix-org/matrix-rust-sdk/pull/2862 short-cuts
`Device::encrypt` and calls `ReadOnlyDevice::encrypt` directly: that was
functionally fine but of course means that we no longer benefit from the
`instrument` annotation.
This PR rectifies the situation by pushing the annotation down to
`ReadOnlyDevice::encrypt`. It also adds some documentation for that function,
since we are using it in more places now.
(Longer-term, I think we should probably aim to get rid of `Device::encrypt`
altogether, but that's a refactor I don't want to take on today.)
This is useful if we ever decide to switch to X3DH for the session
establishment. It also refactors a bit the /keys/claim response handling
method.
Co-authored-by: Jonas Platte <jplatte@matrix.org>
The mocks in some notification tests mock any PUT/DELETE request without
the request to hit a certain path. These mocks then might respond to
unintended new requests the client might make.
The tests also assert a bunch of things manually instead of using
expectations when building the mock and calling server.verify().
The meaning of "null" and "undefined" were conflated by Ruma, previously. This updates to the latest Ruma, which changed the `avatar` type from
`Option` to `JsOption` and thus allowed us to distinguish both cases: `null` means the avatar has been unset, `undefined` means it's not changed
since the previous request.
Encryption doesn't usually require access to the user identity, so we can skip loading that from the store.
Unfortunately that means a fair bit of refectoring, in the form of replacing Device with ReadOnlyDevice.
This reduces the time taken to encrypt a message in a large room from about 3 seconds to 1 second.
This is a deprecated flag for element call using livekit. It was used to enable/disable matrix signalling event encryption. This is not relevant for embedded mode at all since there the hosting client is taking care of encryption.
---
* Remove E2EEenabled flag from the rust sdk.
This is a deprecated flag for element call using livekit. It is replaced
by the `password` and the `perParticipantE2EE` flag.
If `perParticipantE2EE` is set to `false`
and there is now password encryption is disabled implicitly.
Signed-off-by: Timo K <toger5@hotmail.de>
* `cargo +nightly fmt`
Signed-off-by: Timo K <toger5@hotmail.de>
* change test based on review
Signed-off-by: Timo K <toger5@hotmail.de>
* Apply suggestions from code review
---------
Signed-off-by: Timo K <toger5@hotmail.de>
Co-authored-by: Benjamin Bouvier <public@benj.me>
Somehow `fmt::pretty` manages to be both overcomplicated and not flexible
enough.
The driver here is that I want to put the event "fields" on a separate line to
the message.
Two sets of improvements to `get_missing_sessions` here:
* Currently, for any users who have an empty device list, we call `KeyQueryManager::wait_if_user_key_query_pending`, which waits up to 5 seconds for a `/keys/query` response. (Arguably, we should be waiting longer: it is not unusual for such a request to take a while.)
The problem is that that user's server may have been blacklisted, in which case we won't even be trying to do `/keys/query` resquests for that user, so we'll be waiting for something that never happens.
To fix this, let's check the failures list when deciding if we should wait for a user's devices.
* Separately, but closely related: for each user thus affected, we do the wait *in series*. This is a bit silly: there is no point waiting 50 times. We can parallelise the work.
Fixes#2793.
---
* Move `get_user_devices` to `IdentityManager`
... since it's going to need to interact with the `FailuresCache` which is
stored there.
* `get_user_devices_for_encryption`: don't wait for failed servers
* `get_missing_sessions`: wait for users in parallel
Rather than waiting for each pending user in series, do all the waits in
parallel.
* Changelog
* Update crates/matrix-sdk-crypto/src/identities/manager.rs
* Address review comments
Includes the following bugfixes:
- Fix the name of the fallback text field for extensible events in
`RoomMessageEventContentWithoutRelation::make_reply_to_raw()`
- Allow to deserialize `(New)ConditionalPushRule` with a missing `conditions`
field
- Fix deserialization of `claim_keys` responses without a `failures` field
It doesn't matter at the moment as the only test using `test_json::MEMBERS`
does not rely on the event being valid, but it shows this error
nonetheless:
```
2023-11-10T08:54:29.920782Z DEBUG receive_members{room_id="!hIMjEx205EXNyjVPCV:localhost"}: matrix_sdk_base::client: Failed to deserialize member event: missing field `room_id` at line 1 column 297 event_id="$151800140517rfvjc:localhost"
```
and https://spec.matrix.org/v1.8/client-server-api/#get_matrixclientv3roomsroomidmembers
says it is a required key.
- Log URI instead of separate homeserver, path
- Always log query parameters (not just for sliding sync)
- Only log request size for requests that could have a body based on the
HTTP verb
Otherwise, it's possible for the `NotificationClient` to be destroyed *after* the ffi `Client`, and then the hack introduced in the previous commit won't work.
A detached task was spawned to react upon session changes, and that task captured a clone of the current `Client`.
This caused a leak of the `Client`, because that task would never get aborted, and would not stop by itself.
The fix here consists in having `Client::set_delegate` return a task handle that needs to be stashed by the FFI
users, and cancelled when the Client gets out of scope. This fixes the leak, by removing the last reference onto
the Client.
Then, when dropping the Client, we have to drop the Stores in it. These stores may be sqlite-based stores, which
make use of deadpool. Deadpool has a sync wrapper that will call `block_on` in a `drop` method, and as such it
requires to be in the scope of a tokio runtime to run properly. To avoid breaking all abstractions and giving
access to the inners of the `Client`, the hack used here to properly be in a runtime when dropping the stores is
to replace the inner sdk `Client` in the FFI `Client::drop` method (and replace it with a dummy client that is
minimally configured and will use in-memory stores).
RoomInfo is often passed around by value, including in futures where
holding types with a large stack size is especially problematic¹.
It might make sense to move the actual data of (Base)RoomInfo into
an inner struct that is always held inside a Box or Arc, but this change
should have most of the benefits of that while being a bit simpler.
¹ https://github.com/rust-lang/rust/issues/69826
* crypto(fix): don't hold the cache lock while waiting on a user key query
Fixes#2802. The lock was only useful to sync the database and the in-memory cache for the users awaiting a key query request.
So it's possible to slightly tweak the API by moving the method from `SyncedKeyQueryManager` to non-synced `KeyQueryManager`, and require a
`StoreCacheGuard` (i.e. the owned lock, so we can manually drop it when we feel like so).
I've looked at all the other methods, and they do require the cache for writing into it and the store.
At the limit we could also move `SyncedKeyQueryManager::users_for_key_query`
into `KeyQueryManager`, but the lock in there is hold for a very short-time, so it shouldn't be an issue.
* Add test for the key query deadlock while waiting for the response.
* Update crates/matrix-sdk-crypto/src/machine.rs
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
`Account::create_outbound_session` no longer takes an entire list of keys;
rather it takes a single key and it is up to the caller to pick a key out of
the list.
This in turn means that `SessionCreationError` loses one of its reason codes.
* Add `perParticipantE2EE` to element call url
* add ffi
* nightly fmt
* refactor to use enum + test
* rename to PerParticipantKeys
* cleanup (spelling + formatting)
Signed-off-by: Timo K <toger5@hotmail.de>
The event_type passed to `encrypt_room_event_raw` must be the one of the cleartext event, not `m.room.encrypted`. The returned event has the expected type.
In the previous situation, running the tests with `cargo test` would sometimes fail because despite appending the number of milliseconds since
the start of epoch to the user names, some user names would clash across different tests, leading to unexpected results. This fixes it by using
an actual RNG in there, so the names don't ever clash.
Instead make futures modules part of the API. It is extremely uncommon
to actually refer to a type that implements `Future` by its name / path,
so it's better if these don't show up in documentation pages that are
already large and somewhat cluttered.
There was a bug previously, that the cross-signing bootstrapping could include a signature for a device key that hadn't been uploaded yet.
This fixes it by returning a third (!) request in `bootstrap_cross_signing`, that must be sent as the second in order, and will upload
keys, if required.
Also tweaks documentation. Fixes#2749.
The store cache can be filled lazily when it's actually needed. It's not needed in this
function here, and may be needed in another function the caller of this function may call
later. No need to preemptively fill the cache here.
These functions are a bit confusing, so I wrote some comments. I also factored out
`get_user_signing_key_from_response`, to reduce duplication and (IMHO) improve clarity.
Hashes computed by `compute_session_hash` may be stored in the crypto store, and may or may not
be encrypted; make them slightly more secure by using a more secure hash. Also use sha2 for
the logged hashes that are useful for debugging OIDC issues.
… allows the FFI type MessageLikeEventContent to hold custom msgtypes,
which is useful for notification.
Also allows sending custom msgtypes, as long as no fields other than the
msgtype itself and body are needed.
This PR adds a ffi binding for sending voice messages in the legacy format (before extensible events). It also makes minor changes in the `matrix-sdk` crate to accomodate this change.
* Add send_voice_message api
* Update ruma-events dependency
* Fix attachment info mapping
* Remove unstable dependency in attachment.rs
* Bump ruma-events
* Fix matrix-sdk Cargo.toml
* Fix formatting issues
* Refactor Voice case
* Remove clone from AttachmentInfo
* Remove duplicate code
* Remove unused imports
* Fix formatting issue
* Rename update function
This includes the requests that are sent to the matrix client (i.e.
a request to send a matrix event) as well as requests that are sent to
the widget.
The difference between this and `Action` is that `Action` is rather
"low-level" (i.e. it has a generic `SendToWidget(String)` variant),
whereas this API is a high-level API that we will use internally to
conveniently issue and `await` for outgoing responses.
This code path could never be taken:
- the only way to set the cross_process_token_refresh_manager is from calling the same function
- this function is guarded against a lock, so it's not reentrant
- this function's only internally called, during `restore_session` or after a successful login:
both will set a session, hence those functions would fail beforehands if another session had
been set earlier.
#2587 introduced a breaking change in the serialized format of `RoomInfo`, changing the `event`
field format in particular, without a migration. As a result, this lead to users state stores
being invalidated for containing incorrect data, and thus being logged out.
This mitigates the issue by making sure that the LatestEvent type is either deserialized as its
newer format (aka `SerializedLatestEvent`) or its older format (aka `SyncTimelineEvent`). This
way, we maintain compatibility with the previous format. We always serialize to the new one, so
we'd only run this migration once.
- `content` is now of type `RoomMessageEventContentWithoutRelation`,
the relation was previously always overwritten if set
- `add_mentions` is now inferred from `content.mentions`
This adds a new `StoreTransaction` type, that wraps a `StoreCache` and a `Store`. The idea is that it will allow write access to the `Store` (and maintains the cache at the same time), while the `Store::cache` method will only give read-only access to the store's content.
Another new data type is introduced, `PendingChanges`, that reflects `Changes` but for fields that are properly maintained in the `StoreCache` and that one can write in the `StoreTransaction`. In the future, it wouldn't be possible to use `save_pending_changes` from the outside of a `StoreTransaction` context.
The layering is the following:
- `Store` wraps the `DynCryptoStore`, contains a reference to a `StoreCache`.
- When read-only access is sufficient, one can get a handle to the cache with `Store::cache()`.
- When a write happens, then one can create a `StoreTransaction` (later, only one at a time will be allowed, by putting the `StoreCache` behind a `RwLock`; this has been deferred to not make the PR grow too much).
- Any field in the `StoreCache` will get a method to get a reference to the cached thing: it will either load from the DB if not cached, or return the previously cached value.
- Any field that can be written to will get a method to get a mutable reference in the `StoreTransaction`: it will either load from the cache into a `PendingChanges` scratch pad, or return the scratchpad temporary value.
- When a `StoreTransaction::commit()` happens, fields are backpropagated into the DB *and* the cache.
Then, this `StoreTransaction` is used to update a `ReadOnlyAccount` in multiple places (and usage of `ReadOnlyAccount` is minimized so as not to require a transaction or cache function call as much as possible). With this, the read-only account only exists transiently, and it's only stored long-term in the cache.
Followup PRs include:
- making the `ReadOnlyAccount` not cloneable
- remove inner mutability from the `ReadOnlyAccount`
- add a `RwLock` on the `StoreTransaction`
Part of https://github.com/matrix-org/matrix-rust-sdk/issues/2624 + https://github.com/matrix-org/matrix-rust-sdk/issues/2000.
---
* crypto: Replace some uses of `ReadOnlyAccount` with `StaticAccountData` and identify tests
* crypto: introduce `StoreTransaction` to modify a `ReadOnlyAccount`
* crypto: introduce `save_pending_changes`, aka `save_changes` v2
* crypto: Start using `StoreTransaction`s to save the account, get rid of `Store::save_account` + `Account::save`
* crypto: use `StoreTransaction` to save an account in `keys_for_upload`
* crypto: use `StoreTransaction` and the cache in more places
* crypto: remove `Account` from the `Changes` \o/
* crypto: remove last (test-only) callers of `Store::account()`
* crypto: move `ReadOnlyAccount` inside the cache only
* crypto: use `ReadOnlyAccount` and `Account` in fewer places
whenever we can use `StaticAccountData` in place.
* crypto: make tests rely less on OlmMachine
* crypto: Don't put the `ReadOnlyAccount` behind a RwLock just yet
+ clippy
Stores may race when performing writes, since in `save_changes` some data is pickled, and live across await points (it could be stale after an
await point). To prevent multiple threads racing when calling into `save_changes`, a new lock is introduced there.
* support creating emotes
adds an emote param to message_event_content_from_markdown and
message_event_content_from_html so that clients can create
m.room.message events with msgtype emote (on the assumption
that the msgtype returned in the RoomMessageEventContentWithoutRelation
is immutable, and so can't be overridden by the client)
* lint
* switch to separate methods per review feedback
This patch adds a `sender_profile: Option<MinimalRoomMemberEvent>`
field. Thus, `cache_latest_events` now receives an
`Option<&StateChanges>` and an `Option<&Store>`. The `StateChanges`
is used to read the most recent sender profile, otherwise the sender
profile will be fetch from the storage.
This patch adds a new `LatestEvent` type. It wraps the
`SyncTimelineEvent` that was used before. The idea is to add more fields
onto this `LatestEvent` struct, but this patch starts easy by using
`LatestEvent` everywhere where it's required.
This patch updates `all_rooms` to require the `m.room.member` state set
to `$LAZY`. That way, it's more likely to get the `m.room.member` event
associated to the latest event without needing to sync all the members.
This is to enable entirely local stacks of Element X to work correctly. It's mostly seamless (no ffi changes) because it ties into `ClientBuilder.insecure_server_name_no_tls()`.
---------
Co-authored-by: Benjamin Bouvier <public@benj.me>
Before this commit, a call to get_notification would fail if decrypting failed on the first attempt, because `Room::decrypt_event` will hard-fail if the key wasn't found.
Sorry, this commit is a bit big. What happened is that I've pulled a thread: put a `StaticAccountData` there, look at caller; it seems to use only a
static account too, so keep up.
The only contender was the `OwnUserIdentity` data structure which really wants to sign things. Lucky for us, we could pass it a `ReadOnlyAccount`
from a store, in those cases, since we always had a `Store` hanging around; in the future it'll read it from the store cache, which is somewhat
identical.
We might need to rely on the data in the DB to already be corrected
before using it for another migration
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
- Bumps ruma to include https://github.com/ruma/ruma/pull/1665
- Enables ruma `compat-arbitrary-length-ids` flag instead of using the now deprecated `lax-id-validation`
Using a static lock would prevent two Client instances to call the
method at the same time.
We're going to assume that people won't have multiple Client instances
for the same user, in which case a static lock would have been helpful.
Co-authored-by: Jonas Platte <jplatte@matrix.org>
Instead of using three separate object stores, use a single one with some structured objects and indexes.
Fixes#2605 (or at least, should make whatever is going wrong much more obvious).
Consists of a series of commits which should be reviewable on their own.
Implement a `tracing_subsciber` `Layer` which sends output to the javascript
console.
Obviously, this only works on the wasm32 target.
This is lifted from `matrix-rust-sdk-crypto-wasm`, so that we can use it in
tests etc for other crates.
Fix a bug which caused `get_outgoing_secret_requests` to return no data.
This is exposed as a public function, and when called that way it does *not*
expect the key to be escaped and encrypted.
An encryption state request was failing, but before the patch, subsequent
requests would end up in success, while this wasn't the case. The new failure
introduced by this patch was a real one, because the encryption state was
not mocked as part of this test (which tries to send a message to a homeserver).
Ruma used to handle this during deserialization when the field was
an Option.
Especially meaningful when computing the room name, where
an empty string means the name is not set.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Adds a dummy fuield to `UnstableVoiceContent`, otherwise uniffi
will generate an empty Kotlin data class which breaks compilation
(Kotlin data classes must have at least 1 field).
Upstream bug: https://github.com/mozilla/uniffi-rs/issues/1760
Without this feature:
```sh
$ cargo check -p matrix-sdk --features experimental-sliding-sync
Checking matrix-sdk-base v0.6.1 (/Users/hwhost/Development/Element/matrix-rust-sdk/crates/matrix-sdk-base)
error[E0433]: failed to resolve: could not find `poll` in `events`
--> crates/matrix-sdk-base/src/latest_event.rs:7:5
|
7 | poll::unstable_start::SyncUnstablePollStartEvent, room::message::SyncRoomMessageEvent,
| ^^^^ could not find `poll` in `events`
error[E0599]: no variant or associated item named `UnstablePollStart` found for enum `AnySyncMessageLikeEvent` in the current scope
--> crates/matrix-sdk-base/src/latest_event.rs:40:68
|
40 | AnySyncTimelineEvent::MessageLike(AnySyncMessageLikeEvent::UnstablePollStart(poll)) => {
| ^^^^^^^^^^^^^^^^^ variant or associated item not found in `AnySyncMessageLikeEvent`
```
With the feature, everything goes well.
This patch removes `SlidingSyncList::room_list_filtered_stream.
Of course, the only place where it was used,
`RoomList::entries_with_dynamic_adapters` now use `.filter` from
`VectorSubscriberExt`. It's basically less code.
This patch adds `Option<ImageInfo>` as an argument of
`Room::upload_avatar`.
To achieve that, this patch implements
`TryFrom<ImageInfo> for ruma::events::room::avatar::ImageInfo`.
The `TimelineError` type only contains error variants used for
`ImageInfo`, `AudioInfo`, `VideoInfo`, `FileInfo` and so on. It's never
used for something strictly related to the `Timeline`.
This patch then renames `TimelineError` to `MediaInfoError`.
The `MissingMediaInfoField` variant becomes `MediaField`. The
`InvalidMediaInfoField` becomes `InvalidField`.
Use `SharedObservable::set_if_not_eq` instead of `::set`, so that it
doesn't update the limit to the same value, which would be unnecesary in
this case.
This patch removes the `Mutex` around `Subscriber<Option<u32>>` inside
the `RoomListDynamicEntriesController`. The `Mutex` was necessary to
get a mutable reference, so that `Subscriber::next_now` could have been
used. However, it's not necessary to use `new_now` in this particular
context. We can use `get` instead, which take an immutable reference,
thus removing the need for the `Mutex`.
A `Mutex` has a non-negligeable cost. This function can be used in a
critical hot path, and must as fast as possible.
This patch updates `chunk_large_query_over`. This function works great
when it's required to chunk some data over a query. However, if the
data don't need to be chunked, it is possible to get a quicker path that
doesn't involve 2 `Vec` allocations, nor a `split_off` of one `Vec`, nor
a `Vec::extend` (which move data in memory). The quicker path, which is
also the most likely to be hit, simply does a single `Vec` allocation,
and that's it.
* chore(oidc): put impl of OIDC server behind a trait and add tests for the OIDC flow
chore(oidc): add first tests for login/AuthorizationResponse
chore(oidc): add tests for finish_authorization/finish_login/refresh_access_token
chore(oidc): add test for logout
chore(🤷): clippy/fmt
* chore(oidc): move account management test to the new `tests` module
* chore(oidc): address review comments
Prior to this patch, we were using 2 constants to define the
sync indicator delays: `SYNC_INDICATOR_DELAY_BEFORE_SHOWING` and
`SYNC_INDICATOR_DELAY_BEFORE_HIDING`. After some discussions with
some users, it appears that it's desirable to make these values
parameterizable.
Thus, this patch updates `RoomListService::sync_indicator` to accept 2
parameters: `delay_before_showing` and `delay_before_hiding`. The patch
also updates the FFI bindings.
In `RoomList::entries_with_dynamic_adapters`, we were using the
`Limit::dynamic` constructor. It builds a `Limit` stream adapter where
the limits come from a `Stream`. The immediate impact is that this
constructor does only return a new `Stream`, but it cannot return
the initial items of the vector. Thus, it wasn't possible to reset
the final `Stream` with a `VectorDiff::Reset { values }`. Instead, we
were emitting a `Clear` (from this code) then a `Append` (from the
`Limit` stream). Sadly, for some client apps, like Element X iOS, these
2 diffs (`Clear` and `Append`) result in a “rendering blink”.
Hopefully for us, now there is new constructors for `Limit`! One of them
is `Limit::dynamic_with_initial_limit`, which returns the initial items
along with the new `Stream`. That's perfect, we can reset the final
`Stream` with `Reset { values }` again, thus removing the “UI blinking”
effect in Element X iOS.
This patch switches `Limit::dynamic` to
`Limit::dynamic_with_initial_limit`, and updates/simplifies the tests
accordingly.
`RoomListService::new_internal` were creating the `invites` sliding
sync list, with a cache. It is not a good idea. It should not be cached.
Let's remove that :-).
`RoomListLoadingState` was initialized with the `NotLoaded` state. This
is always true… except when there is a cache! If the cache contains
N rooms, the loading state should be `Loaded { … }`. Then the next
sync (if any) will update the loading state to `Loaded { … }` with
an adjusted number of rooms or something like that, but semantically
speaking, the presence of the cache should result in a `Loaded` state.
This patch fixes this behavior, and also adds the tests.
For some room lists, the number of entries can be gigantic. For example,
some accounts have 800, 2500, or even 4000 rooms! It's not necessary for
the client/app to display all the 4000 rooms. First, it can create some
performance issues, second, nobody will scroll 4000 rooms to search for
a particular room :-). Such users are more likely to use a search bar or
something equivalent. The idea is that `RoomListService` will continue
to sync all the data, but only a _limited_ version of it will be shared
to the client/app.
This patch takes `RoomList::entries_with_dynamic_filter`, and improves
it to include this (dynamic) limit.
This patch renames `RoomList::entries_with_dynamic_filter`
to `::entries_with_dynamic_adapters`. It now returns a
`RoomListDynamicEntriesController`, which is a renaming of
`DynamicRoomListFilter`. Basically, the “dynamic filter” becomes a
“dynamic controller” because `RoomList::entries_with_dynamic_adapters`
manages more than a filter. It now uses
`eyeball_im_util::vector::DynamicLimit` to dynamically limit the size
of entries. And that's the major idea behind this patch.
`RoomListDynamicEntriesController::set` is renamed `::set_filter`, and 2
new methods are introduced: `add_one_page` and `reset_to_one_page`.
A _page_ is like a chunk of room entries we want to view or add. When
doing `next_page`, the limit increases to `old_limit + page_size`. The
`reset_pages` method resets the `limit` to `page_size` only.
This adds a cross-process lock for refresh to work correctly.
We want to coordinate token refresh across multiple processes. For that, we're using a cross-process lock, and a value in the database identifying the latest session tokens that are valid (a hash of the actual tokens, for security reasons).
Whenever we run into an HTTP error indicating that the tokens have been invalidated, we try to refresh the access tokens; that's already existing prior to this PR. The novelty introduced is that we take a cross-process lock before doing so, now. Taking this lock will also load a session hash from the database, and we'll compare it against the latest "known" session hash (that the current process saved into its memory).
If there's no mismatch (i.e. the database and the currently known are the same), then we're all good and can keep going with the refresh, synchronize the hashes everywhere (in-memory and database), make sure the client is notified about it (through a new user-provided callback `SaveSessionCallback`; on iOS this will save it into the device's keychain).
Otherwise, that means another process has done a refresh under our feet. In that case, we ask an authoritative source for trusted session tokens. On iOS, they're reloaded from the device keychain; that happens through a new user-provided callback `ReloadSessionCallback`. Then, we make sure that the DB and the in-memory value recall this latest value.
An embedder who would like to make use of the cross-process locking mechanism should call `client.oidc().set_session_callbacks` and `client.oidc().enable_cross_process_refresh`. If only interested with the pings for new sessions, the client may only call `client.oidc().set_session_callbacks`.
Fixes https://github.com/matrix-org/matrix-rust-sdk/issues/2418.
Fixes https://github.com/matrix-org/matrix-rust-sdk/issues/2476
## Future improvements
- More testing of the whole flow. Not sure if mocking will be quite fit for OIDC, as this may require setting up an HTTPS server for the authentication code exchange and other OIDC-specific flows.
- Get rid of `SessionChange`, which duplicates in some way how a client can be notified about session changes.
---
* chore: replace manual StateMemoryStore::new with derived Default
* feat: add store backing for cross-process locking in state store
* chore: rename CryptoStoreLock to CrossProcessStoreLock
* chore: generalize cross-process lock
* feat: move the cross-process locking mechanism to the main crate
* feat: add support for cross-process store lock in the state store 🥳
* feat: implement a cross-process lock for OIDC token refresh
* chore: tweak comment + function name
* feat: make restore_session safe wrt cross-process lock
* feat: add FFI method + add mechanism to reload from keychain
* fix rename
* feat: return early when there was another process refreshed tokens
* fix FFI compile error + tweak some comments
* fix: put the reload_session callback and cross-process locks behind Arc to share them across clients
* feat: Add session retrieval to FFI.
* HACKY; KIDS DON'T DO THIS AT HOME
* chore: log if the hash from db isn't the same from the one from the returned session
* make it simpler to test OIDC token refresh
* some work, that includes fixes and a first test
* feat: require that the reload_session_callback be set at the same time as the cross-process lock
* chore: traces, traces everywhere
* fix: inherit session_change_sender when creating the notification client
* Some FFI improvements to help with tokio problems
* feat: resilient mode when DB/callback disagree about session (callback wins!)
* chore: move sender.send to the finish_refreshing function
* feat: add a save_session callback in the FFI and use it to save the session in keychain while holding XP lock
* fix test expectation after adding the check 🤷
* feat: split the ClientDelegate into two parts, including brand new ClientSessionDelegate
* chore: get rid of lease lock impl in the state store, as it's now unused
* a mix of fmt + clippy
* feat: add ctor for the crossprocessrefreshlockctx
* Include user ID when retrieving session.
Necessary as this isn't known when creating the AuthenticationService.
* yo dawg, you can't block while you block
* share auth data between parent and child client, add lock, AAAAAA this is messy
* tweaks
* feat: make the cross-process store locks generic
And move the implementation to the common crate.
* chore: upgrade some code comments to doc comments in `OngoingMigration`
* feat: implement `CryptoStore::remove_custom_value`
As it's going to be used for the OIDC PR, so as to remove a remembered hash of session tokens.
* remove unneeded remnants
* correctly wait for current request to finish
* feat: make it possible to setup session delegates on android too(?)
* put the cross process stuff in its own file
* typos 🤷
* fix: detach before sending token refresh request, to make sure the response tokens are always properly saved
* kleepee
* First round of review, thanks jonas!
* review round 2. FIGHT
* remove useless logs + avoid using deref explicitly
* more specialized error when cross-process lock is enabled without session callbacks
* fix: avoid cyclic reference between the session callback and client
---------
Co-authored-by: Doug <douglase@element.io>
Co-authored-by: Jonas Platte <jplatte@matrix.org>
When receiving a sliding sync room response for a room that had no local events in its timeline cache,
we'd mark the room as limited before, which is incorrect. It was made worse by the fact that later in
the code, we'd clear the local cache if a room was marked as limited, so this is a problem that would
repeat itself over time (assuming empty responses for that room).
This fixes it and unifies logs so there's only one log line per room, at most.
Fixes https://github.com/vector-im/element-x-android/issues/1281
Fixes https://github.com/matrix-org/matrix-rust-sdk/issues/2540
Currently when the AuthenticationService is given updated metadata, it is ignored if a dynamic registration has already been made for a selected issuer. This PR fixes that by storing the metadata's hash and resetting the store when there is a mis-match.
Additionally it moves OidcRegistrations out of the FFI into a new authentication module in the UI crate and adds some tests.
This patch updates `Room::members` to return
`Result<Arc<RoomMembersIterator>, ClientError>`. This
`RoomMembersIterator` type is new, and is implemented in this patch too.
The idea behind this patch is to allow the bindings to “paginate” over
the list of members for a particular room, in case the room has 17k
members for example.
This patch changes the backoff strategy from `fixed` to `exponential`
when a flaky test is retried. The `count` value is also updated to
3. Finally, we try to avoid the thundering herd problems with `jitter
= true`.
This patch adds the `RoomListService::sync_indicator` method, along
with the `RoomListServiceSyncIndicatorListener` callback interface, and
`RoomListServiceSyncIndicator` enum.
This patch implements a new method: `RoomListService::sync_indicator`.
It returns a `impl Stream<Item = SyncIndicator>` where `SyncIndicator`
is a new enum with 2 variants: `Show` and `Hide`.
`SyncIndicator` is the UI equivalent of a sync spinner/loader/toaster,
that the app might want to show to the user to indicate when a _first_
request is sent and might be slow, i.e. the first response is taking
a little bit of time to come. The term _first_ may be innapropriate as
it covers the actual first sync request, but also the recovering sync
request. It means that when a sync error happened, the sync indicator
will be shown too, which is a pretty useful information for the user.
It's not because a `SyncIndicator` should be shown that it must be send
immediately onto the `Stream`. In case of a normal network conditions,
without any delay, it can lead to a “blinking” visual effect. Some
constants configure how long it takes to consider that a request is
“slow”, and that the `SyncIndicator` is necessary to be shown (or
hidden).
Since the verification status of an event can change, we need to be able to
refetch the verification status without doing the whole decryption dance.
Hence, we expose a new `get_room_event_encryption_info` method.
This was previously accepted by the compiler but is being phased out;
it will become a hard error in a future release! See https://github.com/
rust-lang/rust/issues/115010.
Now that we can decrypt on both of {single|multi} process setups (i.e. available for both android
and ios), we can enable retrying decrypting notifications by default.
Also rejigger the parameters passed to the notification client builder, so that it's always required to pass
a process setup. With that, we're one step closer to removing the retry_decryption() function and enable it
by default.
The `Timeline` batches its updates to its subscribers (e.g. a client app,
like Element X). A batch is built every time the inner state lock of
the `Timeline` is released. On the paper, it's nice; in practise, even
a read operation on the `Timeline` leads to building a new batch. This
is inefficient and consumes resources (like CPU cycles, FFI boundary
crossings, memory allocations etc.) even if there is no update to put in
the batch: this will just batch empty updates.
To avoid that, one needs to make the difference between read or write
operations onto the inner state. Only the write operations will fire a
batch.
This patch splits the `TimelineInnerStateLock::lock` method into
`::read` and `::write`. The idea is that a read-only lock doesn't
hold a clone of the lock release observer (`lock_release_ob:
SharedObservable<()>`), it will not notify the observer. Then it's only
the write lock that holds a clone of the lock release observer, and will
notify it.
This patch updates the code accordingly as best as possible.
This patch updates `RoomListService` to install the `invites` sliding
sync list from the start, along with `all_rooms`. Prior to the patch,
`invites` was installed after the first sync.
`invites` is installed in selective sync-mode with a range of `0..=0`.
After the next sync, `invites` switched its sync-mode to growing with a
batch size of `0..=19`.
This patch takes inspiration of
https://github.com/matrix-org/matrix-rust-sdk/pull/2480.
This patch adds a new `Recovering` state, which is a “transition”
between `Error` and `Terminated` to `Running`. When moving from `Error`
or `Terminated` to `Recovering`, `all_rooms`' sync-mode is set to
`Selective` with its initial range. Then when moving from `Recovering`
to `Running`, `all_rooms`' sync-mode is set to `Growing` with its
initial range again. For the `invites` list, it is reset to its initial
range when moving to `Recovering` too.
`Error` and `Terminated` now act the same.
This patch updates the `batch_size` of `all_rooms` once in growing
sync-mode to 100. It was previously increased from 50 to 200 in
12a1f25ec3. In some situations, it
generates very large payloads that can take time to be delivered to the
user with a slow network.
This patch tries to fine-tuned the `batch_size` value.
This patch reverts b2cc279279. On the
paper, it was a good idea. In practise, it has revealed a bug on the
server side. The only solution to mitigate this bug for now is to revert
the `timeline_limit` to 1.
Prior to this patch, we have increased the
`SlidingSyncListInner::room_list` capacity, to avoid trigger
`VectorDiff::Reset` as much as possible. Now that Element X is optimised
to handle larger diff, we can discrease this number, yeeeee :-).
When a user has hundreds of rooms, `RoomListService` will fetch at worst
one event per room, which can generate large responses. Let's use a
`timeline_limit=0`.
This patch first off renames `BaseClient::ignore_user_list_changes_tx`
to `::ignore_user_list_changes`.
This patch then changes the type from `Arc<SharedObservable<_>>` to
simply `SharedObservable` as this type implement `Clone`. We basically
have a double-`Arc` here.
Finally, this patch adds documentation for this field.
… by swapping the branches around. Previously, invocations meant to
match the found + not_found branch were actually hitting the other one
prior to Rust 1.71, likely due to parser supporting type ascription
(even though it was an unstable feature).
Inside `RoomListService`, the `State` enum handles the transition from
one state to another. In case of some `State::Error`, the `all_rooms`
sliding sync was refreshed, i.e. its sync-mode was reset to its initial
value.
This patch also refreshes the `invites` sliding sync list! It adds
the `ResetInvitesListGrowingSyncMode` action, and attaches it to the
`refresh_lists` actions.
New constants are added to represent default `batch_size` values for the
growing sync-mode for various sliding sync list. It could be helpful for
further maintenance.
This patch finally adds and updates the tests accordingly.
This patch changes the `batch_size` of the sliding sync list `invites`
for `RoomListService`. Previous value was 100, new value is 20.
For accounts that have a large number of invites, it won't slow the
rendering of `visible_rooms`.
Usually, when the sliding sync session expires, it leads the state to
be `Error`, thus some actions (like refreshing the lists) are executed.
However, if the sync-loop has been stopped manually, the state is
`Terminated`, and when the session is forced to expire, the state
remains `Terminated`, thus the actions aren't executed as expected.
Consequently, this patch updates the state to `Error` manually.
- Use OIDC for logout when appropriate.
- Allow server's that support OIDC but not passwords to work.
- Only sign out users if token refresh is explicitly refused.
- Expose the OIDC account URL.
- Support for RP initiated logout.
In the previous situation, we had two locks with similar responsibilities, the `response_handling_lock`
and the `position` lock. The latter *almost* covered the former's critical zone, albeit for a single
function call, which left room for a deadlock situation (latter taken, then former, then latter).
This removes the former, and extends the critical zone of the latter up to the end of the response handling,
removing the possibility of the deadlock entirely.
This reverts commit 9b811009e1.
We have realized that the server might not handle timeouts as expected.
Thus, it's hard to know the difference between network timeout and poll
timeout (resp. the server is unreachable vs. the server has nothing to
respond with). We will come back on this later.
As a sequel of the previous commit (d4cbcd397d), this patch updates
`SlidingSync` to “ignore” timeouts, i.e. timeouts aren't reported to the
caller, and aren't stopping the sync-loop.
All errors inside `SlidingSync` are stopping the sync-loop, and errors
are returned to the caller. However, in some situation, some errors
should be ignored, i.e. they should not stop the sync-loop and they
should not be returned to the caller: the sync-loop just continues to
run. This patch does that for `Error::ResponseAlreadyReceived`. More
errors will come.
Why is it annoying? When `matrix_sdk_ui::SyncService` sees an error,
it stops all the sync-loops (`RoomListService`, `EncryptionSync`…) and
restarts them properly. In the case of `Error::ResponseAlreadyReceived`,
this is a waste of time and resources. This error is an error from the
`SlidingSync` point of view, but _not_ from the caller point of view.
This patch updates the `BaseClient::process_sliding_sync` method to put
the `LeftRoom` in the `SyncResponse::leave` list.
To achieve so, this patch updates
`BaseClient::process_sliding_sync_room` to returns `Option<JoinedRoom>`,
`Option<LeftRoom>` and `Option<InvitedRoom>` (previously, it was only
`JoinedRoom` and `Option<InvitedRoom>`).
SlidingSync emits state events inside `required_state`, but _also_
sometimes inside `timeline`! This patch looks for state events inside
both entries.
The rest of the patch is composed of tests.
Imagine the following scenario:
A request $R_1$ is sent. A response $S_1$ is received and is being
handled. In the meantime, the sync-loop is instructed to skip over any
remaining work in its iteration and to jump to the next iteration. As a
consequence, $S_1$ is detached, but continues to run. In the meantime, a
new request $R_2$ starts. Since $S_1$ has _not_ finished to be handled,
the `pos` isn't updated yet, and $R_2$ starts with the _same_ `pos`
as $R_1$.
The impacts are the following:
1. Since the `pos` is the same, even if some parameters are different,
the server will reply with the same response. It's a waste of time
and resources (incl. network).
2. Receiving the same response could have corrupt the state. It has been
fixed in https://github.com/matrix-org/matrix-rust-sdk/pull/2395
though.
Point 2 has been addressed, but point 1 remains to be addresed. This
patch fixes point 1.
How? It changes the `RwLock` around `SlidingSyncInner::position` to
a `Mutex`. An `OwnedMutexGuard` is fetched by locking the mutex when
the request is generated (i.e. when `pos` is read to be put in the new
request). This `OwnedMutexGuard` is kept during the entire lifetime
of the request extend to the response handling. It is dropped/released
when the response has been fully handled, or if any error happens along
the process.
It means that it's impossible for a new request to be generated and to
be sent if a request and response is running. It solves point 1 in case
of successful response, otherwise the `pos` isn't updated because of
an error.
This patch first off renames `Client::deserialize_events` as
`Client::deserialize_state_events`. Then, this patch initially updated
the return type of this method from `Vec<Option<_>>` to `Vec<_>` to
filter out events that failed to deserialize. The patch ultimately
updates the return type of this method to `Vec<(Raw<_>, _)>`, so that
the raw state events that map to the state event are collected too (this
is required for making `Client::handle_state` to work correctly if an
event failed to deserialized). Finally, the rest of the patch updates
the code accordingly.
`matrix_sdk_ui::room_list_service::Error` has a `SlidingSync` variant to
report errors from `matrix_sdk::sliding_sync::Error` (to be more exact,
from `matrix_sdk::Error::SlidingSync`).
This patch updates the error message from `SlidingSync failed` to
`SlidingSync failed: <explanation>`, where `<explanation>` is the
`Display` representation of the inner error.
It will help to provide more details to the end-user when looking at
logs.
The Room::request_encryption_state method employs a DashMap to uphold a
lock, facilitating the de-duplication of requests directed to the server.
The de-duplication logic involves creating a fresh Mutex and embedding
it into the DashMap through these stages:
1. Generate a new de-duplication mutex.
2. Insert a mutex copy into the DashMap.
2. Acquire the mutex.
Due to DashMap's limitation of enabling map locking solely during
insertion (step 1), a race condition emerges. Consequently, multiple
invocations of the Room::request_encryption_state method might
concurrently endeavor to insert the de-duplication mutex.
This commit removes the chance of such a race. It substitutes the
DashMap with a combination of a mutex and a BTreeMap. This adaptation
permits us to lock the map throughout all three specified operations.
This commit addresses a bug in the Room::is_encrypted functionality.
The issue stems from the Room::request_encryption_state method, which
Room::is_encrypted relies on. This method incorporates a de-duplication
mechanism to ensure that only a single request for the m.room.encryption
state event is made to the server.
Due to this de-duplication mechanism, Room::request_encryption_state
follows two code paths. One path receives the state event from the
server, while the other path merely waits for the first path to complete.
The primary goal of the Room::request_encryption_state method is to
furnish the requested state event. However, because the second code path
doesn't receive any content, it returns an Option.
The problem arises when Room::is_encrypted evaluates this Option. It
erroneously determines that if the Option is None, encryption remains
inactive for the room.
To rectify this, the commit proposes that Room::is_encrypted analyze the
information stored in the in-memory RoomInfo, which is maintained by the
Room::request_encryption_state method. This approach mirrors the existing
behavior of Room::is_encrypted when it processes the code path where the
m.room.encryption state event has already been retrieved.
## Feature
There are now three tasks running in the sync service:
- the room list sliding sync task
- the encryption sync task
- a new scheduler task, that listens to messages sent by both of the previous tasks, and will take care of cleaning up tasks, stopping the services, and setting states after it received a message.
When any of the sliding sync fails, it sends a message to the scheduler task, indicating why it stopped (error or not, was it an expired session or not). Then the scheduler task will do any necessary cleanup.
`stop()` (née `pause()`) can now make use of that scheduler task as well, by sending a report requesting to stop both tasks and services. Responsibilities are now cleanly split: in particular, I like it much better that the room list task doesn't have to stop the encryption sync itself, since it's not really its duty. (And this avoids lots of code duplication to cleanup tasks and stop services.)
## Testing
This also tests the `SyncService` states. Unfortunately it's not perfectly deterministic in two places:
- we can't predict how many requests will be sent to the server (although, with the mocking server responding in 50ms, and considering a waiting time of e.g. 300ms, it should send at least two requests).
- the test `pause()` and re`start()` the sync service at some point, and then we can't guess what the next `pos` will be, in any of the syncs: it could either be the previous value (if we aborted while processing the previous response), or the next value (meaning we could finish processing the sliding sync response).
Still, it confirms at least that pausing and resuming work as expected.
---
Fixes#2382
The timeouts were a bit too agressive, according to some user logs, resulting in failing to
load the event mentioned in the notification.
Here's the rationale for the new timeouts: we're only limited by the iOS process which has *at
most* 30 seconds to process a notification.
- We're running at most 3 requests of the notification sliding sync, so that will be (1+3)*3 = 12
seconds allocated for that.
- If we've found an event but it required decryption, we're running the encryption sync up to
2 times, (3+4) seconds each => 14 seconds.
At most we're eating up 26 seconds of the entire time, leaving some ballast for the rest of
the program.
This patch optimises the previous patch by simplifying the use of
`Timeline`. If the `Timeline` exists, let's return the `latest_event`
every time: whether it is a remote _or_ a local event.
This patch updates `Room::latest_event` to return the `Timeline` local
event if any.
First off, it starts by checking if a `Timeline` exists. It won't create
it if it doesn't exist! Second, it checks whether a latest event exists.
Finally it checks whether it's a local event.
This patch also updates the tests accordingly.
A Sliding Sync `v4::Response` contains a `pos` value. It helps to identify
progress during several requests/responses. If two requests with the
same `pos` are sent, the server **must** reply with the same response,
as defined in the specification.
The corollary is: If a response contains a `pos` that has already
been received, the client **must** ignore it, otherwise it can create
duplications. For example, let a response containing sync operations,
like `DELETE` or `INSERT`, with indexes: if this pair of operations are
repeated twice with the same index, it creates a broken state.
To avoid this behaviour, this patch stores the last 20 position markers
received from the server. If a response contains a `pos` that has
already been received, a (new) error is returned. As with all the other
errors, the sync-loop is stopped, and must restarted. The past positions
survive to this restart, as for the rest of the state.
When the server replies with a `M_UNKNOWN_POS`, the `pos` and the
`past_positions` are all cleared.
This patch removes
`RoomListEntriesDynamicFilter::set_with_fuzzy_match_pattern` and
replaces it by a single `::set` method. It takes a new enum as
parameter: `RoomListEntriesDynamicFilterKind`. This enum has a
`FuzzyMatchRoomName` variant to simulate the removed method. It also
adds the `All` variant, to represent the `all` filter.
This splits the method out so the bigger chunks doesn't persist the
changes in the store.
This will be useful if we need to hijack the changes and persist them in
a different store.
This patch tries to fix https://github.com/vector-im/element-x-
ios/issues/1204 by adding the following required states in the
`visible_rooms` sliding sync list: `m.room.member: $LAZY`.
* feat(ffi): add an optional file logger
Also makes logging to stdout/logcat optional.
* WIP: stupidly duplicate code 🤷
* ffi: Get rid of duplication in tracing initialization
* feat: add OtlpTracingConfiguration too
* feat: log either to stdout on non-android or logcat on android
---------
Co-authored-by: Jonas Platte <jplatte@matrix.org>
If we were invited and joined a room, then the computation of is_direct() was happening after
joining, and depended on the presence of a global account event of type "m.direct". This event
is added by the "set_is_direct()" call thereafter, so this couldn't ever happen.
This fixes it by computing the is_direct field *before* joining the room, and then marking the
room as a DM based on that.
The full timeline event contains both the timestamp (requested in #2361) and the sender id;
now the sender id for the invite can also be contained in there, minimizing the amount of copying we're doing in the
FFI code.
Fixes#2361.
There must be at most one sliding sync notification per connection id. While we could use multiple ids, it seems bad
to do so, in terms of proxy performance; so this makes it so that there is at most once notification being handled
at a time.
- include the conn_id at the stream level, not the sync_once level (that's a child scope of the stream)\
- also include whether e2ee is enabled at this level
- shorten most static logs
- remove duplicate "sending request" logs
Even if the two issues are likely caused by the same root cause (the user not being
authenticated), they should be used in different contexts, in my understanding:
- the authentication required error should happen only during HTTP requests
- the absence of olm machine should be signalled when we try to get one and it's not been initialized yet
This has caused a bit of confusion when looking at debug traces today, so this patches fixes it.
This patch updates `RoomListItem::avatar_url` to use
`matrix_sdk_ui::room_list_service::Room::avatar_url` instead of
`matrix_sdk::Room::avatar_url`.
This patch also moves `avatar_url` before `is_direct` (so that it's the
same order as other places in the code).
This patch implements `Room::avatar_url`. It tries to calculate
the best avatar URL as much as possible. It's either the URL from
`SlidingSyncRoom::avatar_url` or from `Room::avatar_url`.
Based on https://github.com/ruma/ruma/pull/1607, this patch adds
support for `avatar` from a sliding sync response. This patch implements
`SlidingSyncRoom::avatar_url` to get the avatar URL of a sliding sync
room.
The batch subscriber exists in `matrix_sdk_ui::RoomList` (https://
github.com/matrix-org/matrix-rust-sdk/pull/2322) instead of being added
here.
We must keep the observable capacity to 4096, but the `TODO` is no
longer relevant.
Why keeping the capacity to 4096? Because if the batch subscriber (in
`RoomList`) isn't “listened” quickly, we don't want to get a `Reset`.
This patch brings a nice code simplification.
Instead of creating a new `Stream` with `tokio` based on
`Subscriber<State>`` to drain the batch subscriber for
`RoomList::entries` and `::filtered_entries`, we can _simply_ use
`Subscriber<State>` directly! It removes one dependency: `tokio-
stream`, and remove possible issues with the broadcast channel
`tokio::sync::broadcast`. The code is much simpler and straighforward.
As discussed, we think the entire UI crate should be considered experimental. We've also
observed a proliferation of feature flags there (many of those are my wrong doings, sorry).
Since the one internal user (FFI) of that crate enabled all experimental features, it seems
fine to make them all default, while not providing any extra stability guarantee based on that
action.
* feat: run a room sliding sync upon receiving a notification, to get its full content
* test: add test for the new sliding-sync in notifications \o/
* fix: try to get the push rules *after* a possibly-successful event decryption
* feat: set `is_noisy` only if we could build a push context, and test it
* feat: expose the `legacy_get_notification` in the FFI layer
* feat: retrieve events with a `/context` query
* fix: also request the client's user id's member information to get their display name
* feat: include the list of invites in the notification sliding sync
* feat: repeat the query multiple times if the event hasn't been immediately found
* chore: simplify retrying decryption
* chore: fail the sliding sync when fetching a notification if not using a memory store
* chore: update test expectations + sort invites by recency
* chore: cargo fmt
* chore: simplify getting push actions
Either they were already available in the timeline event if we had to re-run decryption, or
we manually compute them. (Previous comment was incorrect, the `push_actions` are now optional
because of another PR of yours truly.)
* fixup! chore: fail the sliding sync when fetching a notification if not using a memory store
* feat: try to handle invites correctly
* chore: build a local client with an in-memory store instead of reusing the parent client
* fix: remove dubious annotation
* feat: allow cloning a Client and modify it on the fly, use that for the notification client
* feat: get rid of the with_memory_state_store public func on ClientBuilder
* feat: include sender_id in the notification invite event
* feat: put sender's id in the `NotificationSenderInfo`
* feat: inherit the parent session when creating a notification client
* chore: reformat comments
* TMP: add logs
* chore: regenerate the olm machine when inheriting a session too
* chore: keep the parent client around for legacy_get_notification/with_context
The proxy server can (and does) redispatch unrelated lists/rooms from other sliding sync connections,
so the encryption sync would sometimes see room events. Apparently since this is not considered a bug
in the SS proxy, so we shouldn't spam error traces every time this happens.
Since `RoomList::entries` returns a batch stream, the listener
now receives a `Vec<VectorDiff<RoomListEntry>>` instead of a
`VectorDiff<RoomListEntry>`.
Only updated the macro is required here. Instead of calling
`StreamExt::now_or_never` on the `$stream`, we call `Iterator::next` on
`$entries` which is a `Vec<VectorDiff<_>>` now.
This patch uses a newly implemented `async-rx` crate, that provides
`StreamExt`. This trait provides new features on `Stream`, like
`StreamExt::batch_with` which allows to batch values generated by a
`Stream` into a `Vec<Stream::Item>`. The batch is drained based on
another `Stream`: every time a value is produced, it drains the batch
stream.
This feature is used in `RoomList::entries` and
`RoomList::entries_filtered` to batch `Stream<Item = VectorDiff<_>>`
into `Stream<Item = Vec<VectorDiff<_>>>`.
The “drainer” is a broadcast sender, which sends an (empty) value every
time the room list service state changes, so every time something
happens during a sync. Note that it even drains when the room list
service state jumps to `Error` or `Terminated`.
In `RoomListService`, there is a cache over the `Room`s to avoid re-
computing them every time the user calls `RoomListService::room`. It
also allows async operations to have time to finish while the user
jumps back to the room list and so on.
When reading this cache, which is `matrix_sdk_common::RingBuffer`,
`RingBuffer::iter` is used with `Iter::find` to find if the
`Room` exists in the cache. The cache has a size of 128 (given by
`ROOM_OBJECT_CACHE_SIZE`), which is quite large.
`Iter::find` will iterate from the oldest to the newest room in the
cache, but realistically, I reckon we need to iterate from the newest
to the oldest room. Indeed, the app user is more likely to jump between
recently opened rooms more frequently, thus saving quite a lot of
iterations for usual cases.
One may argue than in practice, a user won't open 128 rooms anyway… :-p.
Up until now, users had to listen for to-device events to check for
secrets that were received as an `m.secret.send` event.
This has a bunch of shortcomings:
1. Once the has been given to the consumer, it's gone and can't be
retrieved anymore. Secrets may get lost if an app restart happens
before the consumer decides what to do with it.
2. The consumer can't be sure if the event was received in a
secure manner.
This commit ads a inbox for our received secrets where we will long-term
store all secrets we receive until the user decides to delete them.
It's deemed fine to store all secrets, since we only accept secrets we
have requested and if they have been received from a verified device of
ours.
It's preferrable that users make use of the `App::observe_state/current_state` methods, instead
as there's another sliding sync in the `App` and it properly unifies the current state of both.
Before this patch, the inability to compute push actions would result in an empty vector of
push actions, leading to the false assumption that there's no push actions to account for,
while we were unable to compute them properly. This changes things up so that it's now the
user's responsibility to decide what to do when those push actions are missing.
Previously we would generate one-time keys, if needed, whenever we tried
to upload them. This code-path is critically missing a `save_account()`
call and we would only persist the account once we uploaded the one-time
keys.
This patch changes things up to generate one-time keys whenever we
receive new one-time key counts from the sync response. This aligns
with the way we generate fallback keys and removes the need to introduce
a new place where we persist the `Account`.
It's still possible to re-upload the same one-time keys, in the case
where the upload process succeeds on the server side but we fail to
receive the response.
Co-authored-by: Denis Kasak <dkasak@termina.org.uk>
This will limit the memory used by the cache entries (while it was unbounded before). It's now possible to do this,
since we have the `latest_room_event` handy for all the rooms; using the unbounded cache before was papering over
the lack of that feature.
We can bikeshed on the number of entries in this cache. It has to be small enough to not blow up memory (and keep
the linear search over room id fast, but it's secondary), and high enough that we don't hit the full timeline
re-build path that often.
* chore: rename EncryptionSyncMode variants
* feat: split the encryption sync modes into two different functions
* feat: make locking optional in the `EncryptionSync`
* feat: experimental notification client that retries decryption if it failed the first time
* fix: don't iloop retrying decryption
* chore: helper to convert from bool to `WithLocking`
* feat: don't loop and just retry decryption of the notification event linearly
* feat: remove unused set_notification_delegate
Dead code is dead.
* ffi: get rid of `get_notification_item` and introduce the `NotificationClient`
* fmt
* feat: don't swallow encryption sync errors when retrying notification event decryption
* keeping a tidy commit history is NP-hard
* will i ever learn
* chore: enable experimental-notification-client in the FFI crate
* test: add basic integration test for the common path
* Address first batch of review comments, thanks Jonas!
f36a5b8cd7 introduced
deserialize_events, moving the deserialization out of handle_state.
However, the new code in deserialize_events used filter_map, which
caused a deserialization failure to lead to the indices of the
serialized and deserialized events to no longer match up.
This was not caught because iter::zip also does not require that its
arguments have matching lengths, causing the mismatch to cascade for
that batch of events, and persist in the store. An application
affected by this form of corruption can, for example, call
room.get_state_events requesting events of a certain type, and getting
back events of a different type.
Fix the bug by using Option to preserve the length of
deserialize_events' return value, and add an assertion to ensure
handle_state's contract.
Signed-off-by: Vladimir Panteleev <git@cy.md>
This file claimed to have lots of tests, but actually they are not independent
- they are just one big test.
It's not ideal to have one massive test like this, but I don't really have time
to rewrite them and we should stop pretending.
and the latest few encrypted events in Room.
Use the latest event to provide a room preview, and use the encrypted
events to replace the laest event when they are decrypted.
This patch introduces a new `chunk_large_query_over` function. Imagine
there is a _dynamic_ query that runs potentially large number of
parameters, so much that the maximum of parameters can be hit. Then,
this function is for you. It will execute the query on chunks of
parameters.
`chunk_large_query_over` uses `Vec::split_off` to avoid cloning `Key`s
as much as possible. It's difficult to use references here because of
the async nature of `SqliteObjectExt::prepare`.
This patch updates `get_kv_blobs`, `get_room_infos`,
`get_maybe_stripped_state_events_for_keys`, `get_profiles`,
`get_user_ids`, and `get_display_names` to use this new function.
This patch finally adds the `repeat_vars` function to replace in a
more efficient way the `vec!["?"; n].join(", ")` pattern. There is less
memory allocations.
The `RoomList` and `EncryptionSync` API provide a better developer
experience and address concrete needs. Nobody no longer uses the Sliding
Sync bindings, so we can remove them.
For sliding sync that starts both the actual sliding sync request along the e2ee requests,
we need to make sure that, in case of failure of sliding sync, we reset `pos` as soon as possible.
With the code before this patch, the sliding sync response might be available, but the whole
processing could be waiting for the e2ee requests to finish. In the updated version, we spawn
the e2ee requests in a background task, then run the sliding sync request immediately and fail
if it failed.
Another nice benefit is that the e2ee requests won't be interrupted in the middle of processing,
if the sliding sync changed parameters and we cancelled the whole sync a bit too early.
Fixes#2206.
`RoomListServer` defines an `all_room` sliding sync list. This list
starts in selective sync-mode, then it switches to growing sync-mode.
The previous batch size of the growing sync-mode was 50. This patch
updates it to 200, because empirically it seems a better value for
perceived performance.
This patch also rewrites how `State::next` is written. No change in the
code, just comestic.
* sdk: Allow to update notification settings
* Add an event listener for push rule events
* Fix: simplify insertion and deletion of rules
* Fix: Limit rules cloning
* Fix: Unit tests
* Fix: set_room_notification_mode
* Fix: potential race condition when updating the local ruleset
* Refactor RuleCommands
* RuleCommands Unit Tests
* Fix: limit lock usage
* nit: use expression assignment by default
* nit: pass Ruleset by ownership in RuleCommands' ctor
* a few nits: use to_owned() for &str -> String; use free function for notify actions; use clone() where it's more obvious
and use explicit variants so we know we have to consider this match if we were to add a new variant later
* nit: rename RuleCommands::set_enabled to set_enabled_internal
* nit: test modules don't need to be publicized
* tidy tests
* nit: pass an owned RuleCommands in Rules::apply
* add comments/questions in Rules tests
* nit: no need to publicize the tests module
* tweak NotificationSettings tests too
* rustfmt
---------
Co-authored-by: Benjamin Bouvier <public@benj.me>
When the sync has been terminated and restarts (e.g. an app is going from the background to the foreground), then
the sliding sync is always restarting in growing mode, independently of the previous state of the sync (error or normal
termination). This is something the current sliding sync proxy can't handle nicely, because it prioritizes a change in
parameters over live data; as a matter of fact, to alleviate the burden of the proxy, we can try to only restart in
growing mode if we ran into an error (and not if we had a normal termination).
It ensures that we know exactly what's happening here. Tests are still
valid without any changes in the expected data, but it prevents a
potential breaking.
The Sliding Sync list `all_rooms` and `visible_rooms` of the `RoomList`
must have the exact same configurations for `sort`, `filters`, and
`bump_event_types`. Instead of maintaining the same code, this patch
adds a new function: `configure_all_or_visible_rooms_list` that
configures the lists exactly the same. This function also explicitely
configures `sort` so that we do no rely on the default values from
`SlidingSyncListBuilder`.
This patch configures the same `bump_event_types` for the
`visible_rooms` list than for the `all_rooms` list, so that we may be
sure that the sorting/ordering is the same between the two lists.
* ui: Move EventSendState into timeline::event_item::local module
* [WIP] ui: Make pending local echoes stick to the bottom of the timeline
* test: update more test expectations
* chore: tweak comment to slightly better reflect reality
* nit: remove else after return
* fix: the item's insert position is insert_idx, not `items.len()` anymore
* fix: look for remote echo before local echo when processing send state
Previous code assumed that the latest timeline items would be the most recent, and that
if there was a remote echo, it would always be after the local echo, because of that.
That's not the case anymore, so we must look for possibly a remote echo first, and then
if we find it, apply the late update process.
Also, there might remain a day divider added by the local echo, if it were inserted last.
I'm not sure it covers all the cases, but I've now made it so that the day divider is removed
if it was the last element.
* feat: switch strategy; keep on pushing if there's nothing in the timeline yet
* Revert "test: update more test expectations"
This reverts commit 400cc93ba7c98042a28b5e8d5042899e854f6cff.
* test: reset test expectations
* Address review comments
* fix: don't mix up latest event with any status, with latest non-failed event index
---------
Co-authored-by: Jonas Platte <jplatte@matrix.org>
For verification-by-emoji, the spec uses the word "Aeroplane" rather than
"Airplane". Element-web expects us to use the specced words (and will otherwise
complain about the lack of i18n data).
It seems like following the spec will help maintain consistency.
Common extensions are confusing, and they've included `e2ee` and `to-device` by default. This is not a sane default anymore,
now that there's the concept of `EncryptionSync`: it's either we have the encryption sync that enables e2ee and to-device +
a room list sync that doesn't, OR we have a single room list that has both.
Room List was misconfigured to always use `e2ee` and `to-device`, which was incorrect when it's running with the `EncryptionSync`
in the background. This is now removed, and properly tested.
This implements a new time lease based lock for the `CryptoStore`, that doesn't require explicit unlocking, so that's more robust in the context of #1928, where any process may die because the device is running out of battery, or unexpected flows cause a lock to not be released properly in one or the other process.
```
//! This is a per-process lock that may be used only for very specific use
//! cases, where multiple processes might concurrently write to the same
//! database at the same time; this would invalidate crypto store caches, so
//! that should be done mindfully. Such a lock can be acquired multiple times by
//! the same process, and it remains active as long as there's at least one user
//! in a given process.
//!
//! The lock is implemented using time-based leases to values inserted in a
//! crypto store. The store maintains the lock identifier (key), who's the
//! current holder (value), and an expiration timestamp on the side; see also
//! `CryptoStore::try_take_leased_lock` for more details.
//!
//! The lock is initially acquired for a certain period of time (namely, the
//! duration of a lease, aka `LEASE_DURATION_MS`), and then a "heartbeat" task
//! renews the lease to extend its duration, every so often (namely, every
//! `EXTEND_LEASE_EVERY_MS`). Since the tokio scheduler might be busy, the
//! extension request should happen way more frequently than the duration of a
//! lease, in case a deadline is missed. The current values have been chosen to
//! reflect that, with a ratio of 1:10 as of 2023-06-23.
//!
//! Releasing the lock happens naturally, by not renewing a lease. It happens
//! automatically after the duration of the last lease, at most.
```
---
* feat: implement a time lease based lock for the crypto store
* feat: switch the crypto-store lock a time-leased based one
* chore: fix CI, don't use unixepoch in sqlite and do time math in rust
* chore: dummy implementation in indexeddb, don't run lease locks tests there
* feat: in NSE, wait the duration of a lease if first attempt to unlock failed
* feat: immediately release the lock when there are no more holders
* chore: clippy
* chore: add comment about atomic sanity
* chore: increase sleeps in timeline queue tests?
* feat: lower lease and renew durations
* feat: keep track of the extend-lease task
* fix: increment num_holders when acquiring the lock for the first time
* chore: reduce indent + abort prev renew task on non-wasm + add logs
Calling `RoomListItem::full_room` creates its associated `Timeline`.
ElementX calls `full_room` to get the `is_direct`, `avatar_url` and
`canonical_alias` values for _all_ rooms in the room list. Instead of
doing so, let's add those methods directly in `RoomListItem` so that the
`Timeline` isn't created for _all_ rooms.
So. `SlidingSyncList::update_room_list()` does the following:
1. It adjust room list entries,
2. It updates the `maximum_number_of_rooms`,
3. It applies the sync operations on the `room_list` entries,
4. It updates the `room_list` entries for rooms outside the sync
operations.
SlidingSync answers with a `lists` and `rooms`. The `room_list` entries
is updated as follows: either a sync operation from `lists` has been
applied and the entries are updated, or `rooms` contains rooms that
are not updated by `lists` but that receive new events, and thus must
trigger an update in the `room_list` entries.
This patch changes a little bit the last part of this. Initially,
we were updating rooms that are part of `rooms` but absent of
`lists` sync operations, only **for the current list's ranges**.
But that's wrong! First, we have noticed a bug here: the
correct code wasn't `skip(start).take(end.saturating_add(1))` but
`skip(start).take(end.saturating_add(1) - start)`. Note a big deal, we
were iterating on more entries like it was necessary, but everything
was filtered later, so no bug, just useless computations. Second,
this `skip` and `take` is actually… useless. A room to which we have
subscribed can be out of any range, but we still want `room_list`
entries to receive an update for that particular room.
This patch fixes that once and for all.
In practise, the bug wasn't happening because if someone subscribes
to a room, its timeline is likely to be fetched, and updates will be
received, but still, this is better now from a SlidingSync strict point
of view.
This patch does the following:
1. changes `add_timeline_listener` to be async, thus removing one
`RUNTIME.block_on`.
2. simplifies the logic by adopting a more classical pattern we use
elsewhere:
* removes one `spawn_blocking`,
* let's move the `listener` into the task as a `Box` instead of
cloning it as an `Arc`.
Imagine the room list has the viewport set to the range `0..=19`. The
user scrolls quickly to `50..=69` to see something below and immediately
scrolls back to its initial position, without stopping the scroll
at any moment in between. The app using this API might update the
viewport from `0..=19` to… `0..=19`. Updating the viewport, updates the
`visible_rooms` sliding sync list. Since a list is modified, the current
sync-loop iteration is skipped over and a new one restarts, i.e. the
current in-flight request is cancelled… for nothing.
This patch prevents this situation. The current viewport ranges is
stored in `RoomListService`. `RoomListService::apply_input` used to
return `Result<(), Error>`, now it returns `Result<InputResult, Error>`.
The `InputResult` enum is a new type. It represents whether an input has
been applied or ignored, which must be differentiate from errors.
So now, if the viewport changes and it's not different from the previous
value, `InputResult::Ignored` is returned.
According to the spec, when a device receives a verification request and wants
to refuse it, it should send the `m.key.verification.cancel` to the originating
device, rather than broadcasting it.
This opens up a possible race condition where an embedder calls the method twice, generating two new rooms,
but then it's fine as long as they don't cache them somewhere, which we expect they don't.
That avoids recreating a timeline object every time a user calls `RoomListService::room()` with the same room id, so that should speed up
operations like fetching the latest event for rooms we've already entered before.
This patch changes the capacity of the internal buffer of
`ObservableVector` for `SlidingSyncList::room_list` from 16 to 4096.
With an increased capacity, we reduce the probability to send a
`VectorDiff::Reset` to subscribers. `Reset` are happening quite often
for apps using `matrix-sdk`, and it impacts their performance. This
patch tries to improve this situation. This `room-list` contains
`RoomListEntry`, which is can cheap in terms of memory, compared to the
impact of sending `Reset`s too often. That's a tradeoff.
* bindings: Ignore silently malformed userId in migration of tracked users
* Clean: collect will now never fail
Co-authored-by: Benjamin Bouvier <public@benj.me>
* crypto: implement more primitives for the MemoryStore to work in tests
* crypto: change the shape of the `CryptoStoreLock` API
In particular:
- make the lock work across multiple threads of the same process trying to acquire it,
using `num_holders`.
- add a mechanism to get the lock only once (for the NSE process, in case the main app
had acquired the lock before).
* client: add a cross-process crypto store lock, enable it with `Encryption::enable_cross_process_store_lock`
* client: make `preshare_room_key` a critical section of the cross-process lock
* sliding sync: make it possible to define different timeouts for a `SlidingSyncInstance`
This will be handy for the NSE process on iOS, which has very little time to wait for the proxy's responses.
* feat: implement the `EncryptionSync` API (renamed from `Notification` API)
* fixup! client: add a cross-process crypto store lock, enable it with `Encryption::enable_cross_process_store_lock`
* feat: allow disabling e2ee / to-device in the RoomList API
* feat: use same SS id for main/NSE process, reload to-device token from disk before each encryption sync
* fix: better error handling if restoring the to-device token failed
* feat: add logs for the locking functions
* test: add a few tests for encryption sync
* feat: add `reload_caches` method in the `EncryptionSync` + FFI bindings
* chore: clean up FFI loop
* encryption sync: Remove unused errors, specialize some errors
* feat: include termination reason in the encryption sync loop
* feat: add more logs
* chore: fmt + clippy + doc
* comment: precise only in the presence of another process
* Tweak `room_list` APIs to include `_with_encryption` variants
* chore: rustfmt
This patch moves `RoomListService::entries` and `::filtered_entries`
inside `RoomList`. It also implements `RoomList::new`. Finally,
it implements `RoomListService::all_rooms` and re-implement
`RoomListService::invites`.
Basically:
```rust
// 1.
room_list.entries().await?
// 2.
room_list.invites().await?
```
becomes:
```rust
// 1.
room_list.all_rooms().await?.entries()
// 2.
room_list.invites().await?.entries()
```
`all_rooms` and `invites` both return a `RoomList`.
This patch introduces `State::Error`, which replaces
`State::Termianted`. And a “new” `State::Terminated` state is added.
They do the same, but their semantics is different. `Error` is when an
error happened, and it's fine to restart the sync again. `Terminated`
is when a termination has been reached (either because SS sync-loop
has ended naturally, or because `RoomList::stop_sync` has been called),
in this case, it may not be fine to restart the sync automatically for
example.
This patch renames `State::FirstRooms` to `State::SettingUp`, and
`State::Running` as requested by users.
It hides the “inner Sliding Sync logic”, the Sliding Sync lists etc.
This patch implements `RoomList::stop_sync` on the FFI bindings.
Technically, it's no more useful to store the `TaskHandle` of the
`sync`, but it's still here, in case of.
This patch implements `RoomList::stop_sync`. The goal is twofold:
1. It forces to stop the syncing, thus putting the state-machine into
the `Terminated` state, which is semantically better than “stop
polling the `sync`'s `Stream`”.
2. It literally forces to stop the syncing. It cancels pending futures,
it cancels in-flight HTTP requests etc. It's a more robust way to
stop the `RoomList` sync.
It has an account field besides the issuer field.
Also store it as immutable, the data used for authentication will be
stored in another variable.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
`SlidingSync::sync` had a legacy behaviour when a `M_UNKNOWN_POS`
error was received from the server: It was resetting `pos` and sticky
parameters before re-running a sync-loop iteration, hoping to get a
valid response from the server after that. This retrying
mechanism was running up to 3 times in row (represented by the
`MAXIMUM_SLIDING_SYNC_SESSION_EXPIRATION` constant) before stopping the
`sync` for real.
While it seemed a good idea, it actually brings numerous problems:
1. Each iteration in the sync-loop generates a new request, thus making
the `ranges` of the requests to move forwards. For `SlidingSyncList`
that are in `Selective` sync-mode, there is no problem, but for
`Growing` or `Paging` sync-modes, the `ranges` increased. Thus,
when a `SlidingSync` session expires, instead of returning to
the caller to do something clever (like what `RoomList` does: start
again with a small range so that the “first” sync after a session
expiration is guaranteed to be fast), it was running larger and
larger requests, up to 3 times.
2. `M_UNKNOWN_POS` _is_ an error. Yes, `SlidingSync` must reset `pos`
and must invalidate sticky parameters, but the `sync` must be
stopped, and the error must be returned so that the caller can do
something about it. Until now, this error was likely to be missed by
the caller.
3. This legacy mechanism was forbidding `SlidingSync::sync`'s caller to
do something with this special error. And since the caller was blind to
this error, it disallowed more smart error management.
This patch removes this legacy retry mechanism entirely. When
`M_UNKNOWN_POS` is received from the server, `pos` is reset and sticky
parameters are invalidated as before, but the error is returned like any
other error.
This patch renames and updates an existing test that was testing sticky
parameters invalidation on `M_UNKNOWN_POS`, to include some assertions
on `pos` and to ensure that the sync-loop is stopped accordingly.
This is the first PR for splitting the sync loop into two. This offers a new high-level API, `NotificationApi`, that makes use of a separate `SlidingSync` instance which sole role is to listen to to-device events and e2ee; it's pre-configured to do so. That means we're not force-enabling e2ee and to-device by default for every sliding sync instance, and as such we won't either generate Olm requests to the home server in general.
In the future, this new high-level API will hide some low-level dirty details so that its can be instantiated in multiple processes at the same time (lock across process, invalidate and refill crypto caches, etc.).
An embedder who would want to make use of this would need the following:
- a main sliding sync instance, without e2ee and to-device. Using the `matrix_sdk_ui::RoomList` would be the best bet, at this time.
- an instance of this `matrix_sdk_ui::NotificationApi`, with a different identifier.
Note that this is not ready to be used in an external process; or it will cause the same kind of issues that we're seeing as of today: invalid crypto caches resulting in UTD, etc.
Fixes https://github.com/matrix-org/matrix-rust-sdk/issues/1961.
Usually it's impossible to create a Olm session from a pre-key message
twice. The one-time key that should be used for the 3DH step will be
used up and we're going to throw a `MissingOneTimeKey` error.
This used to be true and unproblematic until we added fallback keys,
these keys will not get discarded immediately after they have been used
once.
This means that a pre-key message, for which we already have a Session,
but decryption for it fails, might create a new Session overwriting the
existing one which will essentially reset the ratchet.
This implements a value-based lock in the crypto stores. The intent is to use that for multiple processes to be able to make writes into the store concurrently, while still cooperating on who does them. In particular, we need this for #1928, since we may have up to two different processes trying to write into the crypto store at the same time.
## New methods in the `CryptoStore` trait
The idea is to introduce two new methods touching **custom values** in the crypto store:
- one to atomically insert a value, only if it was missing (so, not following the semantics of `upsert` used in the `set_custom_value`)
- one to atomically remove a custom value
Those two operations match the semantics we want:
- take the lock only if it ain't taken already == insert an entry only if it was missing
- release the lock = remove the entry
By looking at the number of lines affected by the query, we can infer whether the insert/remove happened or not, that is, if we managed to take the lock or not.
## High-level APIs
I've also added an high-level API, `CryptoStoreLock`, that helps managing such a lock, and adds some niceties on top of that:
- exponential backoff to retry attempts at acquiring the lock, when it was already taken
- attempt to gracefully recover when the lock has been taken by an app that's been killed by the environment
- full configuration of the key / value / backoff parameters
While it'd be nice to have something like a `CryptoStoreLockGuard`, it's hard to implement without being racy, because of the `async` statements that would happen in the `Drop` method (and async drop isn't stable yet).
## Test program
There's also a test program in which I shamelessly show my rudimentary unix skills; I've put it in the `labs/` directory but this could as well be a large integration test. A parent program initially fills a custom crypto store, then creates a `pipe()` for 1-way communication with a child created with `fork()`; then the parent sends commands to the child. These commands consist in reading and writing into the crypto store, using a lock. And while the child attempts to perform these operations, the parent tries hard to get the lock at the same time. This helps figuring out a few issues and making sure that cross-process locking would work as intended.
This patch renames `SlidingSyncState` to `SlidingSyncListLoadingState`
because:
1. It's about a list information,
2. It's about the loading state, not a generic state.
The MSC3575 says that if no `ranges` for a list is provided, it defaults
to `0..=99`. We don't want that! This patch sets the default value to
`0..=19` for the `visible_rooms`.
* feat: lazily generate and include the transaction id, only if it's useful
* chore: add a small `LazyTransactionId` wrapper that ensures it's only created once
---------
Signed-off-by: Benjamin Bouvier <public@benj.me>
`RoomList::state` provides a `Subscriber` to the `State`. This patch
modifies the way the state is tested, to ensure that there is always
an update broadcasted even if the state is the same (e.g. if the state
moves from `CarryOn` to `CarryOn`).
This patch updates `RoomInner::timeline` and `::sneaky_timeline` to
`AsyncOnceCell<Arc<Timeline>>`. Adding `Arc` allows to copy the timeline
if necessary more easily.
This patch updates `Room::add_timeline_listener` to return a `RoomTimelineListenerResult`, a new type defined as:
```rust
pub RoomTimelineListenerResult {
items: Vec<Arc<TimelineItem>>,
items_stream: TaskHandle,
}
```
It is not possible to cancel the `items_stream` by cancelling the
`TaskHandle`. Without this, dropping `Room` won't drop the `Timeline`,
as a clone is moved inside the spawned task. As a consequence, it will
endlessly call the listener.
What the FFI user wants is to subscribe a listener to the
timeline updates. This feature is already supported by
`room_item.full_room().add_timeline_listener()`. So we can safely
remove `RoomListItem::timeline` as we are sure it's never going to be
used for now.
It's possible to do `room.inner_room().room_id()` but it's just simpler
to call `room.id()`. This patch adds this shortcut.
It's especially useful for FFI users, where creating a “full
room” (`room_item.full_room().room_id()`) may be expensive.
This patch adds the `subscribe` and `unsubscribe` method on `Room`.
To achieve that, `Room` receives an `Arc<SlidingSync>`, as the
subscription methods are on `SlidingSync`. It's just easier to put them
on `Room` for the API consumer. It makes the API also more elegant.
Finally, the patch adds the appropriate test.
Since stripped and non-stripped room infos use the same types,
the separation is not necessary anymore.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
… and simplify wasm spawn implementation.
This reduces the differences on wasm vs. non-wasm. Of course it's still
possible to rely on details of the different error types, but at least
both implement a few common traits:
`Debug`, `Display`, `Error`.
Look through m.room.member events when processing Sliding Sync
responses, finding those that refer to our user's membership, so that we
can recognise which rooms we have left.
Instead of storing `sliding_sync_proxy` inside the `Client`, this patch
updates the code to store it inside `matrix_sdk::Client` directly.
That way, there is unique place where to store the sliding sync proxy
URL.
This patch changes `matrix_sdk::Client::sliding_sync_proxy` to be a
`std::sync::RwLock<Option<Url>>` instead of `tokio::sync::RwLock<_>`.
It means that all methods reading or writing this field are sync instead
of async, which makes the code a lot more easier. Having an async-aware
lock wasn't necessary here.
* chore: add comments explaining what the position markers are
* feat(sdk): add scaffolding for sticky parameters
* test(sdk): add tests for the sticky parameters API
* feat(sdk): include the bump_event_types in the sticky parameters
* WIP: add TODO comments for deeply nested sticky parameters
* test(sdk): add extra test for sticky parameters
* feat: add extensions in the sticky parameters + test request generation + test since token
* chore: fix merge + get rid of unused inner extensions
* feat: use sticky parameters for lists too
* chore: Introduce the StickyManager, a reusable way to manage per-data sticky parameters
* chore: move the sticky scaffolding to its own module
* chore: clippy + fmt
* chore: get rid of `update_to_device_token`
* review: append SlidingSync prefix to Stick* data structures
* address review comments
* Replace nbsp with regular space
* Get rid of a write-access to a lock (because thanks inner mutability!)
* Add an integration test when losing `pos` in sticky parameters, and avoid `block_on`
Signed-off-by: Benjamin Bouvier <public@benj.me>
This patch implements `RoomList::rooms` to fetch one room. The type
name is `RoomListRoom` to avoid any collision with an existing `Room`
type already.
`RoomListRoom` implements `name`, `timeline` and `latest_event`.
By returning a `Subscriber`, one can call `Subscriber::get` to get the
inner value. Thus, having `state` and `state_stream` is useless. It's
possible to return have `state` which returns `Subscriber`, and we get
one value for two usages.
The example in MSC3575 shows a room with invite_state property, but also
timeline, joined_count etc. properties. That may or may not be very
likely in practice, but I think it makes sense to check for all these
properties even when the room is an invite, and use them if they are
present.
Added some comments based on my understanding of MSC3575. Previously we
had a FIXME about how we were setting the room state to invite or join
based on the presence or absence of the invite_state property. I think
this behaviour is correct, so I added some comments explaining why.
The functional change here is to note that we call
store.get_or_create_stripped_room to find/create the stripped room, and
this actually deletes any room of this ID that is in the store, so our
later call to store.get_room would always fall back to getting the
stripped room, which we already have, so there was no need to do the
get_room call at all.
This patch uses a new `async-once-cell` crate to make `Room::timeline`
and `Room::sneaky_timeline` by making them lazy. Their values are
computed lazily, on-demand, when calling the `Room::timeline` method or
`Room::latest_event`.
This extension was enabled by both ElementX apps by default, along with the e2ee / to-device extensions that will be handled
by the new Notification API. To maintain the current behavior, it's important to re-enable this extension before getting
RoomList in production.
Signed-off-by: Benjamin Bouvier <public@benj.me>
By default, get_media_file will attempt to use a default directory
for temporary files and directories. Unfortunately, there might not be
such a thing on some older Android devices, which use per-application
directories.
For this specific case, an optional temporary directory parameter
is added to the `get_media_files` parameters, so one can provide their own
temporary directory path.
This new parameter is expected to be set at least on Android; other platforms
should work just fine without it.
Signed-off-by: Benjamin Bouvier <public@benj.me>
This was a temporary replacement I've made, waiting for a PR of mine to be merged, and it's been merged since then. Oh well.
Signed-off-by: Benjamin Bouvier <public@benj.me>
* sliding sync: infer the storage key from the loop id and user id
* chore: rename `sync_id` to `id`
* chore: check that the sliding sync id is less than 16 chars
* chore: rejigger the storage key creation logic
Now the prefix is visible only in the `format_storage_key_prefix` function, and other `format_storage_key` function will be based off that.
* chore: update sliding sync README with API updates and fix outdated information
* chore: clippy + fix test
Signed-off-by: Benjamin Bouvier <public@benj.me>
This patch handles errors from the Sliding Sync loop properly.
When an error is received, the `RoomList` state machine enters the
`Terminated` state. Immediately after that, the sync stream is stopped.
When the sync stream is restarted, the next state will be calculated.
When the current state is `Terminated`, the next state is the state
that led to the `Terminated` state. To avoid having a “first” huge sync
(imagine a room list of 1000 rooms, you don't want to “resume” from an
error with a first sync for 1000 rooms), lists are partially “reset”
depending of the state where the machine is in.
An important test has been added to test all possibles cases with errors
and list' states.
First off, `SlidingSync::sync_once` no longer returns
`Option<UpdateSummary>` but `UpdateSummary`. It simplifies the rest of
the patch.
Next, `SlidingSync::sync` returns either `Ok(…)` and continues to sync,
or it returns `Err(…)` and it stops the sync stream immediately. No more
error could be returned with the stream to continue syncing.
Finally, the `UnknownPos` error no longer emits an err, but silently
skip over the sync-loop until the threshold is reached; which in this
case will generate a proper error.
This patch changes `RoomInner` to have an `room:
Option<matrix_sdk::room::Room>` field to `room: matrix_sdk::room::Room`.
`room` is not longer an `Option`.
Then, it's easier to have a new `timeline: Timeline` field, along with a
`sneaky_timeline: Timeline` field. Both are used by the new
`Room::timeline()` and `Room::latest_event()` method.
This patch implements `RoomList::room`, which returns a `Result<Room>`:
the room ay or may not exist.
A new `Room` type is created. It wraps an `Arc<RoomInner>` type, so that
it's cheap to clone it.
The `RoomInner` type owns a `SlidingSyncRoom` and
`matrix_sdk::room::Room` type. If some API or data are missing on
the former, the latter acts as a backup. This is the case for the
`Room::name` method which makes it best to return a name to the caller.
This patch lands the first design of the `Input` API. An `Input` is
something external that can be understood by the `RoomList` state
machine. The first inpuput is `Viewport` to change the “viewport” of the
`RoomList`, which translates to the change of the ranges of one specific
Sliding Sync list.
This patch implements `RoomList::update_entries_stream_filter` which
allows to… change the… entries stream filter. This patch also provides a
test for that. How nice it is.
This patch adds a new method on `SlidingSyncList` named
`room_list_filtered_stream`, which uses the new `eyeball-im-util` crate
with its new `FilteredSubscriber` type.
This patch does several things.
First, `SlidingSync::on_list` is now async, and accept async closures.
Second, `SlidingSync::lists` and `::rooms` are behind an `AsyncRwLock`
instead of a `StdRwLock`. The rest of the patch updates the consequence
of this.
This patch replaces a panic (`Option::expect`) by a `tracing::error`
log.
Imagine the Sliding Sync proxy responds with the following payload:
```json
{
"pos": …,
"lists": {
…: {
"count": …,
"ops": [
{
"op": "INSERT",
"index": 0,
"room_id": !foo:bar.org",
}
]
}
},
"rooms": []
}
```
It's an invalid response, as it will update the room list entry (because
it's present in `lists.$list.ops`), but the room won't be created (it's
absent in the `rooms`).
This situation creates a panic in the patched code. We don't want to
crash if the server replies with invalid data.
Ultimately, we should check that the sync operations are valid regarding
the rooms' updates.
This patch uses `FutureExt::now_or_never` so that we don't wait on the
future to be resolved to get a result. If we have a bug in our code and
the test expects a value, the test will hang forever, which is not a
desired behavior. With `now_or_never`, this situation cannot happen.
The only reason why a sender can fail to send a message is because there
is no receiver. In the current design of `SlidingSync`, there is only
one receiver active per sync-loop. Thus, calling `SlidingSync::add_list`
may fail if `sync` has not been started. Hence the need for a new
`internal_channel_send_if_possible` method which will never fail: it
will send a message is possible, otherwise it won't do anything.
This turns more functions infallible.
`tokio::sync::broadcast` is interesting as it's possible to
generate receivers per subscribers. Being able to do that allows
us to remove the new for an `AsyncLock` around the receiver in
`SlidingSync::internal_channel`, and it can even remove the need to hold
a receiver entirely!
Another improvement is that new receivers can't receive past messages,
so we no longer need to drain the internal channel.
Another improvement is that the sender' `send` method is synchronous!
Which helps to make many functions no longer `async`.
Something weird is happening with ElementX iOS. When
`SlidingSync::stop_sync` is called, the internal message `SyncLoopStop`
has time to be sent in the internal channel, but iOS decides to suspend
the code before the sync-loop processes it. When ElementX decides to
start the sync-loop again, it immediately processes the `SyncLoopStop`
message, and… stops the sync-loop.
This patch ensures that `SlidingSync::sync` drains the internal channel
before starting the sync-loop for real. `tokio::sync::mpsc::Receiver`
type has no `drain` method, so this patch implements its
own logic by calling `try_recv` in a loop, until it returns
`Err(TryRecvError::Empty)`.
* feat: introduce SlidingSyncSelectiveModeBuilder
* feat: get rid of `CannotModifyRanges` \o/
* chore: rustfmt
* chore: remove scope in test
* chore: move request generator update to its own mod, gets rid of `set_ranges`
* chore: add comments on the request generator methods
* chore: sink one range_end definition into the match arm that matters
* ffi: remove unused `add_range` method
* test: update incorrect test expectation
* chore: make clippy happy
* address first review comments
* review: don't reuse the ranges field for two things
* chore: use a builder pattern for sliding sync selective mode
* address review comments + CI
Signed-off-by: Benjamin Bouvier <public@benj.me>
In case it's not obvious to drop the `Stream` returned by
`SlidingSync::sync` immediately to “stop” the sync-loop, one can use the
new `stop_sync` method to do achieve the same result.
The `VerificationRequest` object is used to control the flow of the
verification but only up to a certain point.
Once we start handling of different specific verification flows (i.e.
SAS or QR code verification) the `VerificationRequest` object creates a
child object of the Verification type.
This patch adds a new `VerificationRequestState` variant called
`Transitioned` which holds the child verification object as associated
data.
This makes it much simpler to go through the whole verification flow by
allowing users to just listen to the `VerificationRequest::changes()`
method.
We suspect that there might be too many internal messages being pushed, causing a deadlock on the senders' side.
This attempts to increase the value of the buffer to give it more leeway.
Signed-off-by: Benjamin Bouvier <public@benj.me>
The `SlidingSyncListInner::timeline_limit` and `::ranges` fields were
observable (behind `eyeball::Observable`). It was actually useless
as those fields were never exposed to the public API, thus it was
impossible to subscribe to them.
This patch cleans up that. `timeline_limit` and `ranges` are no longer
observable.
The comment and the check in the code were contradicting each other; the comment was wrong, and the code was right, so there's that.
Signed-off-by: Benjamin Bouvier <public@benj.me>
This patch removes our dependency to libolm completely. This should
allow WASM targets to use the backups_v1 feature of the
matrix-sdk-crypto crate as well.
First off, this patch changes `pushd(…)` to `pushd(…)?` so that errors
are propagated.
Second, instead of assuming that all crates live in `crates/`, let's
allow to precise a prefix, like `crates/` or `bindings/` directly in the
“folder” path of `args`.
Changing the sync-mode of a list on-the-fly is necessary to optimise the
new `RoomList` API. For example, we can start with a list in a selective
mode, a range of `0..=50` and a `timeline_limit=0` to fetch the
beginning of the room list, and then _change_ the sync-mode to growing
to continue to sync the room list in the background.
Today, this is done with 2 lists and a merge of the lists, but
(i) this is error-prone, (ii) this is not optimal. Thank to
`SlidingSyncList::set_sync_mode`, merging lists is no more necessary,
thus removing a class of bugs in client's code.
This patch replaces `sync_mode` on `SlidingSyncListBuilder` by
`sync_mode_selective`, `sync_mode_paging` and `sync_mode_growing`, which
removes the need to use `SlidingSyncMode` directly.
This patch also removes `batch_size`, `room_limit` and `no_room_limit`
as it's now arguments of `sync_mode_paging` and `sync_mode_growing`.
`SlidingSyncMode` was previously declared as:
```rust
enum SlidingSyncMode {
Selective,
Paging,
Growing,
}
```
Now, the new declaration is:
```rust
enum SlidingSyncMode {
Selective,
Paging {
batch_size: u32,
maximum_number_of_rooms_to_fetch: Option<u32>,
},
Growing {
batch_size: u32,
maximum_number_of_rooms_to_fetch: Option<u32>,
}
}
```
First off, it helps to remove the `full_sync_batch_size` and
`full_sync_maximum_number_of_rooms_to_fetch` methods and fields from
`SlidingSyncListBuilder`. It was containing default values in case of.
That was useless and needed to be fixed. Also, calling a `full_sync_*`
method with the `Selective` mode had no effect, which could disturb
the user. Well, now everything is clean on that front.
Second, `SlidingSyncListRequestGenerator` no longer has constructors,
but a `From<SlidingSyncMode>` implementation. However, the constructors
now live in `SlidingSyncMode` with `new_selective`, `new_paging` and
`new_growing` methods which are helpers. All in all, it makes creating a
`SlidingSyncListRequestGeneator` simpler: just call `sync_mode.into()`
and boom.
Finally, this patch removes the `default_with_fullsync` list builder
helper, which makes no sense at all.
`SlidingSync::subscribe`, `::unsubscribe` and `::add_list` are
now async. `subscribe` has been renamed to `subscribe_to_room` and
`unsubscribe` to `unsubscribe_from_room`.
`SlidingSyncRoom::subscribe_and_add_timeline_listener` has
been removed, and replaced by a new `::subscribe_to_room` and
`::unsubscribe_from_room` methods (the `::add_timeline_listener` method
already exists).
`TaskHandle::finalizer` is no longer necessary, thus this code has been
cleaned up.
This bug has been found by @bnjbvr, all the credits go to him. I've just
added some comments around his code.
Prior to this patch, the room unsubscription buffer
(`SlidingSync::room_unsubscriptions`) was reset before the request was
sent. So if something went wrong, the next request would not include the
room unsubscriptions.
This patch updates this behavior. First, it replaces `Vec` by `HashSet`
to avoid a O(n^2) look up.
Second, a copy of room unsubscriptions used by the request is kept, so
that it can be used to cherry-pick which room unsubscription to remove
from the buffer once a response from the server is received. It's
important to not clear the entire room unsubscriptions buffer as more
unsubscriptions could have been inserted meanwhile.
This has slightly drifted from the initial design I thought about in the issue.
Instead of having `build()` be fallible and mutably borrow some substate (namely `BTreeMap<OwnedRoomId, SlidingSyncRoom>`) from `SlidingSync` (that may or may not already exist), I've introduced a new `add_cached_list` method on `SlidingSync` and `SlidingSyncBuilder`. This method hides all the underlying machinery, and injects the room data read from the list cache into the sliding sync room map.
In particular, with these changes:
- any list added with `add_list` **won't** be loaded from/written to the cache storage,
- any list added with `add_cached_list` will be cached, and an attempt to reload it from the cache will be done during this call (hence `async` + `Result`).
- `SlidingSyncBuilder::build()` now only fetches the `SlidingSync` data from the cache (assuming the storage key has been defined), not that of the lists anymore.
Fixes#1737.
Signed-off-by: Benjamin Bouvier <public@benj.me>
Co-authored-by: Jonas Platte <jplatte+git@posteo.de>
Currently, the `body` of a `SignatureUploadRequest` includes a spurious
`signed_keys: {...}` property in which the actual content is wrapped. Fix that.
We were using `txn_id` as a way to identify the running stream. Now
things are cleaner so we can remove this “debug tool”. This class of
problems should not appear anymore. For the record, the biggest problem
was the following: It was possible to start multiple `stream`s at the
same time, and thus a stream could receive a response sent by another
stream. Since we no longer need to restart SlidingSync anymore (i.e. no
need to start multiple `stream`s at the same time), this problem should
not happen.
Prior to this patch, the `v4::Response`, aka the SlidingSync response,
was processed by the client and transformed into a `SyncResponse` into
`sync_once` before being passed to `handle_response`. Now it's done in
`handle_response`.
This is not mandatory _now_, but it will be helpful in a close future.
* chore: use RangeInclusive instead of raw start/end integers for ranges in sliding sync
* chore: have timeline_limit use a u32 instead of a Ruma UInt
* chore: Remove all the `Into<u32>` generics on timeline_limit and ranges APIs
* chore: introduce a `Bound` type alias for u32
Signed-off-by: Benjamin Bouvier <public@benj.me>
This patch changes `SlidingSyncRoom` to move all its fields inside an
inner type `SlidingSyncRoom` behind an `Arc`, so that cloning is cheap,
and all clones are sharing the same state.
A `SlidingSyncList` has a state, which can be either `NotLoaded`,
`Preloaded`, `PartiallyLoaded`, or `FullyLoaded`.
A `SlidingSyncList` can be reset, either manually when
`SlidingSyncList::reset` is called, or when a range is updated for
example.
Resetting a list is modifying its state. Prior to this patch, the state
was set to `NotLoaded`. However, it's not entirely true. This patch
updates this behavior to the following rules:
* When the state is `NotLoaded`, it's kept at this state as nothing has
happened yet,
* When the state is `Preloaded`, the list is restored from the cache,
but nothing has happened yet too, so it's kept at this state,
* When the state is `PartiallyLoaded` or `FullyLoaded`, it means some
(or all) updates have been done, so the new state is `PartiallyLoaded`
after the reset.
The list' state is used mostly by the client to know whether a loader
should be prompted to the users. The ranges are modified when the
users scroll inside the room list for example: scrolling in the room
list doesn't imply the state should go to `NotLoaded`. Some data
have potentially be loaded, so changing the ranges should result in a
`PartiallyLoaded` state. It seems more logical once explained like this.
`invite_state` is managed when the Sliding Sync response is processed/
handled by the `Client`. Inside `SlidingSyncRoom`, it's not necessary to
maintain it.
This patch creates a constant to represent the
maximum number of timeline event to put in the cache:
`NUMBER_OF_TIMELINE_EVENTS_TO_KEEP_FOR_THE_CACHE`. Verbose, but easily
understandable.
This patch also renames a few variables.
This patch simplifies `SlidingSyncRoom::timeline_queue` from
```rust
Arc<RwLock<ObservableVector<SyncTimelineEvent>>>
```
to
```rust
Vector<SyncTimelineEvent>
```
First, we don't need to be observable. It's never observed since
it's private, and even privately, it's never observed, there is no
subscriber.
Second, no lock is required as updates happen synchronously.
Third, `Arc` is not necessary. We want each clone of `SlidingSyncRoom`
to not share any state across them.
Finally, this patch simplifies the iterator + `.push_back` by a simple
`.extend` to update the `timeline_queue`. Behind the scene, `impl Extend
for Vector` actually does an iterate + `.push_back`; let's keep our code
simple though.
`SlidingSyncRoom::is_cold` is not well-named. This patch introduces an
enum, named `SlidingSyncRoomState` which contains more detailed state:
`NotLoaded`, `Preloaded`, and `Loaded`.
The use of `AtomicBool` is also removed. Thus, we no longer need an
`Arc` for the state, which makes `Clone` more obvious: the state is not
shared across clones anymore.
A `SlidingSyncRoom` receives an Ruma
`api::client::sync::sync_events::v4:SlidingSyncRoom`. This value is
stored in the `SlidingSyncRoom::inner` field. From here, some getters
like `name()`, `is_dm()` etc. are using the `inner` field to compute a
result. There was one exception though: `prev_batch`. This value is part
of `v4::SlidingSyncRoom` but it was copied and updated in its own field:
`SlidingSyncRoom::prev_batch`.
I was wondering why. Turns out, there is no reason. Its getter
`prev_batch()` is public, but it's not used by the FFI bindings, so
basically nobody uses it (as this project is experimental as the time of
writing, we know our users).
This patch removes the `SlidingSyncRoom::prev_batch` field.
This patch also removes the `SlidingSyncRoom::prev_batch()` getter.
This patch finally removes the `FrozenSlidingSyncRoom::prev_batch` field
too.
Prior to this patch, to check a room exists, it was removed from the
collection, then re-added if it exists. Instead of doing this `remove`
+ `insert` dance, we can simply use `get_mut`, which also returns an
`Option`. This patch does that, in addition to rewrite the code to use a
`match` instead of a `if` + `else`.
`SlidingSync::subscribe` and `SlidingSync::unsubscribe` will cancel in-
flight requests, i.e. the `SlidingSyncInternalMessage::ContinueSyncLoop`
will be sent in the internal channel, just like what `SlidingSyncList`s
already do when a parameter is changed.
If the redaction already happened server-side, it is a bug for the
server to even still have access to the non-redacted form.
We don't accept the server data in this case, and also log a warning.
`on_built` is a method that registers a closure. This closure is called
when a list is built by `SlidingSyncListBuilder::build`. It receives
a `SlidingSyncList` and returns a `SlidingSyncList`, the list can be
updated or returned as is. It allows to configure a `SlidingSyncList`
right after it's built. `SlidingSyncBuilder::build` is responsible to
finalize the configuration, and to build all the lists. Once they are
built, the state is restored from the cache. If one wants to configure
a list before the state is restored from the cache, `once_built` will
serve well.
The timestamp value of the Notification was not reliable since it was the
timestamp in which it was generated internally, not the timestamp of when the
event was sent, now we instead expose the `origin_server_ts` of the
TimelineEvent.
`is_read` is also very unreliable, so it's just removed for now.
Currently this takes a `CrossSigningKeyExport`, but that's not a class you can
construct from the JS side. Instead, let's just pass in the individual keys.
Using the #[derive(uniffi::Object)] on certain types make them not show up
in the .aar file when building for Android. Could not determine why that is,
but removing the derive macro, and adding the empty interface declarations
inside the UDL, makes them show up.
We can still use #[uniffi::export] and #[uniffi::constructor].
Signed-off-by: Simon Farre <simon.farre.cx@gmail.com>
Currently, we can't pass the response to a `SigningKeysUploadRequest` into
`mark_request_sent`, which apparently we should be able to do.
So, we need to make `SigningKeysUploadRequest` have a `type` like the other
`OutgoingRequest`s, and add support for `SigningKeysUploadResponse` to
`OwnedResponse`.
Because since SlidingSync no longer restarts, the behaviour is really
different. The current way the tests are written cannot assert the full
behaviour. We need to rewrite this test suite entirely.
The unknown filter can be difficult to match with MembershipState,
depending on the store implementation.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Storing as bitset is not future-proof.
Just fix the latest migration. We assume no-one used it yet.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
The MPSC channel that has been introduced in recent commits is now used
in this patch. The `SlidingSync::stream` method can be controlled via
the `internal_channel`.
This patch updates `SlidingSync::stream` to use the `tokio::select!
` macro to select any future that resolves first between the
`internal_channel` receiver, or `SlidingSync::sync_once`. Fairness is
biaised as the `internal_channel` has the priority.
This mechanism is already used by this patch: `SlidingSyncList::reset`
will send the `SlidingSyncInternalMessage::ContinueSyncLoop` to
“continue”… well… the sync loop, i.e. it will cancel in-flight waiting
for a response.
This entire mechanism removes the need to “stop” and “start”, i.e.
“restart” the `SlidingSync::stream` method manually, which was the
source of many bugs. Now everything is controlled internally.
Prior to this patch, `SlidingSync::add_list` was taking a
`SlidingSyncList`. However, we need to inject more data when building
the list, so let's modify `add_list` to take a `SlidingSyncListBuilder`
instead. It's even better for the user as calling `build()` isn't
necessary anymore.
It's not possible to clone a `SlidingSyncList` anymore. Why? Because
it's not correct. Prior to this patch, it was possible to add a list
to a `SlidingSync` instance, then add a clone of the same list to
another `SlidingSync` instance. Weird behaviors could happen, but more
importantly, for the next Sliding Sync design we are working on, it
could lead to gigantic bugs.
Removing `Clone` from `SlidingSyncList` makes the code simpler.
For example, `SlidingSyncList.inner` no longer needs an `Arc`, or
`SlidingSync::stream` no longer needs to clone all the lists.
The `SlidingSyncBuilder` now has a new method: `bump_event_types`, to
configure the `bump_event_types` HTTP request parameter. This value
is passed to `SlidingSync`, which is used to build the `SlidingSync`
request.
The format of the outbound group session struct has changed. We nowadays correctly rotate the group session if we can't restore it, but it's still good to avoid logging the error in this case.
An error is currently thrown if loading an outbound group session fails.
This error may only affect loading outbound group sessions, while other
store operations continue to work fine. As a result, the user is unable
to send messages, but can still use the application without any problems.
Since outbound group sessions are rotated frequently, we can simply
rotate the session if loading fails. However, if the error is related to
a more serious storage issue, persisting the newly rotated outbound group
session will also fail. If the error is specific to outbound group
sessions, the user will be able to continue using the application without
interruption.
The SQLite crypto store uses rmp_serde to serialize all the data we're
going to store. This works nicely for most things, one exception to this
is the OutboundGroupSession type.
The OutboundGroupSession type stores to-device requests to ensure that
the session doesn't get used before it is shared with the whole group
and to ensure that the to-device requests get restored if the
session gets restored after an application restart.
The to-device requests type critically contain `Raw<AnyToDeviceEvent>`,
the `Raw` type here being the serde_json::Raw type. rmp_serde seems to
serialize this just fine, but later deserialization fails.
We're avoiding the issue by using serde_json to serialize the
OutboundGroupSession.
Those methods are public, but never used by our current users.
Moreover, there is some big overlaps. For example, `storage_key`,
`cold_cache` and `no_cache` do the same thing: They update the
`storage_key` field. Or `add_fullsync_list` is just a helper that is
never used except in the tests etc.
The test does the following:
1. Create 2 (identical) lists,
2. Do a sync.
3. Assert that the 1st and 2nd lists are receiving an update,
4. Add a 3rd (identical) list,
5. Do a new sync,
6. Assert that 3rd list is receiving an update.
This last step is wrong. All lists should receive an update as they
are identical.
This patch updates a test to ensure that `room_list` receives “diff”
for rooms that are modified by a sync operations, but also by another
updates (like a new event).
The `SlidingSyncList.room_list` field is used to store the list of rooms
for a particular Sliding Sync list. It's used by the user to receive
“diff”s when a room sees its position modified. For example when a new
room receives a message, it “climbs back up” the entire room list to
be at the top place. Another example is when a new room is created,
some rooms will move around to give the new room a space. All those
moves will create “diff”.
However, the user also expects to receive a “diff” when a room has
received some updates, even if it's position doesn't change. For
example, when a room is already at the top of the list but receives a
new message: It has received an update, but its position stays the same.
This specific latter feature was implemented before, but it has been
removed by accident in https://github.com/matrix-org/matrix-rust-sdk/
pull/1699 (more specifically in https://github.com/matrix-org/matrix-
rust-sdk/pull/1699/commits/861a05be69a566d9a4ad125dc6ecb418d2b3210f).
At that time, it was not clear why the code was filtering for specific
filled room entires, to set the filled room entries, at the same
position. Zero comment, zero test. I didn't consider this as a feature
but as a bug.
So this patch re-introduces this feature. Hopefully in a more optimal
way. If a room has already triggered a first “diff” because of a
position change, it won't trigger a second “diff” because it has
received an update (which was the case before).
The `SlidingSyncList::handle_response` method has also been renamed
`update`, as it does update, whenever it comes from.
The code has been commented and documented to explain this feature.
Existing tests have been updated, especially for `apply_sync_operations`
which now ensure the `rooms_that_have_received_an_update` collections is
updated accordingly. Another test has been updated specifically to test
the “diff”s received by `room_list`.
This patch also re-uses the `format_storage_key_*` functions to use the
same key as the `restore_sliding_sync_state` function. It fixes a bug
where the keys were mismatching.
This PR is self explanatory.
Exposes said function to the FFI. Defined in the .adl file,
and defined to throw `ClientError`
Signed-off-by: Simon Farre <simon.farre.cx@gmail.com>
Co-authored-by: Damir Jelić <poljar@termina.org.uk>
In some cases, restoring client state from backups may cause Olm sessions
to become corrupted, resulting in a backward ratchet. As a consequence,
the receiving side is unable to decrypt messages. To address this issue,
the failed side initiates a new Olm session and sends a dummy encrypted
message.
However, this dummy message also creates a new Olm session on the
sender's side. To ensure that both sides use the same new session, we
must sort sessions by their creation timestamp before encrypting new
messages.
Although the session list was sorted correctly previously, we selected
the wrong side of the list, resulting in an outdated session being
selected instead of the newest one. This patch resolves the issue by
selecting the newest Olm session and adds a test to the session getter
logic to prevent reintroduction of this bug.
Prior to this patch, when `SlidingSync` was built with a storage key,
the storage was read to find a previous `SlidingSync` version that
was in a cache. To put a `SlidingSync` instance in the cache, it
was serialized to JSON. When one reads from the cache, the value is
deserialized. A problem happens when the internal representation of
`SlidingSync` and other types (e.g. `SlidingSyncList`) have changed (due
to an SDK update for example): Loading a previous state from the cache
results in an error.
After discussion with the teams, clients don't want to deal with
obsolete cache values. In such a scenario, (i) they cannot interact
with the cache to remove the obsolete entry, and (ii) their only
possibility is to log out and log in the user again. What an unfriendly
user experience!
This patch modifies this behaviour. When loading a `SlidingSync` type or
a `SlidingSyncList` type from the cache, 3 cases are now handled:
1. The cache entry exists and has been successfully deserialized: in
this case, we do our business.
2. The cache entry exists, but it wasn't possible to deserialize it: the
cache entry is declared as obsolete, and the entire `SlidingSync`
cache is cleaned up, so all entries for `SlidingSync` and
`SlidingSyncList` are removed from the storage,
3. The cache entry doesn't exist: we do nothing particular.
Note that if one cache is obsolete, all cache entries are removed.
The `SlidingSync::stream` method no longer resets the lists everytime
it is called. If one wants to reset the lists, they need to call
`SlidingSync::reset_lists`.
`SyncOp::Invalidate` means _invalidating_ a particular range. When a
room is `Filled`, it becomes `Invalidated`, when it is `Invalidated` it
stays `Invalidated`, and when it is `Empty` it stays `Empty`.
Before this patch, `Empty` was becoming `Invalidated`, which apparently
is a bug.
This patch also fixes out-of-bound accesses, and adds many tests.
Finally, this patch renames `update_state` to `update_room_lists`.
`SyncOp::Insert` means _inserting_ a new room ID. The `rooms_list`
contains all possible rooms (based on `maximum_number_of_rooms`, a value
returned by the server). Here, inserting = setting
`RoomListEntry::Filled` at a particular index of `rooms_list`, that's
it.
The previous code was doing very complex stuff, like removing things
around the `index` if something etc. It was using the requested ranges
(the range passed to the request) etc.
Applying `SyncOp` should be simple and is focused on updating
`rooms_list` only: the requested ranges have nothing to do here.
This patch also prevents against out-of-bounds acccesses, which wasn't
the case before.
This patch prevents out of bounds acceses for `SyncOp::Delete`, and adds
more tests.
This patch also removes a cast from `UInt` to `u32` to `usize`. It's now
from `UInt` to `usize` directly.
First off, this patch renames `ops` to `sync_operations` and `room_ops`
to `apply_sync_operations`.
Second, the `SlidingOp::Sync` was creating an out-of-bounds access
depending of the range present in the server's response. For example, if
the `rooms_list` contains 5 elements (because the
`maximum_number_of_rooms` is set to 5), and the server replies with:
```json
{
"op": "SYNC",
"ranges": [3, 17],
"room_ids": […]
}
```
the previous code was setting a new `RoomListEntry` at indices `3..=17`,
whilst the `rooms_list` contains only indices from `0..=4`. That's
annoying.
The previous code was also counting the number of `room_ids` for
nothing, just to execute the iterator that was applying the actual
changes in a `map`. Well, everything was fishy.
This patch updates the code to protect against an unexpected server's
reply by raising an `Err`. This patch also adds tests.
The `updated_rooms` argument was passed to `find_rooms_in_list` to
update the `room_list`: the update is setting the filtered room list
entry to `RoomListEntry::Filled`.
_But_, `find_rooms_in_list` was already filtering rooms which are
`Filled`. So it does… nothing: it filters rooms which are `Filled` to
update them to `Filled`.
So we can remove `find_rooms_in_list` because it becomes useless. And we
can remove `updated_rooms` too.
The `rooms_list` is updated by `rooms_ops` itself. Let's keep
modifications in one unique place.
The `update_state` method of `SlidingSyncListInner` has basically
2 cases:
1. For an initial response,
2. For other responses.
The code between the 2 cases were almost identical. Or, they could be
identical. The few exceptions are:
* In the first case, the `rooms_list` updates were taking the
form of a `VectorDiff::Append`, while the second case, it was a
`VectorDiff::PushBack`.
* In the first case, the `is_cold` flag was set to `false`.
It's fine for the clients to receive only a `VectorDiff::Append` event
only. So let's make it uniform.
And it appears that the `is_cold` field is now private, and never read
anywhere else. So… it's… basically useless. We can remove it! It was
previously used here to know which flow to use, but since we can make
both flows identical, its role becomes insignificant.
`create_range` and `SlidingSyncList::next_request` return a `Result`
instead of an `Option`. It was a legacy from the old `impl Iterator` of
`SlidingSyncListRequestGenerator`. But actually, when `create_range`
fails, it must return a `Err` value, not a `None` value.
And since it's a regular error, `sync_once` can propagate the
error. Thus, it is no longer necessary to “update” the list of
`SlidingSyncList`. It's not necessary to remove some items in this list
when the `next_request` method can no return a value: it always returns
a value, except when a error happens.
First off, `set_ranges`, `set_range`, `add_range` and `reset_ranges`
take a `&[(U, U)]` instead of a `Vec<(U, U)>` when a vector was needed,
or takes a `(U, U)` instead of 2 arguments when it was needed.
Second, all those methods now return a `Result<(), Error>`. The
`Error::CannotModifyRanges` is raised if the chosen `SlidingSyncMode`
doesn't allow to modify ranges. Basically, `Selective` does allow a
user to modufy the ranges, but there is no real ranges with `Growing` or
`Paging`. Let's make it an explicit error.
Finally, `SlidingSyncListInner` has 2 methods: `set_ranges` and
`add_range`, but without any “ranges check”; `SlidingSyncList` does call
the inner methods, but does the checks.
The `SlidingSyncListInner::find_rooms_in_list` method can be greatly
improved by using the `Iterator` API.
It was already using `Iterator::skip`, great! Let's continue by using
`Iterator::take` instead of using a stop index.
And let's use `Iterator::enumerate` to avoid using a mutable index.
`SlidingSyncList::find_room_in_list` and
`SlidingSyncList::find_rooms_in_list` aren't useful (and never used) by
the public consumer of this API. Let's remove it. They are only used for
internal purposes.
Before, the `update_request_generator_state` method was emitting an
`error!` log, but not an error. But that's clearly an error!
This patch updates the method to return a `Result<(), Error>`, and adds
a new variant to `Error`.
There are problems with `SlidingSyncListRequestGenerator`:
* It's part of the module API,
* It contains a clone of `SlidingSyncList`.
To create a `SlidingSyncListRequestGenerator`, one has to
call `SlidingSyncList::request_generator`. It was done in
`SlidingSync::stream`. The problem is that it clones `SlidingSyncList`.
So theoritically it is possible to create multiple request generators
for the _same_ list, and use them to send many requests and to update
the _same_ list with multiple responses. This is utterly error-prone and
can lead to really complex bugs to discover.
Moreover, it's a lot of clones. Cloning a
`SlidingSyncListRequestGenerator` isn't cheap as it means cloning a
`SlidingSyncList`. Moreover, cloning a `SlidingSyncList` isn't cheap as
it means cloning the entire struct.
Having `SlidingSyncListRequestGenerator` inside the module API also
makes the code of `SlidingSync` more complex (why having to deal with
lists and request generators at the same time? we must be very careful
to maintain both side by side? what if a list is removed but not its
request generator? and so on).
So. This patch simplifies all that.
First off, it extracts all the fields of `SlidingSyncList` into a
`SlidingSyncListInner` struct. Then, `SlidingSyncList` has only one
field: `Arc<SlidingSyncListInner>`. Boom, it's now cheap to clone it.
Second, `SlidingSyncListRequestGenerator` is only a
struct with constructors, but there is no extra methods.
`SlidingSyncListRequestGenerator` _no longer_ contains a
`SlidingSyncList`, and doesn't no need a list at all to work. It's just
values.
Third, `SlidingSyncList` holds a `SlidingSyncListRequestGenerator`, and
only one.
Fourth, `SlidingSyncList` (and `SlidingSyncListInner`) now has methods
to handle the internal request generator. The initial `impl Iterator for
SlidingSyncListRequestGenerator` becomes a simple
`SlidingSyncList::next_request` method.
Fifth, previously, the `SlidingSyncList::handle_response`
was never called directly by `SlidingSync`. It was called by
`SlidingSyncListRequestGenerator`! How confusing! `SlidingSync` called
`SlidingSyncListRequestGenerator::handle_response` which was calling
`SlidingSyncList::handle_response`. Now, the flow is more natural:
`SlidingSync` calls `SlidingSyncList::handle_response` and that's it.
Sixth, the `SlidingSyncList::handle_response` is now composed of 2
parts: updating the list itself, and updating the request generator. It
was kind of the case before, but onto two different types.
It was unclear which types were updating `SlidingSyncList`.
For example, `SlidingSyncList::state` was updated by…
`SlidingSyncListRequestGenerator`. Now `SlidingSyncList` is responsible
to update itself, and no one else.
Finally, `SlidingSync` no longer have to deal with
`SlidingSyncListRequestGenerator`. All it has is a set of
`SlidingSyncList`, and that's it!
The tests are still passing, hurray.
In `SlidingSyncListBuilder`, the `ranges`, `set_range`, and `add_range`
methods do not take a `u32` but a `U: Into<UInt>`. That's great!
However, in `SlidingSyncList`, the same methods take a `u32`. It makes
the API inconsistent.
This patch fixes that.
This patch renames `SlidingSyncList.set_ranges` to `ranges` so that
it matches the same terminology of the `SlidingSyncListBuilder` with
`ranges`, `set_range`, `add_range` and `reset_ranges`.
Consistency is important here.
The `SlidingSyncList::new_builder` method creates a new builder from a
`SlidingSyncList`. This method was missing some configurations, like
`full_sync_maximum_numbre_of_rooms_to_fetch`, `send_updates_for_items`,
`filters`, `ranges` and `timeline_limit`.
This patchs adds those missing configurations.
The `SlidingSyncListBuilder` struct has a `state` and a `rooms_list`
fields. Those fields are never updated and no setters or getters exist
to use them. There are just set with defaults, and passed to the final
`SlidingSyncList` type within the `build` method.
If a field is present in a builder type, it means they can be modified,
but it's not the case. So this patch removes them.
Event deduplication is supposed to be handled by the `Timeline`. Sliding
Sync is not responsible to de-duplicate the events, in our model. Having
two different de-duplication logics (one in Sliding Sync, and one in the
Timeline) could create weird situations, and hard to debug issues.
We give it a try by removing the deduplication from Sliding Sync.
The `m.no_olm` withheld code is supposed to be sent out only once for a
given device.
This patch prevents double sending of m.no_olm withheld code by adding
a test that ensures only one group session will send it out.
Previously, the logic for sharing group sessions was mostly correct, but
an edge case was forgotten where two group sessions attempted to share
the withheld code concurrently.
Although there's still a possibility for multiple m.no_olm withheld codes
to be sent out. This may happen during the propagation of the flag
remembering the fact that the `m.no_olm` code has been sent from the
OutboundGroupSession to Device. This scenario is likely unrealistic and
the consequences of sending things twice aren't problematic.
Currently, the tags used for releases of this binding have the format:
"matrix-sdk-crypto-js-v0.1.0-alpha.5". This is inconsistent with tags used for
other parts of the project, which omit the `v` prefix.
The url crate normalizes the string, but during OIDC verification steps,
the issuer verification must be made against the exact string that was
provided.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Like for the state store, the bindings expose the version as an f64
while the web API expects an unsigned integer.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch should ideally be split into multiple smaller ones, but life
is life.
This main purpose of this patch is to fix and to test
`SlidingSyncListRequestGenerator`. This quest has led me to rename
mutiple fields in `SlidingSyncList` and `SlidingSyncListBuilder`, like:
* `rooms_count` becomes `maximum_number_of_rooms`, it's not something
the _client_ counts, but it's a maximum number given by the server,
* `batch_size` becomes `full_sync_batch_size`, so that now, it
emphasizes that it's about full-sync only,
* `limit` becomes `full_sync_maximum_number_of_rooms_to_fetch`, so that
now, it also emphasizes that it's about ful-sync only _and_ what the
limit is about!
This quest has continued with the renaming of the `SlidingSyncMode`
variants. After a discussion with the ElementX team, we've agreed on the
following renamings:
* `Cold` becomes `NotLoaded`,
* `Preload` becomes `Preloaded`,
* `CatchingUp` becomes `PartiallyLoaded`,
* `Live` becomes `FullyLoaded`.
Finally, _le plat de résistance_.
In `SlidingSyncListRequestGenerator`, the `make_request_for_ranges`
has been renamed to `build_request` and no longer takes a `&mut self`
but a simpler `&self`! It didn't make sense to me that something
that make/build a request was modifying `Self`. Because the type of
`SlidingSyncListRequestGenerator::ranges` has changed, all ranges now
have a consistent type (within this module at least). Consequently, this
method no longer need to do a type conversion.
Still on the same type, the `update_state` method is much more
documented, and errors on range bounds (offset by 1) are now all fixed.
The creation of new ranges happens in a new dedicated pure function,
`create_range`. It returns an `Option` because it's possible to not be
able to compute a range (previously, invalid ranges were considered
valid). It's used in the `Iterator` implementation. This `Iterator`
implementation contains a liiiittle bit more code, but at least now
we understand what it does, and it's clear what `range_start` and
`desired_size` we calculate. By the way, the `prefetch_request` method
has been removed: it's not a prefetch, it's a regular request; it was
calculating the range. But now there is `create_range`, and since it's
pure, we can unit test it!
_Pour le dessert_, this patch adds multiple tests. It is now
possible because of the previous refactoring. First off, we test the
`create_range` in many configurations. It's pretty clear to understand,
and since it's core to `SlidingSyncListRequestGenerator`, I'm pretty
happy with how it ends. Second, we test paging-, growing- and selective-
mode with a new macro: `assert_request_and_response`, which allows to
“send” requests, and to “receive” responses. The design of `SlidingSync`
allows to mimic requests and responses, that's great. We don't really
care about the responses here, but we care about the requests' `ranges`,
and the `SlidingSyncList.state` after a response is received. It also
helps to see how ranges behaves when the state is `PartiallyLoaded`
or `FullyLoaded`.
The part of code that converts a list of users to the actual
/keys/query request uses the chunks() method. This method operates on
the slice. Our list/vec of users gets dereferenced into a slice before
we create our chunks. The chunks can't take ownership of the data, which
in turn requires us to clone the user IDs for them to be put into the
request.
Itertools has a chunks() method which operates on an iterator which we
can utilize to remove, not only the clone, but also a collect call.
At the same time, let's make the conversion step a simple functional
mapping and document what's going on.
The Option conveys our intention a bit better compared to the default
value. While nothing bad can happen, since the request IDs map will be
empty by default, it's a bit scary to have a valid sequence number since
an i64 defaults to 0.
This patch updates `SlidingSync.pos` and `.delta_token`. Each field has their
own lock. It's annoying because they must updated at the same time to not be
out-of-sync.
So a new field `SlidingSync.position` of type `SlidingSyncPositionMarkers` is
created. It holds `pos` and `delta_token. This new field is behind a single
`RwLock`.
The comments in the code should explain the general approach, but to give an overview:
We assign a unique sequence number to each device-list-invalidation notice, and keep track
of the sequence number at which each user was most recently invalidated. Then, when we
build a `/keys/query` request, we take note of the current sequence number. Finally, when
we get the response from that `/keys/query` request, we compare that sequence number
with the most-recent-invalidation of each user that is now supposedly up-to-date. All
of that means that we can see if the user has been invalidated *since* we built the
`/keys/query` request, in which case our request may have raced against the new device,
so we do not mark the user as up-to-date.
To make that work, we need to keep track of the sequence number for in-flight `/keys/query`
requests; we do that in the `IdentityManager`. Since there should only ever be at most one
batch of requests in flight, that is relatively easy; however, we also record the request ids
just to check that the application isn't doing something weird.
There is no need to persist the sequence numbers to permanent storage: provided the
`IdentityManager` and `Store` objects have a lifetime at least as long as an in-flight
`/keys/query` request, it is sufficient just to retain the `dirty` bit in permanent storage,
and then, when we reload the ephemeral `Store` cache, to treat everything with a `dirty`
bit as a "new" invalidation.
Fixes: #1386.
With https://github.com/matrix-org/matrix-rust-sdk/pull/1601, it's safer to
cancel a stream at any time, as any `Future` cancellation. This documentation
section of `SlidingSync` is no longer relevant as is, and can be rephrased.
With https://github.com/matrix-org/matrix-rust-sdk/pull/1601,
`TaskHandle::with_callback` is no longer necessary.
This patch updates `TaskHandle` as follows:
1. Before, the `handle` and `callback` fields were not mutually exclusive!
Indeed, the `callback` field was used as an abort mechanism, but _also_ as a
finalizer, i.e. a code that runs _after_ the cancellation of `handle`. Now
that the callback-based solution is no longer used, we can keep one usage for
this field and rename it `finalizer`.
2. The `handle` field is no longer optional, as it will always be set.
3. The `with_callback` method is renamed `new` as it's now the only constructor.
4. The `set_callback` method is renamed `set_finalizer`.
Tadaaa.
Move all `SlidingSync`'s fields into a new `SlidingSyncInner` type, and then
update `SlidingSync` to contain a single `inner: Arc<SlidingSyncInner>` field.
First off, it's much simpler to understand what's behind an `Arc` for
“clonability” of `SlidingSync`, and what's not. We don't need to worry about
adding a new field that is not behind an `Arc` for a specific reason or
anything. The pattern is clear now.
Second, this is required for next commits.
Before this patch, `SlidingSync::sync` was returning a callback-based
`TaskHandle`. It was waiting on the “stream loop” to finish (since it's a long-
poll, it means waiting 2*30s, cf. our code), before checking an atomic flag has
some value to decide whether it was time to leave the loop or not. So when the
user is cancelling this `TaskHandle`, a response (if any) was always handled.
But in the meantime, it was possible to start a new `sync`, and it seems like
it creates bugs.
After this patch, `SlidingSync::sync` now returns a handle-based `TaskHandle`.
It means that cancelling it will cancel the “stream loop” immediately. If
no response was in-flight from the server, that's perfect, no problem. If a
response was in-flight, the inner `pos` of the `SlidingSync` instance won't be
updated as the response won't be handled. So the server will re-send the same
response with the next sync request.
I guess it's better this way. Thoughts?
This patch adds a new feature.
Each call to `SlidingSync::stream` generates a unique `stream_id`. This
`stream_id` is passed to requests as a `txn_id` field. Returned responses must
hold the same `txn_id`, otherwise it means `stream` has received unexpected
responses from another `stream` or something else.
So far, when the `txn_id` field of the response and the `stream_id` don't
match, a log with the `ERROR` level is raised. Should we do more, like ignoring
the response entirely? Not sure yet.
The IndexedDB API expects an unsigned integer for the version.
The web-sys bindings expose it as a f64 so versions like 1.1 and 1.2
were used. The were converted back to uint in JS which means the version
was always 1.0 and no real upgrade was processed
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Simplify handling of sliding sync extensions
This patch simplifies the logic for handling sliding sync extensions.
These extensions are sticky, meaning that once enabled in one request,
they do not need to be repeatedly enabled for subsequent requests.
However, the previous logic tried to accommodate the case where the
extensions were initially disabled to reduce the size and processing
time of the initial response. This resulted in complex and error-prone
code.
With this patch, we have removed support for disabling to-device events
and streamlined the handling of sync extensions.
Co-authored-by: Jonas Platte <jplatte@matrix.org>
You can't update a GHA cache once you create it, so if we use the same cache
for each job in a matrix build, then we end up populating it for the first job
that completes, so any slower jobs don't get their dependencies cached.
On the other hand, if we create 20 500MB cache items on each build, we're going
to exhaust the cache storage as soon as we do a build. So, instead, let's just
do the caching for the main branch, and hope that other branches can still
benefit from it.
* Replace custom cancellation action with `concurrency`
* Improve step names
... so don't have three steps with the same name
* Bump version of checkout action
checkout@v2 uses an old version of nodejs, which is deprecated.
To declare the tests, the macro is still used but the content of the
tests is moved in the StateStoreIntegrationTests trait.
Allows to get the proper line reported when a panic occurs,
and have linting, formatting and IDE suggestions.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Currently, the cache of the xtask binary isn't working terribly well:
* we seem to build it on each run anyway, presumably because we don't cache any of the intermediate build
artifacts. Running the binary directly rather than indirecting via "cargo" prevents this.
* There is no sharing of the cache between the "rust" and "bindings" CI, because we use different cache keys.
This PR addresses both problems, and hopefully speeds up CI a bit as a result.
The Sliding Sync server might not send some parts of the response, because they
were sent before and the server wants to save bandwidth. So let's update values
only when they exist.
Running the same example twice
borks encryption.
Instead point to an example that shows how to do it.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch cleans up the `SlidingSync::stream` method. It renames variables,
improves log messages etc.
This patch also creates a new `MAXIMUM_SLIDING_SYNC_SESSION_EXPIRATION`
constant. This value was previously hardcoded and lost in the code, now it's
easier to spot it for further updates.
This patch finally renames the `failure_count` field to `reset_counter`,
because it doesn't count the number of failure, but the number of
`ErrorKind::UnknownPos` exactly, i.e. the number of times we reset the
`SlidingSync` state.
This patch cleans up the `SlidingSync::sync_once` method by renaming variables,
adding more comments, simplifying the code by reducing the number of variables
etc.
This patch also changes `futures_util::join!` to `futures_util::future::join`.
It does the same but the macro needs the `async-await-macros` feature to be
turned on, while the second works without any features.
Finally, this patch improves the log messages by making them more clear for a
new reader.
I admit this patch is quite tricky. Please try to follow me.
So, first off, in `SlidingSyncRoom::new`, we were clearing the timeline,
because somehow it exists twice in memory at this step.
Which led me to understand how `SlidingSync::handle_response` was working.
I've clarified how this part of the code works. We are dealing with 2 kind of
responses for a specific reason: `SyncResponse` and `v4::Response`, now it's
documented and I hope it's clearer.
Then, I notice that we were passing a clone of the entire sliding sync
response (`v4::Response`) to `Client::process_sliding_sync`. I thought it was
suboptimal, so I've updated the code to take a reference. It led me to update
`BaseClient::process_sliding_sync`. It was a little bit tricky, but I reckon we
have less clones now than before.
And now, back to `SlidingSync::handle_response`, I was able to compute the
`timeline` correctly by draining it from the `v4::Response`, or by moving it
from `SyncResponse`. So it's no longer necessary to have this clearing code
inside `SlidingSyncRoom::new`. Honestly it has nothing to do at this place
before.
To conclude: We have cleaner code, and less clones.
What thing I reckon could be optimized, is that the entire `timeline`
(`Vec<TimelineEvent>`) is cloned to be passed to `Client::handle_timeline`. So
this timeline exists in 2 places: in Sliding Sync, and somewhere else. I don't
believe it's a problem now, that's how it works, but we must be aware of that.
The `Client::login_custom` allows to login by using a custom login
method. In particular, it is possible to login to Synapse which supports
JWT authentication.
Signed-off-by: boxdot <d@zerovolt.org>
First off, it's not necessary for `SlidingSyncRoom::update` to take a reference
to `room_data: v4::SlidingSyncRoom`. When `update` is called, the iterator owns
its items.
Second, by taking ownership of `room_data`, we no longer need to clone all the
data we need to assign to `self.inner`.
First off, `SlidingSyncRoom.from` doesn't need to be visible to the entire
crate, only to `crate::sliding_sync.
Second, it's a constructor, so let's call it `new`.
1. Put `FrozenSlidingSyncRoom` at the bottom of the module.
2. Put merge 2 `impl SlidingSyncRoom` together.
3. Remove the `AliveRoomTimeline` type alias.
4. Improve the documentation.
So far, the `SlidingSync.pos` field was public to the crate. In order to avoid
breaking the internal state of this type, its visibility is now private.
However, we need to be able to change the value when testing the
`SlidingSync` type itself. To achieve that, this patch removes the old
`force_sliding_sync_pos` function, and implements 2 new functions: `set_pos`
and `pos` directly on `SlidingSync` only when `#[cfg(any(test, feature ="testing"))]`.
Previously, it was possible for us to use invalid indices when:
- We retried decrypting multiple events at once
- One of them (not the last) was an edit or reaction
This lead to a situation where we would remove the UTD item once
decryption for it was successfully retried, but not account for the
resulting index shift for all later timeline items.
Before this patch, `SlidingSyncView` has the following fields that were public:
`state`, `rooms_list` and `rooms_count`. Since they are `Mutable`, they can be
changed from the outside, and then will break the internal state of the view.
This problem is mentioned in https://github.com/matrix-org/matrix-rust-sdk/
issues/1474.
This patch solves this by making them prviate. Phew. That was simple!
But wait, we have a problem now. `matrix-sdk-ffi` was relying on them. So
this patch adds “helpers” methods on `SlidingSyncView`, like `state_stream`,
`rooms_list`, `rooms_list_stream`, `rooms_count` and `rooms_count_stream`.
Let's add new ones when it's necessary.
In https://github.com/matrix-org/matrix-rust-sdk/pull/1478, I've introduced
a regression. The FrozenSlidingSyncRoom.timeline` field has been renamed to
`timeline_queue`.
I didn't notice that this type can be serialized and deserialized. That's
actually a breaking change because this type is stored somewhere once
serialized. So with #1478, it was impossible to deserialize them!
This patch fixes that. It also adds a test to ensure the serialization form of
`FrozenSlidingSyncRoom` doesn't change.
This patch allows an `OlmMachine` to use a different store than Sled, like SQLite.
A new enum is introduced: `StoreType` which 2 variants: `Sled` (the default), and `Sqlite`.
The `OlmMachine.initialize` constructor now takes a new optional `store_type:
Option<StoreType>` argument. If no value is passed, the default `StoreType`
variant will be used (as mentioned: `StoreType.Sled`).
The code has been rewritten a little bit to make the type system happy without
introducing too much type indirections.
This patch finally adds a parameterized tests that exhaustively test: no store
type, `Sled` and `Sqlite`.
Building the crypto-js bindings in release mode is very slow and not really
necessary for local development.
`--release` is the default, so there is no need to specify it
explicitly. Instead, allow `wasm-pack` args to be specified by an env var, and
add a `build:debug` npm script which will build in debug mode.
We already have support for proper `ToDeviceRequest` objects in
`outgoing_requests`, so let's use it in `share_room_key` as well. It's much
easier for the application to work with the proper object than an untyped json
blob.
In the code `if let Some(global_data) = account_data.as_ref().map(|a|
&a.global)`, the `map` is not purely necessary. This patch simplifies the code
to remove the `map` and read the `global` field later on.
By using `Default::default` for the parameter value, we don't know which type
we are passing to the function. Hence the useful comment `room_info.ephemeral`.
This patch changes this to `Ephemeral::default`, so that we now know what we
pass without the need of a comment.
Javascript's `console.trace` is not a good equivalent for Rust's
`Level::TRACE`.
In particular:
* `console.trace` causes a stacktrace to be written with each message
* Under nodejs, `console.trace` writes the message to `stderr`, while
`console.debug` writes to stdout
`console.trace` is therefore somewhat analogous to `console.error`.
Since we already include the log level in the message, we might as well just
send trace messages to `console.debug` rather than `console.trace`.
This is useful for platforms which might want to avoid the cost of
password based key-derivation and have the ability to securely store an
encryption key.
Co-authored-by: Denis Kasak <dkasak@termina.org.uk>
Contrary to other tables that use their own name to encode their keys,
the `ROOM_INFO` table uses the `ROOM` table name to encode its keys.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
`wasm_timer` doesn't work outside the browser, which prevents it being used in
tests.
We can replace it with `gloo-timers` wich uses the standard Javascript
`setTimeout`.
This makes it easier to implement it. However, using a different error
type than CryptoStoreError is a non-trivial change, so left for the
coming commits for all stores except the memory store.
The key encoding in key-value stores is backwards-compatible so
old receipts are counted as unthreaded receipts.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
A fix for (de)serialization of redacted state events in Ruma made
old events undeserializable.
Bump the DB version.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
A fix for (de)serialization of redacted state events in Ruma made
old events undeserializable.
Bump the DB version.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This implementation is wrong in the sense of its semantics is not about
dereferencing a thin pointer to something, but just to give access to one
specific field of the entire structure. That's not how `Deref` is supposed to
be used.
Moreover, it creates conflict between the `SlidingSyncRoom.timeline` field,
and `SlidingSyncRoom.inner.timeline` field, which both exist, but not for the
same purposes. It creates confusion in the code.
Finally, it's better to expose proper getters to the outside world, so that we
control _and_ test _and_ know exactly what API we provide.
`Timeline::with_fully_read_tracking` was public although all public
Timeline constructors enable it by default.
It is also meant to be called when constructing the timeline so
using a builder pattern makes sense.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
`SlidingSyncView` exposes a `room` field. Problem: It's not updated
correctly and leads to error. The canonical way to get a room is by using
`SlidingSync::get_room`. This patch thus removes `SlidingSyncView.rooms`
entirely.
Before this patch, `SlidingSync::get_room` was taking ownership of an
`OwnedRoomId`, to only use it as a reference. Thus, calling this method was
creating useless clones.
This patch updates `SlidingSync::get_room` to receive a reference to
`OwnedRoomId`.
Note about "Write-Ahead Log" (WAL) mode: The SQLite WAL mode has a
bunch of advantages that are quite nice to have:
1. WAL is significantly faster in most scenarios.
2. WAL provides more concurrency as readers do not block writers and a
writer does not block readers. Reading and writing can proceed
concurrently.
3. Disk I/O operations tends to be more sequential using WAL.
4. WAL uses many fewer fsync() operations and is thus less vulnerable
to problems on systems where the fsync() system call is broken.
The downsides of WAL mode don't really affect us. So let's turn it on.
More info: https://www.sqlite.org/wal.html
Co-authored-by: Jonas Platte <jplatte@matrix.org>
Co-authored-by: Damir Jelić <poljar@termina.org.uk>
When we store an Olm session in the database, we encode the sender key and
session ID using the store cipher. That means that we also need to encode them
when we look them up, otherwise we won't find them.
This patch is trying to resolve the following issue:
When a local event is sent to the server, in case of an error, the
`Timeline::send` method just returned the error but it wasn't reflected inside
the `LocalEventTimelineItem` type itself. It's easy to loss this information.
So now, there is a new enum: `LocalEventTimelineItemSendState` with 3 states:
1. `NotSentYet`, when the local event has not been sent yet, it's theorically
the initial state,
2. `SendingFailed`, when the local event has been sent but it's failed!
3. `Sent`, when the local event has been sent successfully.
Therefore, the `LocalEventTimelineItem` struct has a new field: `send_state`.
The send state is managed by its `with_event_id` method which now receives an
`Option<OwnedEventId>` (prev. `OwnedEventId` directly):
* If it's `Some(_)`, then it means we got a successful response from the server
after the sending, so the send state is set to `Sent`,
* If it's `None`, then it means we got an error response from the server, so
the send state is set to `SentFailed`.
(A small change: The method `TimelineInner::add_event_id` has been renamed
`TimelineInner::update_event_id_of_local_event`.)
The `Timeline::send` still returns a `Result`, but the server's response is
passed to a new method: `TimelinerInner::handle_local_event_send_response`
(it's logically named after the `handle_local_event` method), which is
responsible to call `TimelineInner:update_event_id_of_local_event` (which was
previously called directly by `Timeline::send`).
And `TimelineInner::update_event_id_of_local_event` was already calling
`LocalEventTimelineItem::with_event_id`. So everything works like a charm here.
The local send state is closely managed by `LocalEventTimelineItem`, I hope it
will avoid state breakage as much as possible.
The sliding sync logic has another room type called `SlidingSyncRoom`.
This room type stores a limited amount of events in something called
`AliveRoomTimeline` in a in-memory cache. This room also gets persisted
to the store via another room type called `FrozenSyncRoom`.
These types are used when clients restore their view of the room, i.e.
when the user enters the room to look at messages.
When the client enters a room, a `Timeline` object will be created, but
it will be initially populated with events coming from
`AliveRoomTimeline`.
In a hot potato contest of who should be responsible to decrypt injected
events, everybody claims to be allergic to potatoes.
This patch modifies the `Timeline` constructor that populates the
timeline with events from the `AliveTimeline` to retry decryption on the
events that the `AliveTimeline` pushes into the `Timeline`.
Note that, because the `AliveRoomTimeline` never replaces encrypted
events with decrypted ones, every time the client enters/exits the room
events well get re-decrypted.
This is an omission from the spec where the subkeys are supposed to be
signed by the master key, but signing the master key wasn't mentioned.
This is still fine, since we always treated the master key and their
subkeys as a whole and never accepted one without the others.
To create an event next to the server, the event must be accompanied by
a transaction ID. When the server receives the event creation request, it
responds with a new event ID (which we will name `event_id_1`). Later on, when
a sync is done to the server, the server may respond with the transaction ID
and a new event ID (which we will name `event_id_2`).
The transaction ID is used to update the state of the local event. The event
ID received from the sync may ideally be the same as the one received by the
creation request's response.
Knowing that…
`EventTimelineItem` had a field: `event_id: Option<OwnedEventId>`. It was
`Some(event_id)` where `event_id` is the event ID responded by the server when
an event was created, i.e. it's `event_id_1`; otherwise it was `None`. This is
never `event_id_2`. Why?
Because `EventTimelineItem` has an other field: `key: TimelineKey`, which
represents the following states:
* `TimelineKey::TransactionId(OwnedTransactionId)`,
* `TimelineKey::EventId(OwnedEventId)`, where the event ID is `event_id_2`!
So it becomes obvious that `event_id_1` should be part of `TimelineKey`, not
`EventTimelineItem`. `TimelineKey` is where this transaction ID and event ID
logic is managed.
This patch updates `TimelineyKey::TransactionId` to be defined as:
* `TimelineKey::TransactionId { txn_id: OwnedTransactionId, event_id: Option<OwnedEventId> }`.
Why an `Option`? Because before receiving `event_id_1`, we may still want to
create a `TimelineKey`, the API must reflect that state.
So basically, `EventTimelineItem.event_id` is moved inside
`TimelineyKey::TransactionId`.
… rather than prepended, or added at a specific point as replacement for
an existing timeline item.
Previously, we would log lots of useless tracing events when retrying to
decrypt UTD items.
This patch fixes https://github.com/matrix-org/matrix-rust-sdk/issues/1379/.
This is similar to https://github.com/matrix-org/matrix-rust-sdk/pull/1197/.
We need to close the `OlmMachine`. Problem, `napi-rs` doesn't allow methods to
take ownership of `self`, so the following code:
```rs
fn close(self) {}
````
isn't allowed. There an [`ObjectFinalize`](https://docs.rs/napi/latest/napi/bindgen_prelude/trait.ObjectFinalize.html)
trait, but it's only called when the JS value is being collected by the GC.
It's not possible to force call `finalize` here.
This patch then introduces a new type:
```rs
enum OlmMachineInner {
Opened(matrix_sdk_crypto::OlmMachine),
Closed
}
```
that derefs to `matrix_sdk_crypto::OlmMachine` when its variant is `Opened`,
otherwise it panics. Calling the new `OlmMachine::close` method will change the
inner state from `Opened` to `Closed`.
Ideally, I wanted to throw an error instead of panicking, but `napi_env`
(used to build an `Env`, used to throw an error) is neither `Sync` nor `Send`.
Honestly, my knowledge of NodeJS and NAPI is too weak to implement `Sync`
and `Send` for `*mut napi_env__` safely. Especially because it can be executed
from various threads within `async fn` functions, that are driven by Tokio in
`napi-rs`. So, yeah, for the moment, it panics!
This moves the caches we have for tracked users out of the Sled, and
IndexedDB, CryptoStore into the Store struct. The Store struct is a
wrapper for the CryptoStore trait, so this is the natural place where
these caches should live.
Co-authored-by: Jonas Platte <jplatte@matrix.org>
Before we'd push one empty entry at a time, which lead to a lot of vecdiff updates pushed. With this
we now only push one event after the entire operation set has been applied on the fresh
or cold view.
A mismatch between the recorded Ed25519 and Curve25519 keys of the room
key and the identity keys of a Device used to be just a warning.
Considering that many clients will aggressively try to hide E2EE related
warnings let's upgrade this clear error into a full blown decryption
failure.
Co-authored-by: Denis Kasak <dkasak@termina.org.uk>
We might have a lot of tracked users, clients will usually have 1-10
thousand tracked users. This might slow down client startup if there are
so many tracked users.
We don't need the tracked users until we sync or try to send a message,
so load them lazily when they are first needed.
The conventional commit specification recommends the "docs" type for
documentation related changes. This also matches the initial usage of
this type in our commit history.
While the default git-cliff config does specify a regex "^doc", where I
suspect our types have been taken from, this regex actually allows "doc"
as well as "docs" for the type.
Nevertheless, let's specify that we're using the one recommended in the
spec.
I had no idea what a tracked user was, so I've attempted to clarify this
somewhat for future readers.
Co-authored-by: Damir Jelić <poljar@termina.org.uk>
Instead of putting all request fields inside a single `body` JSON-
encoded string field, there is now more types. There is less need to call
`JSON.parse`, and the type system will be able to catch more things.
For example, `ToDeviceRequest` now has 2 more fields: `event_type` and
`txn_id`.
This solves the issue of correlating different log lines that are talking
about the same request/response cycle. Plaintext logs would show this as
a flat structure, while something like Jaeger would show you a nice
tree. This allows you to reconstruct the tree in plaintext logs.
Various smaller fixes on sliding-sync for elem-x:
- Infrastructure and smoke test only for integration testing sliding-sync
- Expose Pop and Clear on VecDiff
- Expose adding views during sliding sync live runtime incl integration tests
- bug fix on replacing only in-window-items in the insert-code (was leading to a wrong list setting), including integration test
Required when `PaginationOptions` must implement `Send`,
e.g. with `tokio::runtime::Runtime::spawn`.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
When updating a device after we receive new device keys from a
/keys/query call we implicitly check that the user and device id match
since we need to fetch the device using the mentioned ids.
This moves the check into the update method as well to make it more
explicit. We also prevent upgrades of the Ed25519 key since we don't
support any different key type anyways.
The sliding sync proxy seems to have started to omit the e2ee extension
completely if no changes happened to the one-time key counts.
Our sync response processing assumed that, if there are some to-device
events that need to be passed to the OlmMachine, then we'll have some
data in the e2ee extension of the response as well. In other words, it
wouldn't pass in the to-device events to the OlmMachine if the e2ee
field of the sync response is `None`.
Since the assumption isn't true, we're going to pass in default values
for the one-time key counts when the e2ee extension field of the
response is `None`.
This is mainly done so we create a span with the name of the function,
since we can filter logs by the name of the span we're now able to
enable logs for this method only.
Previously, when we found a since-token in the frozen state, we'd always activate the to-device extensions
regardless of whether the user had activated it in their builder or didn't. With this change we are only
setting the since parameter from cache if the user actually activated the to-device extension.
- Don't actually fire off a request when the top of the timeline has
already been reached
- Allow making multiple requests without removing the loading indicator
in between
Index ranges are inclusive, but our loop would stop one short. This particuarly
tricky when the selective view is moved, as we didn't properly invalidate all items.
It's not a hard-and-fast-rule that Javascript exceptions should be `Error`
instances, but it's certainly a convention, and one that is going to reduce
surprises if we follow.
Add a second full-sync-up mode to sliding sync. Previously - and still the default for backwards compatibility, but now named `PagingFullSync` - was to page through the list by the page-size, de-validating the previous page of items. The newly added `GrowingFullSync` instead grows the window by the given number `batch_size` per request, starting and keeping it from `0`. In the latter we might be pushing more data over the connection and are slightly slower, but the top always stays active and thus reactive to changes.
Furthermore the developer can now configure an optional maximum number to grow/paginate the full-sync up to (so stopping before actually having reached `count` whatever is smaller). This is already exposed via FFI, too.
- [x] add new full-sync-mode
- [x] add limited sync-up mode (similar to full sync but only to a limit `n`)
- [x] implement sync-up in jack-in
Sliding Sync updates
- expose timeline listener via FFI on slidingsyncroom, too - returning a stoppable spawn
- connect room timeline and sliding-sync-room timeline together
- increase HTTP timeout on jack-in
- add jack-in support for actual timeline items.
We automatically request room keys to be forwarded from our other trusted devices if a decryption failure happens because the room key is missing.
This patch introduces automatic room key requests for decryption failures if the room key is available but has been ratcheted forward. In other words, we will now request a better version of the given room key automatically as well.
Co-authored-by: Damir Jelić <poljar@termina.org.uk>
`open_with_db` also has the passphrase parameter without mentioning it
in its name. The old name also sounded like the passphrase was required
when it's actually optional.
Clarify that the JSON-encoded `toDeviceEvents` passed to
`OlmMachine.receiveSyncChanges` must be the list of events themselves,
instead of a wrapper object that contains the event list.
Signed-off-by: Andrew Ferrazzutti <andrewf@element.io>
Tokens are not necessary for the restoration of the crypto/store, only
the meta. Required for OpenID Connect support where we need the tokens to get the
meta.
When the verification support was initially bound, Uniffi did not
support objects (the OlmMachine) returning other objects (our
verification objects). Instead we converted all the verification objects
into pure data structs which the other side had to poll for changes.
Now that Uniffi supports returning objects we can refactor this and make
the API on the other side the same as on the Rust side.
With these changes, the user of sliding sync can configure the SlidingSync-Builder to store and recover the state from storage. For that they should call cold_cache("my-sliding-sync-key") with their preferred storage location during setup time on SlidingSyncBuilder. If a cached version is found under that key, the internal state of the rooms-list as well as each view found (identified by their name) will be set from storage immediately - allowing the user to show the last cached state. The Builder then also uses the same key to store its latest state after every update received from remote.
👉 Note on room list:
As we store a disconnected state, we are saving all RoomList entries as either Empty or Invalidated. This allows for an easier updating when the server comes back with results, as we don't track the ranges in cache - in the view of the server, we haven't seen anything yet.
👉 Note on timeline events:
This does store all existing timeline events - initial and seen during the session (as we keep track of them right now) - and will recover that state. However, as we can't be sure wether we have gaps in the timeline, the timeline items are reset upon receiving updates for them from the server.
👉 Note on (full-sync-)view:
While we are caching the server returned results (view list, room count and room info) as is, we are not caching the internal state (whether we are catching up) nor the ranges (as they are likely to be out of date) - those will be acting the same way you configured them, just with preloaded results. So for a full-sync-view, it will still do requests in batches and replace the corresponding set of rooms. Which could mean you might see the same room appear twice, though the cached one would be marked as invalidated. This might be problematic if the list the UI shows is longer than the batches fetched, but would be resolved quickly when catching up.
👉 On recovery:
When the sliding sync receives a M_UNKNOWN_POS, indicating the server has expired our session, sliding sync now transparently retries with up to three times to restart the sliding sync with full set of extensions and the latest views at their existing windows, the current room state is held. For full sync this starts a new sync-up with the existing room list staying intact. This also works from the offline at start.
- matrix_sdk_base::sync::SyncResponse is the internal representation that
we can update to account for sliding sync
- matrix_sdk::sync::SyncResponse is `/v3/`-specific and should not change
* Move swift build scripts into xtask (#1201)
* fix(ffi): use target_path from `cargo metadata` rather than guessing
* ci(ffi): install necessary target arch for build-framework test
* feat(xtask): copy to target without rsync.
As it appears, the first sync for a room might not include the
encryption state, so we cannot set the encryption state to be synced on
incoming syncs. That way, we always fetch the encryption state manually.
* split logger into platform specific implementations
* logging support for android
* use platform independent wrapper for uniffi:exports
* activate colors for ansi-terminals
It turns out that Google Chrome refuses to initialise the wasm via the
synchronous `WebAssembly.Module` constructor, complaining that it is too
big. To be fair, it has a point.
Anyway, that means we need to provide a way to load the wasm
asynchronously. So, we introduce an `initAsync()` function which applications
can call before they do anything else, to load the wasm in the background.
If the app *doesn't* call `initAsync()`, then we load the wasm synchronously
the first time a function that accesses the wasm is called.
This new `OlmMachine.close` forces to drop/close the `OlmMachine` without
waiting on the JavaScript garbage collector to collect it.
`wasm-bindgen` generates the following JS glue code:
```js
close() {
const ptr = this.__destroy_into_raw();
wasm.olmachine_close(ptr);
}
```
And, `__destroy_into_raw` looks like this:
```js
__destroy_into_raw() {
const ptr = this.ptr;
this.ptr = 0;
OlmMachineFinalization.unregister(this);
return ptr;
}
```
It unregisters itself from the `FinalizationRegistry` correctly. We are
protected from a double-free.
Surprisingly, `indexed_db_futures::IdbDatabase` is not closed when dropped.
Hopefully, there is a [`IdbDatabase::close(&self)`][close] method, which calls
`web_sys::IdbDatabase.close`, aka [`IDBDatabase.close`][websys-close], so let's
use it!
`IDBDatabase.close` returns immediately and closes the connection in a separate
thread. The connection isn't actually closed until all transactions created
using this connection are complete. No new transactions can be created for this
connection once this method is called. Methods that create transactions throw
an exception if a closing operation is pending.
[close]: https://github.com/Alorel/rust-indexed-db/blob/8c106eb418aecdba2f3fde80d91a4673a875fdf6/src/idb_database.rs#L73-L77
[websys-close]: https://developer.mozilla.org/en-US/docs/Web/API/IDBDatabase/close
I have tried for hours and have concluded it is probably not possible
even with GATs to have RawEvent borrow its inner value.
It is also not a commonly-used feature, so removing the unnecessary
clones is not that important.
This patch adds a couple more states to the SAS verification state
machine. Namely we add the following states:
* KeySent
* KeysExchanged
This prevents users from confirming that the short auth string matches
before they have sent out a m.key.verification.key event, which is
necessary for the other side to present the short auth string.
The KeysExchanged state functionally replaces the KeyReceived state.
Meaning that the short auth string can only presented once we
transitioned into the KeysExchanged state.
We can transition into the KeysExchanged state from the KeyReceived
state or from the KeySent state, depending on if we first receive a
m.key.verification.key event or if we first send out our own
m.key.verification.key event.
This means that, if the transition into the KeysExchanged state happens
through the KeySent state, users won't be able to tell if the transition
happened. In other words, listening to `m.key.verification` events
doesn't work anymore, users will need to use the new `Sas::changes()`
API to listen to a stream of state changes.
`wasm_bindgen::JsValue::(from|to)_serde` now emit warnings because of a
circular dependency. To solve this problem, this patch now uses `gloo-utils`,
see https://rustwasm.github.io/wasm-bindgen/reference/arbitrary-data-with-serde.html#an-alternative-approach---using-json.
Ideally, we would like to use `serde_wasm_bindgen` but the behaviour isn't
the same. `gloo-utils` serializes and deserialized through JSON, whilst
`serde_wasm_bindgen` manipulates the `JsValue` directly which can be better or
worst dependending of the case. This patch conserves the JSON approach as it
was the previous and tested one.
By asking `wasm-bindgen` to generate JS glue code for `WeakRef` support,
it removes the need to call `free` manually to free objects, thus we reduce
potential memory leaks inside Rust. See https:// rustwasm.github.io/docs/wasm-
bindgen/reference/weak-references.html to learn more.
It uses [`FinalizationRegistry`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/FinalizationRegistry)
under the hood, I reckon it's quite common and fits into Matrix's clients needs
in terms of browser support.
While technically a type constructor can return a `Promise`, it's not
considered as idiomatic JavaScript to do that. We are changing `new OlmMachine`
to `OlmMachine.initialize`. The rest of the code is strictly the same.
Add general extension framework for sliding sync, implementing the e2ee, to-device and account-data extensions as per existing proxy implementation. Add a new (ffi exposed) function to use activate the extensions.
Also extends jack-in to have permanent login and storage support now, rather than posting an access token and expose messages inside the sliding-sync layer to actually use the decrypted messages if given. Contains a lot of fixes around these aspects, too, like uploading any remaining messages from the olm-machine on every sliding-sync-request or processing even if no room data is present (which can happen now as processing only extensions might takes place).
This patch adds a way for users to listen to changes in the state of a
SAS verification.
This makes it much more pleasant to go through the verification flow and
incidentally easier to document it.
Now that we're not scoping the room keys by the Curve25519 sender key
we're opening the door of multiple devices trying to insert the same
room key into our store.
This patch changes our logic so we only store room keys from an
m.room_key event if we don't have one already or if the new key is
a better version of the one we already have.
This mostly assumes that the first room key with a given session id
is coming from the creator of the room key.
It was very confusing that matrix_sdk::store::make_store_config's first
argument was either a path or a database name depending on the build
configuration. ClientBuilder::{sled_store, indexeddb_store} are also
easier to use.
By default, the new `experimental-nodejs` feature is
disabled. However, when enabled, it uses our own fork of
`indexed_db_futures` that make it work on Node.js, and so this entire
`matrix-sdk-indexebdb` crate.
The crypto crate consumes to-device events, if needed decrypts them, and
in the end reserializes while zeroizing secrets.
It needs to do this reserialization cycle as loslessly as possible, this
is why we have a bunch of BTreeMaps lying in events and contents that
capture unknown fields.
The Deserialize implementation of ToDeviceEvent<C> put the event type
into this BTreeMap but the Serialize implementation figures it out from
the EventType trait of the content.
This patch simplifies things by putting the event type into
ToDeviceEvent<C>, this also means that we can just derive the Serialize
implementation for ToDeviceEvent<C>.
Inspired by the changes in #1013 I was thinking about the use case for `sync_*` and how we handle error cases. Most notably while we give the callback the option to stop the loop, we don't really give an indication to the outside, how to interpret that cancellation: was there a failure? should we restart?
Take e.g. a connectivity issue on the wire, we'd constantly loop and just `warn`, what you might or might not see. Even if you handle that in the `sync_with_result_callback` and thus break the loop, the outer caller now still doesn't know whether everything is honky dory or whether they should restart.
This Changes reworks that area by having all the `sync` return `Result<(), Error>`, where `()` means it was ended by the inner callback (which in `sync()` never occurs) or `Error` is the error either the inner `result_callback` found or the that was coming from the `send` in the first place. Thus allowing us to e.g. back down to sync as it was a dead wire or restart it if there was only a temporary problem. Making all that a just a bit more "rust-y".
… for matrix-sdk, matrix-sdk-base, matrix-sdk-common and matrix-sdk-crypto.
matrix-sdk-indexeddb as well as the JS bindings and wasm_command_bot are
left as-is because they will likely always require JS.
* starting with jack-in
* starting by flying tui
* connecting to real server, showing info
* add .env to gitignore
* infrastructure for tests
* display loading time for syncv2
* minor design updates
* initial sync
* finalise first edition of sliding sync
* directly link to sliding sync and show rooms list
* nicer UI, toggle logs
* passing through sliding sync homeserver
* separate syncs and disable v2 autostart
* selecting rooms
* nicer view
* configurable batches and more default needed events
* selecting rooms
* calculate and show status info per room
* precalculated room stats
* restructure code to allow for cancellation of streams
* finish up merge updates
* fix calculation error in room list len
* cleaning up system flow
* fixing sync up
* new multi-view API
* move sliding sync in separate module
* fixing format
* adding and clearning views
* expose filters and timeline limits
* renamed
* adding room subscriptions to sliding sync
* update summary
* live fetching and subscriptions in jack-in
* subscribe to selected room
* starting to switch to tuireal - using example
* status bar and first linkup
* re-adding rooms
* implementing port for customised update event
* showing details and timeline
* fix formatting
* cleaner UI, updating details quickly
* make it green
* implement other Ops
* proper handling of invalidation
* saving sliding sync results to db, too
* saving new prev_batch field if given
* split events and timeline
* cleaning up
* live updates
* upgrading to latest ruma and matrix-sdk
* Update tui-logger to fix the broken build
* fixing latest ruma sliding-sync-branch updates
* feat: first set of ffi sliding sync
* expose sliding sync via FFI
* implement un-/subscribe
* implement view state updates
* updating to latest JSON format and ruma update
* implementing room selecting for new data model
* fixing room selection
* fixing feature flag requirements for sliding-sync
* fixing style, clippy and docs
* style(sliding-sync): fixing rust format styles
* fix(ffi): fixing sliding sync merge failure
* fix(jack-in): update jack-in to latest ruma
* fix(sliding-sync): need to have a version set before polling proxy
* expose sliding sync builder over ffi
* add SlidingSyncViewBuilder
* add forgotten changes on sdk itself
* new file logger feature on jack for deeper logging
* fix(http-client): log the raw request we are sending out
* feat(sliding-sync): better logging
* fix(sliding-sync): switch to full-listen after reaching live state in full-sync and make sure we replace the correct entries
* feat(ffi): expose sliding sync view ranges
* fix(ffi): fixing sliding sync start_sync loop to actually loop
* feat(sliding-sync): allow lookup of room data
* feat(sliding-sync-ffi): fetching name of room
* feat(ffi): expose unread_notifications of rooms
* feat(ffi): stoppable spawn for sliding sync
* fix(ffi): expose has_unread_notifications on room
* feat(sliding-sync): latest room message
* fix(sliding-sync): update to latest ruma layout
* doc(sliding-sync): adding docs to builder fns
* feat(sliding-sync): extra signal on the view to inform about general updates
* fix(sync-v4-ffi): expose new callbacks via ffi
* fix(sliding-sync): reduce default required states to make things faster
* fix(sliding-sync): fix build params
* feat(jack-in): extended instrumentation
* fix(sliding-sync): unbreak faulty feature-gate
* fix(sliding-sync-ffi): mut where mut is due
* fix(sdk): allow separate homeserver on every request to unbreak using send on client while in sliding sync on a proxy
* fix(jack-in): update to latest dependencies, that work
* feat(ffi): helper to patch sliding sync room into regular room
* style(jack-in): cargo fmt
* fix(sliding-sync): Update to latest ruma changes
* fix(sliding-sync): fix missing FFI updates to latest ruma
* feat(sliding-sync)!: simplify stream cancellation, cancel ffi sync if already existing
* fix: timeline backwards pagination must work without synctoken
* fix(sliding-sync): clarify order of messaes in alive TL; pick correct last item
* fix: update view delegate api for clarity
* style(jack-in): fix cargo warnings
* feat(sliding-sync): update room details
* fix(sliding-sync): only update room info selectively when given
* fix(sliding-sync-ffi): convert and store counts as u32, check against 0 for has notificaitons
* style: cargo fmt, file endings and a few other minor style fixes
* docs(jack-in): improving CLI and Readme
* feat(sliding-sync): allow setting of required event_states on viewbuilder
* style(sliding-sync): docs and minor fixes
* style(sliding-sync): various clippy fixes
* style(jack-in): clippy suggestions applied
* fix(sliding-sync): Delegate becomes observer
* test(sdk): adding test for request config
* docs: Fixing copyright header
* style(ffi): Nicer naming of params for observer
* fix(ffi): sliding sync is not optional for now
* fix(sdk): remove superflusous tracing instrumentation
* fix(sdk): use structured logging
* fix(jack-in): removed unneded log import
* fix(jack-in): use server_name rather than deprecated user_id on ClientBuilder
* style: typo and clippy
* style(sliding-sync): clippy and formatting
* fix(sliding-sync): cleaning up minor syntax issues
* fix: remove unneded feature-definition section
* fix(sliding-sync): minor fixes as per review
* fix(sliding-sync): Make Builders owned
* fix(sliding-sync): more minor style improvements
* fix(sliding-sync): minor style improvements
* fix(sliding-sync): remove homeserver from RequestConfig, use specific internal fn instead
Co-authored-by: Stefan Ceriu <stefanc@matrix.org>
Merge pull request #1013 from matrix-org/ismail/logout
Add logout facilities to `client` and helpers to ffi for tracking the soft-logout issued by a server. Further more this adapts the `sync_with_callback` by adding a new `sync_with_result_callback` that hands the entire `Result` to the callback.
This patch removes the `decode_image` (default) feature for this
`matrix-sdk-qrcode` crate.
First reason is that `rqrr` panics for some particular QR codes, and
we don't want to panic.
Second reason is that it's not a feature that is used. For Element on
iOS and Android, it's very unlikely that every frame from the camera
will be sent to `matrix-sdk-qrcode` to see if it can be decoded. So
it's obvious that an external library is used to read the bytes from
the QR code, that are then sent to `matrix-sdk-qrcode`. For Element on
Web, it's basically the same argument.
This feature is actually used only in our tests to ensure the
generated QR code is valid, but it sometimes fails due to `rqrr`
(cf. First reason).
`AnyDecryptedOlmEvent::Custom` contains at least 440 bytes, while the
second-largest variant (`RoomKey`) contains at least 0 bytes. Let's
box `Custom` so that the size of `AnyDecryptedOlmEvent` stays low.
A room ID can no longer be invalid, it's just a string representing
either a `EventId` (we can validate that but…) or a `TransactionId`
(which is an opaque string, so it can be anything).
This patch updates `QrVerification::new_cross`, by passing a flow ID
as an owned `String` to `VerificationData`, thus removing a panic, and
allowing QR code verification to happen outside a room.
This patch updates `VerificationData` to receive a flow ID,
represented as an owned `String`, instead of an `OwnedEventId`. Why?
Because QR code verification can happen outside a room. In such
scenario, there is no event ID, but a transaction ID, unified behind
the `matrix_sdk_crypto::FlowId` enum. `VerificationData` doesn't
really care about that details. Proof is that `QrVerificationData`
receives an owned `String`, which is then casted into an
`OwnedEventId` to match this API properly; but at the top, it just
receives a string.
This patch brings also a little bit of clean up while editing code
around.
Merge pull request #979 from FlixCoder/cleanup
Fixes a problem, where non-stripped information of the room had predecence in all scenarios of the memory-, sled- and indexeddb-store, thought according to the spec, we prefer the full info only until we've received new stripped info (from another invite). This fixes that to be in line with the spec by removing stripped data when we see a full-event and keep stripped info preferred if found (so, after you joined, stripped data is removed and when you knocked or are invited again, it is preferred), without ever deleting full data. It also adds nice unit- and integration tests to ensure this works as intended.
This patch adds a method to format a list of emojis in a a terminal
friendly way.
This method was borrowed from weechat-matrix but it's also quite useful
in our own emoji verification example.
Since this example prints out messages to stdout, it becomes quite hard
to see when we ask for user input if all the logging is going on.
This patch adds a standard verbosity flag which disables all logging by
default.
Previously, when both a possibly-redacted timeline event handler and a
non-redacted timeline timeline event handler would apply to multiple
events in a sync response, they would individually run for every event
in order. With this change, they will instead both be called
for one event before the next is processed.
The use of `io::Read` wasn't helping since we had to buffer the whole
file in memory anyways, and we are unlikely to get around that in the
near future.
Inbound group sessions would fail to be deserialized if there was a
forwarding curve chain, this patch fixes it so we go through base64 when
we deserialize this field.
This patch introduces a new `pack` NPM script, which runs `wasm-pack
pack` behind the scene.
This patch modifies the `publish` NPM script to run the `pack` script
as a pre-script (so… in the `prepublish` script).
Finally, this patch no longer uses `$npm_execpath` as it doesn't work
on Windows. It should be `%npm_execpath%`. It's not obvious to make
scripts interoperable, so we will stick with `npm` for now.
The `OlmMachine` constructor now has 2 more optional arguments:
`store_name` and `store_passphrase`, to use Indexed DB as the backend
to store the Olm machine keys, rather than having an in-memory Olm
machine.
Node.js is used to test our binding. Indexed DB is absent of
Node.js. So, firstly, we are using the `fake-indexeddb` JavaScript
library to mimic the same API inside Node.js. And, secondly, we use
our own fork of the `indexed_db_futures` Rust crate to provide add
support for Node.js too ([PR is
here](https://github.com/Alorel/rust-indexed-db/pull/11)). It
basically looks in the Node.js global environment if the `indexedDB`
getter is present, just like it is already done for `Window` on any
browser, or `WorkerGlobalScope` for workers.
This patch updates the `publish` NPM script to explicitly run
`wasm-pack pack` to create the NPM package in the `pkg/`
directory. This patch also updates the Github Action workflow for the
`matrix-sdk-crypto-js` release, to create a new Github Release, with
the NPM package attached as an asset file.
This patch adds a new `npm run publish` script that:
1. Run `npm run prepublish` (which runs the `build` and `test` scripts),
2. Run `wasm-pack publish`.
Note 1: The `prepublish` script is using the `$npm_execpath`
environment variable instead of just “`npm`”, so that if someone is
using `yarn` or another JavaScript package manager, it _should_ work
(not tested yet).
Note 2: `wasm-pack publish` is run without running `wasm-pack login`
before that. _But_ we are updating the registry URL in the NPM
configuration file in the Github Action workflow, see below.
This patch then creates a new Github Action workflow that is triggered
when a new tag of the form `matrix-sdk-crypto-js-v[0_9]+.*` is
pushed. This workflow runs `npm run publish` basically, but before
that, it updates the NPM configuration file by setting a value for
`//registry.npmjs.org/:_authToken`. Thus, running `wasm-pack login` is
not necessary.
Implement the `Attachment.encrypt` and `Attachment.decrypt` methods,
along with its `EncryptedAttachment` companion.
The trick is to avoid copies as much as possible. Instead of dealing
with `Uint8Array` as I've initially done, `&[u8]` and `Vec<u8>` is
passed instead. `wasm-bindgen` is smart enough to do as few copies as
possible (from JavaScript to Wasm's memory is required, and we don't
want to do more), while typing everything as `Uint8Array`.
`EncryptedAttachment.encryptedData` returns a copy of the data
everytime it is called, and that's the last copy I'm not happy with.
… rather than deserializing at an enum type and taking the same branch
for getting a different enum variant¹ as failed deserialization.
¹ which should be impossible anyways
Initial state events aren't actually received through sync, they're sent from
the client to the server when creating a room.
This change makes it impossible to register event handlers for initial state
events. Previously, this was allowed, but the event handler was never called.
This patch adds some more stronger guarantees that received room key
events and other similarly security critical event types are only
handled if they have been received over a secure Olm channel.
… by waiting for the corresponding event confirming the action. Affects:
* Creating a room
* (Re-)Joining a room
* Leaving a room
* Accepting an invitation
* Rejecting an invitation
This patch delegates the decision making if a session is better to
vodozemac. Vodozemac has better insights if a group session can be
considered to be better.
This allows smoother development as a regular 'cargo check' will
validate most of the code. It also makes rust-analyzer work for the
crate without any configuration.
… instead of through `compile_error!` invocations. This helps avoid
unhelpful errors from the unsupported feature configuration by aborting
compilation earlier.
This patch ensures that the received secret has been sent by one of our
verified devices. We already ensure so for the secrets we import, since
we check that the cross signing keys match the public keys of our own
user idenity.
No other secrets get imported by this method but it could quite easily
become an issue if we start accepting more secret types.
This fixes a bug where we would lose already generated to-device
requests carrying an m.room_key when the client gets restarted.
This only happens if the request was generated after the initial
share of the room key. Such requests get generated if a new device
or member join the group.
This functionality is important to allow clients to log in using an OIDC server. The OIDC server returns only an access token, the client needs to fetch the user ID and device ID from the Matrix server separately.
This lives in the bindings for now since OIDC support in Matrix is being actively developed.
If the last sync was less then a 1s ago we wait for a 1s, it doesn't
make sense to wait an additional second on an error. Also the stream
sync api returns the error after a delay of 1s, which doesn't make
sense.
This patch creates one `inner` module for when `feature = "tracing"`,
and one for when `no(feature = "tracing")`. Then, let's expose
everything from `inner::*`.
This patch also replaces `Tracing.install` by `new Tracing`. In case
of `not(feature = "tracing")`, `new Tracing` raises an error.
The goal is to remove all the `#[cfg(…)]` annotations everywhere. Now
there is only 2 of them.
The patch updates the code to use `tracing_subscriber::reload`, so
that we get a `Handle` that can be used to modify the tracing at
runtime.
This patch also adds a new `tracing` feature.
This patch finally adds a test suite for the `Tracing` API.
@@ -24,17 +24,11 @@ The rust-sdk consists of multiple crates that can be picked at your convenience:
- **matrix-sdk-crypto** - No (network) IO encryption state machine that can be
used to add Matrix E2EE support to your client or client library.
## Minimum Supported Rust Version (MSRV)
These crates are built with the Rust language version 2021 and require a minimum compiler version of `1.60`
## Status
The library is in an alpha state, things that are implemented generally work but
the API will change in breaking ways.
The library is considered production ready and backs multiple client implementations such as Element X [[1]](https://github.com/element-hq/element-x-ios) [[2]](https://github.com/element-hq/element-x-android) and [Fractal](https://gitlab.gnome.org/World/fractal). Client developers should feel confident to build upon it.
If you are interested in using the matrix-sdk now is the time to try it out and
provide feedback.
Development of the SDK has been primarily sponsored by Element though accepts contributions from all.
The @matrix-org/rust team is happy to present to the wider public version 0.5 of the [`matrix-rust-sdk`][sdk], now available on crates.io for your convenience. This marks an important milestone after over half a year of work since the latest release (0.4) and with many new team members working on it. It comes with many bug fixes, improvements and updates, but a few we'd like to highlight here:
## Better, safer, native-er crypto
One of the biggest improvements under the hood is the replacement of the former crypto core written in C, libolm, with the fully rustic, fully audited [Vodozemac][vodozemac]. [Vodozemac][vodozemac] is a much more robust new implementation of the olm crypto primitives and engines with all the learning included and pitfalls from the previous C implementation avoided in a freshly baked, very robust library. With this release both matrix-sdk-crypto and matrix-sdk (when the `e2e-encryption` is enabled, which it is by default) use vodozemac to handle all crypto needs.
## Stores, stores, stores
This release has also seen a major refactoring around the state and crypto store primitives and implementations. From this release forward, the implementation of the storage layer is truly modular and pluggable, with the default implementation (based on `sled`) even being a separate crate. Furthermore this release has also additional support for the [`indexeddb`-storage][indexeddb] layer for in-browser WASM needs. As before, the base still ships with an in-memory store, but if none of them fits your needs, you are very much invited to build your own - we recommend looking at the implementation of the existing [`matrix-sdk-sled`][sled] to understand what is needed.
We've further extended and cleaned up the storage API and hardened the two existing implementations: both `sled` and `indexeddb` will now encrypt _all metadata_, including all keys, if a passphrase is given. Even a core dump of a database won't show any strings or other identifiable data. Both implementation also hold a database version string now, allowing future migrations of the database schema while we move forward.
## WebAssembly
It already came through in the previous paragraph: we now fully support `wasm` as a primary target for the main sdk and its dependencies (browser for now). While it was already possible to build the SDK for wasm before, if you were lucky and had the specific emscripten setup, even crypto didn't always fail to compile, our move to vodozemac frees us from these problems and our CI now makes sure all our PRs will continue to build on WebAssembly, too. And with the `indexeddb`-store, we also have a fully persistent storage layer implementation for all your in-browser matrix needs built in.
## More features
Of course a lot more has happened over the last few months as we've ramped up our work on the SDK. Most notably, you can now check whether a room has been created as a `space`, and we offer nice automatic thumbnailing and resizing support if you enable `image`. Additionally, there is a very early event-timeline API behind the `experimental-timeline`-feature-flag. Just to name a few. We really recommend you migrate from the older version to this one.
## Process and Workflow
We've also ramped up our CI, parallelized it a lot, while adding more tests. In general we don't merge anything breaking the tests, thus our `main` can also be considered `stable` to build again, though things might be changed without prior notice. We recommend keeping an eye out in our [team chat][chat] to stay current.
### Versions
This and all future releases of matrix-sdk adhere to [semver][semver] and we do our best to stick to it for the default-feature-set. As you will see only the main crate `matrix-sdk` is actually versioned at `0.5`, while other crates - especially newer ones - have different versions attached. We recommend to sticking to the high-level matrix-sdk-crate wherever reasonable as lower level crates might change more often, also in breaking fashion.
With this release all our crates are build with `rust v2021` and need a rust compiler of at least `1.60` (due to our need for weak-dependencies).
### Conventional Commit
We also want to make it easier for external parties to follow changes and stay up to date with what's-what, so, with this release, we are implementing the [conventional commits][cc] standard for our commits and will start putting in automatic changelog generation. Which makes it easier for us to create comprehensive changelogs for releases, but also for anyone following `main` to figure out what changed.
**matrix-rust-sdk** leverages [UniFFI](https://mozilla.github.io/uniffi-rs/) to generate bindings for host languages (eg. Swift and Kotlin).
Rust code related with bindings live in the [matrix-rust-sdk/bindings](https://github.com/matrix-org/matrix-rust-sdk/tree/main/bindings) folder.
Developers can expose Rust code to UniFFI using two different approaches:
- Using an `.udl` file. When a crate has one, you find it under the `src` folder (an example is [here](https://github.com/matrix-org/matrix-rust-sdk/blob/main/bindings/matrix-sdk-ffi/src/api.udl)).
- Add UniFFI directivies as Rust attributes. In this case Rust source files (`.rs`) contain attributes related to UniFFI (e.g. `#[uniffi::export]`). Attributes are preferred, where applicable.
## Expose Rust definitions to UniFFI
### Check if the API is already on UniFFI
First of all check if the Rust definition you are looking for exists on UniFFI already. Most of exposed matrix definitions are collected in the crate [matrix-sdk-ffi](https://github.com/matrix-org/matrix-rust-sdk/tree/main/bindings/matrix-sdk-ffi).
This crate contains mainly small Rust wrappers around the actual Rust SDK (e.g. the crate [matrix-sdk](https://github.com/matrix-org/matrix-rust-sdk/tree/main/crates/matrix-sdk))
If the Rust definition is on UniFFI already, you either:
- find it in a `.udl` file like [matrix-sdk-ffi/src/api.udl](https://github.com/matrix-org/matrix-rust-sdk/blob/main/bindings/matrix-sdk-ffi/src/api.udl)
- see it marked with a proper UniFFI Rust attribute like this `#[uniffi::export]`
### Adding a missing matrix API
1. Unless you want to contribute on the crypto side, you probably need to add some code in the [matrix-sdk-ffi](https://github.com/matrix-org/matrix-rust-sdk/tree/main/bindings/matrix-sdk-ffi) crate. After you find the crate you need to understand which file is best to contain the new Rust definition. When exposing new matrix API often (but not always) you need to touch the file [client.rs](https://github.com/matrix-org/matrix-rust-sdk/blob/main/bindings/matrix-sdk-ffi/src/client.rs)
2. Identify the API to expose in the target Rust crate (typically in [matrix-sdk](https://github.com/matrix-org/matrix-rust-sdk/tree/main/crates/matrix-sdk). If you can’t find it, you probably need to touch the actual Rust SDK as well. In this case you typically just need to write some code around [ruma](https://github.com/ruma/ruma) (a Rust SDK’s dependency) which already implements most of the matrix protocol
3. After you got (by finding or writing) the required Rust code, you need to expose to UniFFI. To do that just write a small Rust wrapper in the related UniFFI crate (most of the time is **matrix-sdk-ffi**) you found on step 1.
4. When your new (wrapping) Rust definition is ready, remember to expose it to UniFFI.
It’s best to do it using UniFFI Rust attributes (e.g. `#[uniffi::export]`). Otherwise add the new definition in the crate’s `.udl` file. For the **matrix-sdk-ffi** crate the definition file is [api.udl](https://github.com/matrix-org/matrix-rust-sdk/blob/main/bindings/matrix-sdk-ffi/src/api.udl). **Remember**: the language inside a `.udl` file isn’t Rust. To learn more about how map Rust into UDL read [here](https://mozilla.github.io/uniffi-rs/udl_file_spec.html)
## FAQ
**Q**: I wrote my Rust code and exposed it to UniFFI. How can I check if the compiler is happy?\
**A**: Run `cargo build` in the crate you touched (e.g. matrix-sdk-ffi). The compiler will complain if the Rust code and/or the `.udl` is wrong.
**Q**: The compiler is happy with my code but the CI is failing on GitHub. How can I fix it?\
**A**: The CI may fail for different reasons, you need to have a look on the failing GitHub workflow. One common reason though is that the linter ([Clippy](https://github.com/rust-lang/rust-clippy)) isn’t happy with your code. If this is the case, you can run `cargo clippy` in the crate you touched to see what’s wrong and fix it accordingly.
This project and build script demonstrate how to create an XCFramework that can be imported into an Xcode project and run on Apple platforms. It can compile and bundle an [entire SDK](#Building-the-SDK), or only a smaller [Crypto module](#Building-only-the-Crypto-SDK) that provides end-to-end encryption for clients that already depend on an SDK (e.g. [Matrix iOS SDK](https://github.com/matrix-org/matrix-ios-sdk))
## Prerequisites for building universal frameworks
## Building
* the Rust toolchain
* UniFFI - `cargo install uniffi_bindgen`
* Apple targets (e.g. `rustup target add aarch64-apple-ios`)
*`xcodebuild` command line tool from [Apple](https://developer.apple.com/library/archive/technotes/tn2339/_index.html)
*`lipo` for creating the fat static libs
### Prerequisites for building universal frameworks
-`xcodebuild` command line tool from [Apple](https://developer.apple.com/library/archive/technotes/tn2339/_index.html)
-`lipo` for creating the fat static libs
### Building the SDK
```
sh build_xcframework.sh
cargo xtask swift build-framework
```
The `build_xcframework.sh` script will go through all the steps required to generate a fully usable `.xcframework`:
The `build-framework` task will go through all the steps required to generate a fully usable `.xcframework`:
1. compile `matrix-sdk-ffi` libraries for iOS, the iOS simulator, MacOS, and Mac Catalyst under `/target`. Some targets are not part of the standard library and they will be built using the nightly toolchain.
1. compile `matrix-sdk-ffi` libraries for iOS, the iOS simulator and macOS under `/target`. Some targets are not part of the standard library and they will be built using the nightly toolchain.
2.`lipo` together the libraries for the same platform under `/generated`
3. run `uniffi` and generate the C header, module map and swift files
4.`xcodebuild` an `xcframework` from the fat static libs and the original iOS one, and add the header and module map to it under `generated/MatrixSDKFFI.xcframework`
5. cleanup and delete the generated files except the .xcframework and the swift sources (that aren't part of the framework)
## Building only the Crypto SDK
For development purposes, it will additionally generate a `Package.swift` file in the root of the repo that can be used to add the framework to your project and enable debugging through the use of [rust-xcode-plugin](https://github.com/BrainiumLLC/rust-xcode-plugin) (make sure to run the task with the argument `--profile=reldbg`).
When building the SDK for release you should pass the `--release` argument to the task, which will strip away any symbols and optimise the created binary.
### Building only the Crypto SDK
```
sh build_crypto_xcframework.sh
build_crypto_xcframework.sh
```
The `build_crypto_xcframework.sh` script will go through all the steps required to generate a fully usable `.xcframework`:
@@ -38,16 +43,31 @@ The `build_crypto_xcframework.sh` script will go through all the steps required
4.`xcodebuild` an `xcframework` from the fat static libs and the original iOS one, and add the header and module map to it under `generated/MatrixSDKCryptoFFI.xcframework`
5. cleanup and delete the generated files except the .xcframework and the swift sources (that aren't part of the framework)
## Running the Xcode project
### Building & testing the Swift package
The Xcode project is meant to provide a simple example on how to integrate everything together but also a place to run unit and integration tests from.
The `Package.swift` file in this directory provides a simple example on how to integrate everything together but also a place to run unit and integration tests from.
It's pre-configured to link to the generated .xcframework and .swift files so successfully running the script first is necessary for it to compile.
It makes the compiled code available to swift by importing the C header through its bridging header.
Once all the generated components are available running it should be as easy as choosing a platform and clicking run.
It's pre-configured to link to the generated static lib and .swift files so successfully running `cargo xtask swift build-library` first is necessary for it to compile. Afterwards you can execute the tests with `swift test`. Note that for the moment this only works on macOS but we're planning to add Linux support in the future.
## Distribution
The generated framework and Swift code can be distributed and integrated directly but in order to make things simpler we bundle them together as a Swift package available [TBD](here) in the case of SDK, and as CocoaPods podspec in the case of Crypto SDK.
The generated framework and Swift code can be distributed and integrated directly but in order to make things simpler we bundle them together as a [Swift package](https://github.com/matrix-org/matrix-rust-components-swift/) in the case of SDK, and as a CocoaPods podspec in the case of Crypto SDK.
### Publishing MatrixSDKCrypto
1. Navigate into `bindings/apple` and run [`build_crypto_xcframework.sh`](#building-only-the-crypto-sdk) script which generates a .zip file with the framework in `./generated` folder
Note that whilst you can run this command from any git branch, if you want to make a public release, ensure you are building from latest `main`
2. Tag the commit you just used to build the release and push the tag to GitHub
Use `matrix-sdk-crypto-ffi-<major>.<minor>.<patch>` tag name
3. Create a new [GitHub release](https://github.com/matrix-org/matrix-rust-sdk/releases) using this tag (see [example](https://github.com/matrix-org/matrix-rust-sdk/releases/tag/matrix-sdk-crypto-ffi-0.3.4))
4. Upload the previously generated .zip file to this release
5. Increment the version in [`MatrixSDKCrypto.podspec`](./MatrixSDKCrypto.podspec)
Note that this is not automated and is a local-only change. To determine the most recent published version, run `pod repo update && pod search MatrixSDKCrypto`
or check git tags via `git tag | grep matrix-sdk-crypto-ffi`
6. Push new Podspec version to Cocoapods via `pod trunk push MatrixSDKCrypto.podspec --allow-warnings`
describe('setup workflow to encrypt/decrypt events',()=>{
letm;
constuser=newUserId('@alice:example.org');
constdevice=newDeviceId('JLAFKJWSCS');
constroom=newRoomId('!test:localhost');
beforeAll(async()=>{
m=awaitmachine(user,device);
});
test('can pass keysquery and keysclaim requests directly',async()=>{
{
// derived from https://github.com/matrix-org/matrix-rust-sdk/blob/7f49618d350fab66b7e1dc4eaf64ec25ceafd658/benchmarks/benches/crypto_bench/keys_query.json
// derived from https://github.com/matrix-org/matrix-rust-sdk/blob/7f49618d350fab66b7e1dc4eaf64ec25ceafd658/benchmarks/benches/crypto_bench/keys_claim.json
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Damir Jelić
