Currently this takes a `CrossSigningKeyExport`, but that's not a class you can
construct from the JS side. Instead, let's just pass in the individual keys.
Using the #[derive(uniffi::Object)] on certain types make them not show up
in the .aar file when building for Android. Could not determine why that is,
but removing the derive macro, and adding the empty interface declarations
inside the UDL, makes them show up.
We can still use #[uniffi::export] and #[uniffi::constructor].
Signed-off-by: Simon Farre <simon.farre.cx@gmail.com>
The unknown filter can be difficult to match with MembershipState,
depending on the store implementation.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Storing as bitset is not future-proof.
Just fix the latest migration. We assume no-one used it yet.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
The `SlidingSyncBuilder` now has a new method: `bump_event_types`, to
configure the `bump_event_types` HTTP request parameter. This value
is passed to `SlidingSync`, which is used to build the `SlidingSync`
request.
The format of the outbound group session struct has changed. We nowadays correctly rotate the group session if we can't restore it, but it's still good to avoid logging the error in this case.
An error is currently thrown if loading an outbound group session fails.
This error may only affect loading outbound group sessions, while other
store operations continue to work fine. As a result, the user is unable
to send messages, but can still use the application without any problems.
Since outbound group sessions are rotated frequently, we can simply
rotate the session if loading fails. However, if the error is related to
a more serious storage issue, persisting the newly rotated outbound group
session will also fail. If the error is specific to outbound group
sessions, the user will be able to continue using the application without
interruption.
The SQLite crypto store uses rmp_serde to serialize all the data we're
going to store. This works nicely for most things, one exception to this
is the OutboundGroupSession type.
The OutboundGroupSession type stores to-device requests to ensure that
the session doesn't get used before it is shared with the whole group
and to ensure that the to-device requests get restored if the
session gets restored after an application restart.
The to-device requests type critically contain `Raw<AnyToDeviceEvent>`,
the `Raw` type here being the serde_json::Raw type. rmp_serde seems to
serialize this just fine, but later deserialization fails.
We're avoiding the issue by using serde_json to serialize the
OutboundGroupSession.
Those methods are public, but never used by our current users.
Moreover, there is some big overlaps. For example, `storage_key`,
`cold_cache` and `no_cache` do the same thing: They update the
`storage_key` field. Or `add_fullsync_list` is just a helper that is
never used except in the tests etc.
The test does the following:
1. Create 2 (identical) lists,
2. Do a sync.
3. Assert that the 1st and 2nd lists are receiving an update,
4. Add a 3rd (identical) list,
5. Do a new sync,
6. Assert that 3rd list is receiving an update.
This last step is wrong. All lists should receive an update as they
are identical.
This patch updates a test to ensure that `room_list` receives “diff”
for rooms that are modified by a sync operations, but also by another
updates (like a new event).
The `SlidingSyncList.room_list` field is used to store the list of rooms
for a particular Sliding Sync list. It's used by the user to receive
“diff”s when a room sees its position modified. For example when a new
room receives a message, it “climbs back up” the entire room list to
be at the top place. Another example is when a new room is created,
some rooms will move around to give the new room a space. All those
moves will create “diff”.
However, the user also expects to receive a “diff” when a room has
received some updates, even if it's position doesn't change. For
example, when a room is already at the top of the list but receives a
new message: It has received an update, but its position stays the same.
This specific latter feature was implemented before, but it has been
removed by accident in https://github.com/matrix-org/matrix-rust-sdk/
pull/1699 (more specifically in https://github.com/matrix-org/matrix-
rust-sdk/pull/1699/commits/861a05be69a566d9a4ad125dc6ecb418d2b3210f).
At that time, it was not clear why the code was filtering for specific
filled room entires, to set the filled room entries, at the same
position. Zero comment, zero test. I didn't consider this as a feature
but as a bug.
So this patch re-introduces this feature. Hopefully in a more optimal
way. If a room has already triggered a first “diff” because of a
position change, it won't trigger a second “diff” because it has
received an update (which was the case before).
The `SlidingSyncList::handle_response` method has also been renamed
`update`, as it does update, whenever it comes from.
The code has been commented and documented to explain this feature.
Existing tests have been updated, especially for `apply_sync_operations`
which now ensure the `rooms_that_have_received_an_update` collections is
updated accordingly. Another test has been updated specifically to test
the “diff”s received by `room_list`.
This patch also re-uses the `format_storage_key_*` functions to use the
same key as the `restore_sliding_sync_state` function. It fixes a bug
where the keys were mismatching.
This PR is self explanatory.
Exposes said function to the FFI. Defined in the .adl file,
and defined to throw `ClientError`
Signed-off-by: Simon Farre <simon.farre.cx@gmail.com>
Co-authored-by: Damir Jelić <poljar@termina.org.uk>
In some cases, restoring client state from backups may cause Olm sessions
to become corrupted, resulting in a backward ratchet. As a consequence,
the receiving side is unable to decrypt messages. To address this issue,
the failed side initiates a new Olm session and sends a dummy encrypted
message.
However, this dummy message also creates a new Olm session on the
sender's side. To ensure that both sides use the same new session, we
must sort sessions by their creation timestamp before encrypting new
messages.
Although the session list was sorted correctly previously, we selected
the wrong side of the list, resulting in an outdated session being
selected instead of the newest one. This patch resolves the issue by
selecting the newest Olm session and adds a test to the session getter
logic to prevent reintroduction of this bug.
Prior to this patch, when `SlidingSync` was built with a storage key,
the storage was read to find a previous `SlidingSync` version that
was in a cache. To put a `SlidingSync` instance in the cache, it
was serialized to JSON. When one reads from the cache, the value is
deserialized. A problem happens when the internal representation of
`SlidingSync` and other types (e.g. `SlidingSyncList`) have changed (due
to an SDK update for example): Loading a previous state from the cache
results in an error.
After discussion with the teams, clients don't want to deal with
obsolete cache values. In such a scenario, (i) they cannot interact
with the cache to remove the obsolete entry, and (ii) their only
possibility is to log out and log in the user again. What an unfriendly
user experience!
This patch modifies this behaviour. When loading a `SlidingSync` type or
a `SlidingSyncList` type from the cache, 3 cases are now handled:
1. The cache entry exists and has been successfully deserialized: in
this case, we do our business.
2. The cache entry exists, but it wasn't possible to deserialize it: the
cache entry is declared as obsolete, and the entire `SlidingSync`
cache is cleaned up, so all entries for `SlidingSync` and
`SlidingSyncList` are removed from the storage,
3. The cache entry doesn't exist: we do nothing particular.
Note that if one cache is obsolete, all cache entries are removed.
The `SlidingSync::stream` method no longer resets the lists everytime
it is called. If one wants to reset the lists, they need to call
`SlidingSync::reset_lists`.
`SyncOp::Invalidate` means _invalidating_ a particular range. When a
room is `Filled`, it becomes `Invalidated`, when it is `Invalidated` it
stays `Invalidated`, and when it is `Empty` it stays `Empty`.
Before this patch, `Empty` was becoming `Invalidated`, which apparently
is a bug.
This patch also fixes out-of-bound accesses, and adds many tests.
Finally, this patch renames `update_state` to `update_room_lists`.
`SyncOp::Insert` means _inserting_ a new room ID. The `rooms_list`
contains all possible rooms (based on `maximum_number_of_rooms`, a value
returned by the server). Here, inserting = setting
`RoomListEntry::Filled` at a particular index of `rooms_list`, that's
it.
The previous code was doing very complex stuff, like removing things
around the `index` if something etc. It was using the requested ranges
(the range passed to the request) etc.
Applying `SyncOp` should be simple and is focused on updating
`rooms_list` only: the requested ranges have nothing to do here.
This patch also prevents against out-of-bounds acccesses, which wasn't
the case before.
This patch prevents out of bounds acceses for `SyncOp::Delete`, and adds
more tests.
This patch also removes a cast from `UInt` to `u32` to `usize`. It's now
from `UInt` to `usize` directly.
First off, this patch renames `ops` to `sync_operations` and `room_ops`
to `apply_sync_operations`.
Second, the `SlidingOp::Sync` was creating an out-of-bounds access
depending of the range present in the server's response. For example, if
the `rooms_list` contains 5 elements (because the
`maximum_number_of_rooms` is set to 5), and the server replies with:
```json
{
"op": "SYNC",
"ranges": [3, 17],
"room_ids": […]
}
```
the previous code was setting a new `RoomListEntry` at indices `3..=17`,
whilst the `rooms_list` contains only indices from `0..=4`. That's
annoying.
The previous code was also counting the number of `room_ids` for
nothing, just to execute the iterator that was applying the actual
changes in a `map`. Well, everything was fishy.
This patch updates the code to protect against an unexpected server's
reply by raising an `Err`. This patch also adds tests.
The `updated_rooms` argument was passed to `find_rooms_in_list` to
update the `room_list`: the update is setting the filtered room list
entry to `RoomListEntry::Filled`.
_But_, `find_rooms_in_list` was already filtering rooms which are
`Filled`. So it does… nothing: it filters rooms which are `Filled` to
update them to `Filled`.
So we can remove `find_rooms_in_list` because it becomes useless. And we
can remove `updated_rooms` too.
The `rooms_list` is updated by `rooms_ops` itself. Let's keep
modifications in one unique place.
The `update_state` method of `SlidingSyncListInner` has basically
2 cases:
1. For an initial response,
2. For other responses.
The code between the 2 cases were almost identical. Or, they could be
identical. The few exceptions are:
* In the first case, the `rooms_list` updates were taking the
form of a `VectorDiff::Append`, while the second case, it was a
`VectorDiff::PushBack`.
* In the first case, the `is_cold` flag was set to `false`.
It's fine for the clients to receive only a `VectorDiff::Append` event
only. So let's make it uniform.
And it appears that the `is_cold` field is now private, and never read
anywhere else. So… it's… basically useless. We can remove it! It was
previously used here to know which flow to use, but since we can make
both flows identical, its role becomes insignificant.
`create_range` and `SlidingSyncList::next_request` return a `Result`
instead of an `Option`. It was a legacy from the old `impl Iterator` of
`SlidingSyncListRequestGenerator`. But actually, when `create_range`
fails, it must return a `Err` value, not a `None` value.
And since it's a regular error, `sync_once` can propagate the
error. Thus, it is no longer necessary to “update” the list of
`SlidingSyncList`. It's not necessary to remove some items in this list
when the `next_request` method can no return a value: it always returns
a value, except when a error happens.
First off, `set_ranges`, `set_range`, `add_range` and `reset_ranges`
take a `&[(U, U)]` instead of a `Vec<(U, U)>` when a vector was needed,
or takes a `(U, U)` instead of 2 arguments when it was needed.
Second, all those methods now return a `Result<(), Error>`. The
`Error::CannotModifyRanges` is raised if the chosen `SlidingSyncMode`
doesn't allow to modify ranges. Basically, `Selective` does allow a
user to modufy the ranges, but there is no real ranges with `Growing` or
`Paging`. Let's make it an explicit error.
Finally, `SlidingSyncListInner` has 2 methods: `set_ranges` and
`add_range`, but without any “ranges check”; `SlidingSyncList` does call
the inner methods, but does the checks.
The `SlidingSyncListInner::find_rooms_in_list` method can be greatly
improved by using the `Iterator` API.
It was already using `Iterator::skip`, great! Let's continue by using
`Iterator::take` instead of using a stop index.
And let's use `Iterator::enumerate` to avoid using a mutable index.
`SlidingSyncList::find_room_in_list` and
`SlidingSyncList::find_rooms_in_list` aren't useful (and never used) by
the public consumer of this API. Let's remove it. They are only used for
internal purposes.
Before, the `update_request_generator_state` method was emitting an
`error!` log, but not an error. But that's clearly an error!
This patch updates the method to return a `Result<(), Error>`, and adds
a new variant to `Error`.
There are problems with `SlidingSyncListRequestGenerator`:
* It's part of the module API,
* It contains a clone of `SlidingSyncList`.
To create a `SlidingSyncListRequestGenerator`, one has to
call `SlidingSyncList::request_generator`. It was done in
`SlidingSync::stream`. The problem is that it clones `SlidingSyncList`.
So theoritically it is possible to create multiple request generators
for the _same_ list, and use them to send many requests and to update
the _same_ list with multiple responses. This is utterly error-prone and
can lead to really complex bugs to discover.
Moreover, it's a lot of clones. Cloning a
`SlidingSyncListRequestGenerator` isn't cheap as it means cloning a
`SlidingSyncList`. Moreover, cloning a `SlidingSyncList` isn't cheap as
it means cloning the entire struct.
Having `SlidingSyncListRequestGenerator` inside the module API also
makes the code of `SlidingSync` more complex (why having to deal with
lists and request generators at the same time? we must be very careful
to maintain both side by side? what if a list is removed but not its
request generator? and so on).
So. This patch simplifies all that.
First off, it extracts all the fields of `SlidingSyncList` into a
`SlidingSyncListInner` struct. Then, `SlidingSyncList` has only one
field: `Arc<SlidingSyncListInner>`. Boom, it's now cheap to clone it.
Second, `SlidingSyncListRequestGenerator` is only a
struct with constructors, but there is no extra methods.
`SlidingSyncListRequestGenerator` _no longer_ contains a
`SlidingSyncList`, and doesn't no need a list at all to work. It's just
values.
Third, `SlidingSyncList` holds a `SlidingSyncListRequestGenerator`, and
only one.
Fourth, `SlidingSyncList` (and `SlidingSyncListInner`) now has methods
to handle the internal request generator. The initial `impl Iterator for
SlidingSyncListRequestGenerator` becomes a simple
`SlidingSyncList::next_request` method.
Fifth, previously, the `SlidingSyncList::handle_response`
was never called directly by `SlidingSync`. It was called by
`SlidingSyncListRequestGenerator`! How confusing! `SlidingSync` called
`SlidingSyncListRequestGenerator::handle_response` which was calling
`SlidingSyncList::handle_response`. Now, the flow is more natural:
`SlidingSync` calls `SlidingSyncList::handle_response` and that's it.
Sixth, the `SlidingSyncList::handle_response` is now composed of 2
parts: updating the list itself, and updating the request generator. It
was kind of the case before, but onto two different types.
It was unclear which types were updating `SlidingSyncList`.
For example, `SlidingSyncList::state` was updated by…
`SlidingSyncListRequestGenerator`. Now `SlidingSyncList` is responsible
to update itself, and no one else.
Finally, `SlidingSync` no longer have to deal with
`SlidingSyncListRequestGenerator`. All it has is a set of
`SlidingSyncList`, and that's it!
The tests are still passing, hurray.
In `SlidingSyncListBuilder`, the `ranges`, `set_range`, and `add_range`
methods do not take a `u32` but a `U: Into<UInt>`. That's great!
However, in `SlidingSyncList`, the same methods take a `u32`. It makes
the API inconsistent.
This patch fixes that.
This patch renames `SlidingSyncList.set_ranges` to `ranges` so that
it matches the same terminology of the `SlidingSyncListBuilder` with
`ranges`, `set_range`, `add_range` and `reset_ranges`.
Consistency is important here.
The `SlidingSyncList::new_builder` method creates a new builder from a
`SlidingSyncList`. This method was missing some configurations, like
`full_sync_maximum_numbre_of_rooms_to_fetch`, `send_updates_for_items`,
`filters`, `ranges` and `timeline_limit`.
This patchs adds those missing configurations.
The `SlidingSyncListBuilder` struct has a `state` and a `rooms_list`
fields. Those fields are never updated and no setters or getters exist
to use them. There are just set with defaults, and passed to the final
`SlidingSyncList` type within the `build` method.
If a field is present in a builder type, it means they can be modified,
but it's not the case. So this patch removes them.
Event deduplication is supposed to be handled by the `Timeline`. Sliding
Sync is not responsible to de-duplicate the events, in our model. Having
two different de-duplication logics (one in Sliding Sync, and one in the
Timeline) could create weird situations, and hard to debug issues.
We give it a try by removing the deduplication from Sliding Sync.
The `m.no_olm` withheld code is supposed to be sent out only once for a
given device.
This patch prevents double sending of m.no_olm withheld code by adding
a test that ensures only one group session will send it out.
Previously, the logic for sharing group sessions was mostly correct, but
an edge case was forgotten where two group sessions attempted to share
the withheld code concurrently.
Although there's still a possibility for multiple m.no_olm withheld codes
to be sent out. This may happen during the propagation of the flag
remembering the fact that the `m.no_olm` code has been sent from the
OutboundGroupSession to Device. This scenario is likely unrealistic and
the consequences of sending things twice aren't problematic.
Currently, the tags used for releases of this binding have the format:
"matrix-sdk-crypto-js-v0.1.0-alpha.5". This is inconsistent with tags used for
other parts of the project, which omit the `v` prefix.
The url crate normalizes the string, but during OIDC verification steps,
the issuer verification must be made against the exact string that was
provided.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Like for the state store, the bindings expose the version as an f64
while the web API expects an unsigned integer.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch should ideally be split into multiple smaller ones, but life
is life.
This main purpose of this patch is to fix and to test
`SlidingSyncListRequestGenerator`. This quest has led me to rename
mutiple fields in `SlidingSyncList` and `SlidingSyncListBuilder`, like:
* `rooms_count` becomes `maximum_number_of_rooms`, it's not something
the _client_ counts, but it's a maximum number given by the server,
* `batch_size` becomes `full_sync_batch_size`, so that now, it
emphasizes that it's about full-sync only,
* `limit` becomes `full_sync_maximum_number_of_rooms_to_fetch`, so that
now, it also emphasizes that it's about ful-sync only _and_ what the
limit is about!
This quest has continued with the renaming of the `SlidingSyncMode`
variants. After a discussion with the ElementX team, we've agreed on the
following renamings:
* `Cold` becomes `NotLoaded`,
* `Preload` becomes `Preloaded`,
* `CatchingUp` becomes `PartiallyLoaded`,
* `Live` becomes `FullyLoaded`.
Finally, _le plat de résistance_.
In `SlidingSyncListRequestGenerator`, the `make_request_for_ranges`
has been renamed to `build_request` and no longer takes a `&mut self`
but a simpler `&self`! It didn't make sense to me that something
that make/build a request was modifying `Self`. Because the type of
`SlidingSyncListRequestGenerator::ranges` has changed, all ranges now
have a consistent type (within this module at least). Consequently, this
method no longer need to do a type conversion.
Still on the same type, the `update_state` method is much more
documented, and errors on range bounds (offset by 1) are now all fixed.
The creation of new ranges happens in a new dedicated pure function,
`create_range`. It returns an `Option` because it's possible to not be
able to compute a range (previously, invalid ranges were considered
valid). It's used in the `Iterator` implementation. This `Iterator`
implementation contains a liiiittle bit more code, but at least now
we understand what it does, and it's clear what `range_start` and
`desired_size` we calculate. By the way, the `prefetch_request` method
has been removed: it's not a prefetch, it's a regular request; it was
calculating the range. But now there is `create_range`, and since it's
pure, we can unit test it!
_Pour le dessert_, this patch adds multiple tests. It is now
possible because of the previous refactoring. First off, we test the
`create_range` in many configurations. It's pretty clear to understand,
and since it's core to `SlidingSyncListRequestGenerator`, I'm pretty
happy with how it ends. Second, we test paging-, growing- and selective-
mode with a new macro: `assert_request_and_response`, which allows to
“send” requests, and to “receive” responses. The design of `SlidingSync`
allows to mimic requests and responses, that's great. We don't really
care about the responses here, but we care about the requests' `ranges`,
and the `SlidingSyncList.state` after a response is received. It also
helps to see how ranges behaves when the state is `PartiallyLoaded`
or `FullyLoaded`.
The part of code that converts a list of users to the actual
/keys/query request uses the chunks() method. This method operates on
the slice. Our list/vec of users gets dereferenced into a slice before
we create our chunks. The chunks can't take ownership of the data, which
in turn requires us to clone the user IDs for them to be put into the
request.
Itertools has a chunks() method which operates on an iterator which we
can utilize to remove, not only the clone, but also a collect call.
At the same time, let's make the conversion step a simple functional
mapping and document what's going on.
The Option conveys our intention a bit better compared to the default
value. While nothing bad can happen, since the request IDs map will be
empty by default, it's a bit scary to have a valid sequence number since
an i64 defaults to 0.
This patch updates `SlidingSync.pos` and `.delta_token`. Each field has their
own lock. It's annoying because they must updated at the same time to not be
out-of-sync.
So a new field `SlidingSync.position` of type `SlidingSyncPositionMarkers` is
created. It holds `pos` and `delta_token. This new field is behind a single
`RwLock`.
The comments in the code should explain the general approach, but to give an overview:
We assign a unique sequence number to each device-list-invalidation notice, and keep track
of the sequence number at which each user was most recently invalidated. Then, when we
build a `/keys/query` request, we take note of the current sequence number. Finally, when
we get the response from that `/keys/query` request, we compare that sequence number
with the most-recent-invalidation of each user that is now supposedly up-to-date. All
of that means that we can see if the user has been invalidated *since* we built the
`/keys/query` request, in which case our request may have raced against the new device,
so we do not mark the user as up-to-date.
To make that work, we need to keep track of the sequence number for in-flight `/keys/query`
requests; we do that in the `IdentityManager`. Since there should only ever be at most one
batch of requests in flight, that is relatively easy; however, we also record the request ids
just to check that the application isn't doing something weird.
There is no need to persist the sequence numbers to permanent storage: provided the
`IdentityManager` and `Store` objects have a lifetime at least as long as an in-flight
`/keys/query` request, it is sufficient just to retain the `dirty` bit in permanent storage,
and then, when we reload the ephemeral `Store` cache, to treat everything with a `dirty`
bit as a "new" invalidation.
Fixes: #1386.
With https://github.com/matrix-org/matrix-rust-sdk/pull/1601, it's safer to
cancel a stream at any time, as any `Future` cancellation. This documentation
section of `SlidingSync` is no longer relevant as is, and can be rephrased.
With https://github.com/matrix-org/matrix-rust-sdk/pull/1601,
`TaskHandle::with_callback` is no longer necessary.
This patch updates `TaskHandle` as follows:
1. Before, the `handle` and `callback` fields were not mutually exclusive!
Indeed, the `callback` field was used as an abort mechanism, but _also_ as a
finalizer, i.e. a code that runs _after_ the cancellation of `handle`. Now
that the callback-based solution is no longer used, we can keep one usage for
this field and rename it `finalizer`.
2. The `handle` field is no longer optional, as it will always be set.
3. The `with_callback` method is renamed `new` as it's now the only constructor.
4. The `set_callback` method is renamed `set_finalizer`.
Tadaaa.
Move all `SlidingSync`'s fields into a new `SlidingSyncInner` type, and then
update `SlidingSync` to contain a single `inner: Arc<SlidingSyncInner>` field.
First off, it's much simpler to understand what's behind an `Arc` for
“clonability” of `SlidingSync`, and what's not. We don't need to worry about
adding a new field that is not behind an `Arc` for a specific reason or
anything. The pattern is clear now.
Second, this is required for next commits.
Before this patch, `SlidingSync::sync` was returning a callback-based
`TaskHandle`. It was waiting on the “stream loop” to finish (since it's a long-
poll, it means waiting 2*30s, cf. our code), before checking an atomic flag has
some value to decide whether it was time to leave the loop or not. So when the
user is cancelling this `TaskHandle`, a response (if any) was always handled.
But in the meantime, it was possible to start a new `sync`, and it seems like
it creates bugs.
After this patch, `SlidingSync::sync` now returns a handle-based `TaskHandle`.
It means that cancelling it will cancel the “stream loop” immediately. If
no response was in-flight from the server, that's perfect, no problem. If a
response was in-flight, the inner `pos` of the `SlidingSync` instance won't be
updated as the response won't be handled. So the server will re-send the same
response with the next sync request.
I guess it's better this way. Thoughts?
This patch adds a new feature.
Each call to `SlidingSync::stream` generates a unique `stream_id`. This
`stream_id` is passed to requests as a `txn_id` field. Returned responses must
hold the same `txn_id`, otherwise it means `stream` has received unexpected
responses from another `stream` or something else.
So far, when the `txn_id` field of the response and the `stream_id` don't
match, a log with the `ERROR` level is raised. Should we do more, like ignoring
the response entirely? Not sure yet.
The IndexedDB API expects an unsigned integer for the version.
The web-sys bindings expose it as a f64 so versions like 1.1 and 1.2
were used. The were converted back to uint in JS which means the version
was always 1.0 and no real upgrade was processed
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Simplify handling of sliding sync extensions
This patch simplifies the logic for handling sliding sync extensions.
These extensions are sticky, meaning that once enabled in one request,
they do not need to be repeatedly enabled for subsequent requests.
However, the previous logic tried to accommodate the case where the
extensions were initially disabled to reduce the size and processing
time of the initial response. This resulted in complex and error-prone
code.
With this patch, we have removed support for disabling to-device events
and streamlined the handling of sync extensions.
Co-authored-by: Jonas Platte <jplatte@matrix.org>
You can't update a GHA cache once you create it, so if we use the same cache
for each job in a matrix build, then we end up populating it for the first job
that completes, so any slower jobs don't get their dependencies cached.
On the other hand, if we create 20 500MB cache items on each build, we're going
to exhaust the cache storage as soon as we do a build. So, instead, let's just
do the caching for the main branch, and hope that other branches can still
benefit from it.
* Replace custom cancellation action with `concurrency`
* Improve step names
... so don't have three steps with the same name
* Bump version of checkout action
checkout@v2 uses an old version of nodejs, which is deprecated.
To declare the tests, the macro is still used but the content of the
tests is moved in the StateStoreIntegrationTests trait.
Allows to get the proper line reported when a panic occurs,
and have linting, formatting and IDE suggestions.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Currently, the cache of the xtask binary isn't working terribly well:
* we seem to build it on each run anyway, presumably because we don't cache any of the intermediate build
artifacts. Running the binary directly rather than indirecting via "cargo" prevents this.
* There is no sharing of the cache between the "rust" and "bindings" CI, because we use different cache keys.
This PR addresses both problems, and hopefully speeds up CI a bit as a result.
The Sliding Sync server might not send some parts of the response, because they
were sent before and the server wants to save bandwidth. So let's update values
only when they exist.
Running the same example twice
borks encryption.
Instead point to an example that shows how to do it.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This patch cleans up the `SlidingSync::stream` method. It renames variables,
improves log messages etc.
This patch also creates a new `MAXIMUM_SLIDING_SYNC_SESSION_EXPIRATION`
constant. This value was previously hardcoded and lost in the code, now it's
easier to spot it for further updates.
This patch finally renames the `failure_count` field to `reset_counter`,
because it doesn't count the number of failure, but the number of
`ErrorKind::UnknownPos` exactly, i.e. the number of times we reset the
`SlidingSync` state.
This patch cleans up the `SlidingSync::sync_once` method by renaming variables,
adding more comments, simplifying the code by reducing the number of variables
etc.
This patch also changes `futures_util::join!` to `futures_util::future::join`.
It does the same but the macro needs the `async-await-macros` feature to be
turned on, while the second works without any features.
Finally, this patch improves the log messages by making them more clear for a
new reader.
I admit this patch is quite tricky. Please try to follow me.
So, first off, in `SlidingSyncRoom::new`, we were clearing the timeline,
because somehow it exists twice in memory at this step.
Which led me to understand how `SlidingSync::handle_response` was working.
I've clarified how this part of the code works. We are dealing with 2 kind of
responses for a specific reason: `SyncResponse` and `v4::Response`, now it's
documented and I hope it's clearer.
Then, I notice that we were passing a clone of the entire sliding sync
response (`v4::Response`) to `Client::process_sliding_sync`. I thought it was
suboptimal, so I've updated the code to take a reference. It led me to update
`BaseClient::process_sliding_sync`. It was a little bit tricky, but I reckon we
have less clones now than before.
And now, back to `SlidingSync::handle_response`, I was able to compute the
`timeline` correctly by draining it from the `v4::Response`, or by moving it
from `SyncResponse`. So it's no longer necessary to have this clearing code
inside `SlidingSyncRoom::new`. Honestly it has nothing to do at this place
before.
To conclude: We have cleaner code, and less clones.
What thing I reckon could be optimized, is that the entire `timeline`
(`Vec<TimelineEvent>`) is cloned to be passed to `Client::handle_timeline`. So
this timeline exists in 2 places: in Sliding Sync, and somewhere else. I don't
believe it's a problem now, that's how it works, but we must be aware of that.
The `Client::login_custom` allows to login by using a custom login
method. In particular, it is possible to login to Synapse which supports
JWT authentication.
Signed-off-by: boxdot <d@zerovolt.org>
First off, it's not necessary for `SlidingSyncRoom::update` to take a reference
to `room_data: v4::SlidingSyncRoom`. When `update` is called, the iterator owns
its items.
Second, by taking ownership of `room_data`, we no longer need to clone all the
data we need to assign to `self.inner`.
First off, `SlidingSyncRoom.from` doesn't need to be visible to the entire
crate, only to `crate::sliding_sync.
Second, it's a constructor, so let's call it `new`.
1. Put `FrozenSlidingSyncRoom` at the bottom of the module.
2. Put merge 2 `impl SlidingSyncRoom` together.
3. Remove the `AliveRoomTimeline` type alias.
4. Improve the documentation.
So far, the `SlidingSync.pos` field was public to the crate. In order to avoid
breaking the internal state of this type, its visibility is now private.
However, we need to be able to change the value when testing the
`SlidingSync` type itself. To achieve that, this patch removes the old
`force_sliding_sync_pos` function, and implements 2 new functions: `set_pos`
and `pos` directly on `SlidingSync` only when `#[cfg(any(test, feature ="testing"))]`.
Previously, it was possible for us to use invalid indices when:
- We retried decrypting multiple events at once
- One of them (not the last) was an edit or reaction
This lead to a situation where we would remove the UTD item once
decryption for it was successfully retried, but not account for the
resulting index shift for all later timeline items.
Before this patch, `SlidingSyncView` has the following fields that were public:
`state`, `rooms_list` and `rooms_count`. Since they are `Mutable`, they can be
changed from the outside, and then will break the internal state of the view.
This problem is mentioned in https://github.com/matrix-org/matrix-rust-sdk/
issues/1474.
This patch solves this by making them prviate. Phew. That was simple!
But wait, we have a problem now. `matrix-sdk-ffi` was relying on them. So
this patch adds “helpers” methods on `SlidingSyncView`, like `state_stream`,
`rooms_list`, `rooms_list_stream`, `rooms_count` and `rooms_count_stream`.
Let's add new ones when it's necessary.
In https://github.com/matrix-org/matrix-rust-sdk/pull/1478, I've introduced
a regression. The FrozenSlidingSyncRoom.timeline` field has been renamed to
`timeline_queue`.
I didn't notice that this type can be serialized and deserialized. That's
actually a breaking change because this type is stored somewhere once
serialized. So with #1478, it was impossible to deserialize them!
This patch fixes that. It also adds a test to ensure the serialization form of
`FrozenSlidingSyncRoom` doesn't change.
This patch allows an `OlmMachine` to use a different store than Sled, like SQLite.
A new enum is introduced: `StoreType` which 2 variants: `Sled` (the default), and `Sqlite`.
The `OlmMachine.initialize` constructor now takes a new optional `store_type:
Option<StoreType>` argument. If no value is passed, the default `StoreType`
variant will be used (as mentioned: `StoreType.Sled`).
The code has been rewritten a little bit to make the type system happy without
introducing too much type indirections.
This patch finally adds a parameterized tests that exhaustively test: no store
type, `Sled` and `Sqlite`.
Building the crypto-js bindings in release mode is very slow and not really
necessary for local development.
`--release` is the default, so there is no need to specify it
explicitly. Instead, allow `wasm-pack` args to be specified by an env var, and
add a `build:debug` npm script which will build in debug mode.
We already have support for proper `ToDeviceRequest` objects in
`outgoing_requests`, so let's use it in `share_room_key` as well. It's much
easier for the application to work with the proper object than an untyped json
blob.
In the code `if let Some(global_data) = account_data.as_ref().map(|a|
&a.global)`, the `map` is not purely necessary. This patch simplifies the code
to remove the `map` and read the `global` field later on.
By using `Default::default` for the parameter value, we don't know which type
we are passing to the function. Hence the useful comment `room_info.ephemeral`.
This patch changes this to `Ephemeral::default`, so that we now know what we
pass without the need of a comment.
Javascript's `console.trace` is not a good equivalent for Rust's
`Level::TRACE`.
In particular:
* `console.trace` causes a stacktrace to be written with each message
* Under nodejs, `console.trace` writes the message to `stderr`, while
`console.debug` writes to stdout
`console.trace` is therefore somewhat analogous to `console.error`.
Since we already include the log level in the message, we might as well just
send trace messages to `console.debug` rather than `console.trace`.
This is useful for platforms which might want to avoid the cost of
password based key-derivation and have the ability to securely store an
encryption key.
Co-authored-by: Denis Kasak <dkasak@termina.org.uk>
Contrary to other tables that use their own name to encode their keys,
the `ROOM_INFO` table uses the `ROOM` table name to encode its keys.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
`wasm_timer` doesn't work outside the browser, which prevents it being used in
tests.
We can replace it with `gloo-timers` wich uses the standard Javascript
`setTimeout`.
This makes it easier to implement it. However, using a different error
type than CryptoStoreError is a non-trivial change, so left for the
coming commits for all stores except the memory store.
The key encoding in key-value stores is backwards-compatible so
old receipts are counted as unthreaded receipts.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
A fix for (de)serialization of redacted state events in Ruma made
old events undeserializable.
Bump the DB version.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
A fix for (de)serialization of redacted state events in Ruma made
old events undeserializable.
Bump the DB version.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This implementation is wrong in the sense of its semantics is not about
dereferencing a thin pointer to something, but just to give access to one
specific field of the entire structure. That's not how `Deref` is supposed to
be used.
Moreover, it creates conflict between the `SlidingSyncRoom.timeline` field,
and `SlidingSyncRoom.inner.timeline` field, which both exist, but not for the
same purposes. It creates confusion in the code.
Finally, it's better to expose proper getters to the outside world, so that we
control _and_ test _and_ know exactly what API we provide.
`Timeline::with_fully_read_tracking` was public although all public
Timeline constructors enable it by default.
It is also meant to be called when constructing the timeline so
using a builder pattern makes sense.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
`SlidingSyncView` exposes a `room` field. Problem: It's not updated
correctly and leads to error. The canonical way to get a room is by using
`SlidingSync::get_room`. This patch thus removes `SlidingSyncView.rooms`
entirely.
Before this patch, `SlidingSync::get_room` was taking ownership of an
`OwnedRoomId`, to only use it as a reference. Thus, calling this method was
creating useless clones.
This patch updates `SlidingSync::get_room` to receive a reference to
`OwnedRoomId`.
Note about "Write-Ahead Log" (WAL) mode: The SQLite WAL mode has a
bunch of advantages that are quite nice to have:
1. WAL is significantly faster in most scenarios.
2. WAL provides more concurrency as readers do not block writers and a
writer does not block readers. Reading and writing can proceed
concurrently.
3. Disk I/O operations tends to be more sequential using WAL.
4. WAL uses many fewer fsync() operations and is thus less vulnerable
to problems on systems where the fsync() system call is broken.
The downsides of WAL mode don't really affect us. So let's turn it on.
More info: https://www.sqlite.org/wal.html
Co-authored-by: Jonas Platte <jplatte@matrix.org>
Co-authored-by: Damir Jelić <poljar@termina.org.uk>
When we store an Olm session in the database, we encode the sender key and
session ID using the store cipher. That means that we also need to encode them
when we look them up, otherwise we won't find them.
This patch is trying to resolve the following issue:
When a local event is sent to the server, in case of an error, the
`Timeline::send` method just returned the error but it wasn't reflected inside
the `LocalEventTimelineItem` type itself. It's easy to loss this information.
So now, there is a new enum: `LocalEventTimelineItemSendState` with 3 states:
1. `NotSentYet`, when the local event has not been sent yet, it's theorically
the initial state,
2. `SendingFailed`, when the local event has been sent but it's failed!
3. `Sent`, when the local event has been sent successfully.
Therefore, the `LocalEventTimelineItem` struct has a new field: `send_state`.
The send state is managed by its `with_event_id` method which now receives an
`Option<OwnedEventId>` (prev. `OwnedEventId` directly):
* If it's `Some(_)`, then it means we got a successful response from the server
after the sending, so the send state is set to `Sent`,
* If it's `None`, then it means we got an error response from the server, so
the send state is set to `SentFailed`.
(A small change: The method `TimelineInner::add_event_id` has been renamed
`TimelineInner::update_event_id_of_local_event`.)
The `Timeline::send` still returns a `Result`, but the server's response is
passed to a new method: `TimelinerInner::handle_local_event_send_response`
(it's logically named after the `handle_local_event` method), which is
responsible to call `TimelineInner:update_event_id_of_local_event` (which was
previously called directly by `Timeline::send`).
And `TimelineInner::update_event_id_of_local_event` was already calling
`LocalEventTimelineItem::with_event_id`. So everything works like a charm here.
The local send state is closely managed by `LocalEventTimelineItem`, I hope it
will avoid state breakage as much as possible.
The sliding sync logic has another room type called `SlidingSyncRoom`.
This room type stores a limited amount of events in something called
`AliveRoomTimeline` in a in-memory cache. This room also gets persisted
to the store via another room type called `FrozenSyncRoom`.
These types are used when clients restore their view of the room, i.e.
when the user enters the room to look at messages.
When the client enters a room, a `Timeline` object will be created, but
it will be initially populated with events coming from
`AliveRoomTimeline`.
In a hot potato contest of who should be responsible to decrypt injected
events, everybody claims to be allergic to potatoes.
This patch modifies the `Timeline` constructor that populates the
timeline with events from the `AliveTimeline` to retry decryption on the
events that the `AliveTimeline` pushes into the `Timeline`.
Note that, because the `AliveRoomTimeline` never replaces encrypted
events with decrypted ones, every time the client enters/exits the room
events well get re-decrypted.
This is an omission from the spec where the subkeys are supposed to be
signed by the master key, but signing the master key wasn't mentioned.
This is still fine, since we always treated the master key and their
subkeys as a whole and never accepted one without the others.
To create an event next to the server, the event must be accompanied by
a transaction ID. When the server receives the event creation request, it
responds with a new event ID (which we will name `event_id_1`). Later on, when
a sync is done to the server, the server may respond with the transaction ID
and a new event ID (which we will name `event_id_2`).
The transaction ID is used to update the state of the local event. The event
ID received from the sync may ideally be the same as the one received by the
creation request's response.
Knowing that…
`EventTimelineItem` had a field: `event_id: Option<OwnedEventId>`. It was
`Some(event_id)` where `event_id` is the event ID responded by the server when
an event was created, i.e. it's `event_id_1`; otherwise it was `None`. This is
never `event_id_2`. Why?
Because `EventTimelineItem` has an other field: `key: TimelineKey`, which
represents the following states:
* `TimelineKey::TransactionId(OwnedTransactionId)`,
* `TimelineKey::EventId(OwnedEventId)`, where the event ID is `event_id_2`!
So it becomes obvious that `event_id_1` should be part of `TimelineKey`, not
`EventTimelineItem`. `TimelineKey` is where this transaction ID and event ID
logic is managed.
This patch updates `TimelineyKey::TransactionId` to be defined as:
* `TimelineKey::TransactionId { txn_id: OwnedTransactionId, event_id: Option<OwnedEventId> }`.
Why an `Option`? Because before receiving `event_id_1`, we may still want to
create a `TimelineKey`, the API must reflect that state.
So basically, `EventTimelineItem.event_id` is moved inside
`TimelineyKey::TransactionId`.
… rather than prepended, or added at a specific point as replacement for
an existing timeline item.
Previously, we would log lots of useless tracing events when retrying to
decrypt UTD items.
This patch fixes https://github.com/matrix-org/matrix-rust-sdk/issues/1379/.
This is similar to https://github.com/matrix-org/matrix-rust-sdk/pull/1197/.
We need to close the `OlmMachine`. Problem, `napi-rs` doesn't allow methods to
take ownership of `self`, so the following code:
```rs
fn close(self) {}
````
isn't allowed. There an [`ObjectFinalize`](https://docs.rs/napi/latest/napi/bindgen_prelude/trait.ObjectFinalize.html)
trait, but it's only called when the JS value is being collected by the GC.
It's not possible to force call `finalize` here.
This patch then introduces a new type:
```rs
enum OlmMachineInner {
Opened(matrix_sdk_crypto::OlmMachine),
Closed
}
```
that derefs to `matrix_sdk_crypto::OlmMachine` when its variant is `Opened`,
otherwise it panics. Calling the new `OlmMachine::close` method will change the
inner state from `Opened` to `Closed`.
Ideally, I wanted to throw an error instead of panicking, but `napi_env`
(used to build an `Env`, used to throw an error) is neither `Sync` nor `Send`.
Honestly, my knowledge of NodeJS and NAPI is too weak to implement `Sync`
and `Send` for `*mut napi_env__` safely. Especially because it can be executed
from various threads within `async fn` functions, that are driven by Tokio in
`napi-rs`. So, yeah, for the moment, it panics!
This moves the caches we have for tracked users out of the Sled, and
IndexedDB, CryptoStore into the Store struct. The Store struct is a
wrapper for the CryptoStore trait, so this is the natural place where
these caches should live.
Co-authored-by: Jonas Platte <jplatte@matrix.org>
Before we'd push one empty entry at a time, which lead to a lot of vecdiff updates pushed. With this
we now only push one event after the entire operation set has been applied on the fresh
or cold view.
A mismatch between the recorded Ed25519 and Curve25519 keys of the room
key and the identity keys of a Device used to be just a warning.
Considering that many clients will aggressively try to hide E2EE related
warnings let's upgrade this clear error into a full blown decryption
failure.
Co-authored-by: Denis Kasak <dkasak@termina.org.uk>
We might have a lot of tracked users, clients will usually have 1-10
thousand tracked users. This might slow down client startup if there are
so many tracked users.
We don't need the tracked users until we sync or try to send a message,
so load them lazily when they are first needed.
The conventional commit specification recommends the "docs" type for
documentation related changes. This also matches the initial usage of
this type in our commit history.
While the default git-cliff config does specify a regex "^doc", where I
suspect our types have been taken from, this regex actually allows "doc"
as well as "docs" for the type.
Nevertheless, let's specify that we're using the one recommended in the
spec.
I had no idea what a tracked user was, so I've attempted to clarify this
somewhat for future readers.
Co-authored-by: Damir Jelić <poljar@termina.org.uk>
Instead of putting all request fields inside a single `body` JSON-
encoded string field, there is now more types. There is less need to call
`JSON.parse`, and the type system will be able to catch more things.
For example, `ToDeviceRequest` now has 2 more fields: `event_type` and
`txn_id`.
This solves the issue of correlating different log lines that are talking
about the same request/response cycle. Plaintext logs would show this as
a flat structure, while something like Jaeger would show you a nice
tree. This allows you to reconstruct the tree in plaintext logs.
Various smaller fixes on sliding-sync for elem-x:
- Infrastructure and smoke test only for integration testing sliding-sync
- Expose Pop and Clear on VecDiff
- Expose adding views during sliding sync live runtime incl integration tests
- bug fix on replacing only in-window-items in the insert-code (was leading to a wrong list setting), including integration test
Required when `PaginationOptions` must implement `Send`,
e.g. with `tokio::runtime::Runtime::spawn`.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
When updating a device after we receive new device keys from a
/keys/query call we implicitly check that the user and device id match
since we need to fetch the device using the mentioned ids.
This moves the check into the update method as well to make it more
explicit. We also prevent upgrades of the Ed25519 key since we don't
support any different key type anyways.
The sliding sync proxy seems to have started to omit the e2ee extension
completely if no changes happened to the one-time key counts.
Our sync response processing assumed that, if there are some to-device
events that need to be passed to the OlmMachine, then we'll have some
data in the e2ee extension of the response as well. In other words, it
wouldn't pass in the to-device events to the OlmMachine if the e2ee
field of the sync response is `None`.
Since the assumption isn't true, we're going to pass in default values
for the one-time key counts when the e2ee extension field of the
response is `None`.
This is mainly done so we create a span with the name of the function,
since we can filter logs by the name of the span we're now able to
enable logs for this method only.
Previously, when we found a since-token in the frozen state, we'd always activate the to-device extensions
regardless of whether the user had activated it in their builder or didn't. With this change we are only
setting the since parameter from cache if the user actually activated the to-device extension.
- Don't actually fire off a request when the top of the timeline has
already been reached
- Allow making multiple requests without removing the loading indicator
in between
Index ranges are inclusive, but our loop would stop one short. This particuarly
tricky when the selective view is moved, as we didn't properly invalidate all items.
It's not a hard-and-fast-rule that Javascript exceptions should be `Error`
instances, but it's certainly a convention, and one that is going to reduce
surprises if we follow.
Add a second full-sync-up mode to sliding sync. Previously - and still the default for backwards compatibility, but now named `PagingFullSync` - was to page through the list by the page-size, de-validating the previous page of items. The newly added `GrowingFullSync` instead grows the window by the given number `batch_size` per request, starting and keeping it from `0`. In the latter we might be pushing more data over the connection and are slightly slower, but the top always stays active and thus reactive to changes.
Furthermore the developer can now configure an optional maximum number to grow/paginate the full-sync up to (so stopping before actually having reached `count` whatever is smaller). This is already exposed via FFI, too.
- [x] add new full-sync-mode
- [x] add limited sync-up mode (similar to full sync but only to a limit `n`)
- [x] implement sync-up in jack-in
Sliding Sync updates
- expose timeline listener via FFI on slidingsyncroom, too - returning a stoppable spawn
- connect room timeline and sliding-sync-room timeline together
- increase HTTP timeout on jack-in
- add jack-in support for actual timeline items.
We automatically request room keys to be forwarded from our other trusted devices if a decryption failure happens because the room key is missing.
This patch introduces automatic room key requests for decryption failures if the room key is available but has been ratcheted forward. In other words, we will now request a better version of the given room key automatically as well.
Co-authored-by: Damir Jelić <poljar@termina.org.uk>
`open_with_db` also has the passphrase parameter without mentioning it
in its name. The old name also sounded like the passphrase was required
when it's actually optional.
Clarify that the JSON-encoded `toDeviceEvents` passed to
`OlmMachine.receiveSyncChanges` must be the list of events themselves,
instead of a wrapper object that contains the event list.
Signed-off-by: Andrew Ferrazzutti <andrewf@element.io>
Tokens are not necessary for the restoration of the crypto/store, only
the meta. Required for OpenID Connect support where we need the tokens to get the
meta.
When the verification support was initially bound, Uniffi did not
support objects (the OlmMachine) returning other objects (our
verification objects). Instead we converted all the verification objects
into pure data structs which the other side had to poll for changes.
Now that Uniffi supports returning objects we can refactor this and make
the API on the other side the same as on the Rust side.
With these changes, the user of sliding sync can configure the SlidingSync-Builder to store and recover the state from storage. For that they should call cold_cache("my-sliding-sync-key") with their preferred storage location during setup time on SlidingSyncBuilder. If a cached version is found under that key, the internal state of the rooms-list as well as each view found (identified by their name) will be set from storage immediately - allowing the user to show the last cached state. The Builder then also uses the same key to store its latest state after every update received from remote.
👉 Note on room list:
As we store a disconnected state, we are saving all RoomList entries as either Empty or Invalidated. This allows for an easier updating when the server comes back with results, as we don't track the ranges in cache - in the view of the server, we haven't seen anything yet.
👉 Note on timeline events:
This does store all existing timeline events - initial and seen during the session (as we keep track of them right now) - and will recover that state. However, as we can't be sure wether we have gaps in the timeline, the timeline items are reset upon receiving updates for them from the server.
👉 Note on (full-sync-)view:
While we are caching the server returned results (view list, room count and room info) as is, we are not caching the internal state (whether we are catching up) nor the ranges (as they are likely to be out of date) - those will be acting the same way you configured them, just with preloaded results. So for a full-sync-view, it will still do requests in batches and replace the corresponding set of rooms. Which could mean you might see the same room appear twice, though the cached one would be marked as invalidated. This might be problematic if the list the UI shows is longer than the batches fetched, but would be resolved quickly when catching up.
👉 On recovery:
When the sliding sync receives a M_UNKNOWN_POS, indicating the server has expired our session, sliding sync now transparently retries with up to three times to restart the sliding sync with full set of extensions and the latest views at their existing windows, the current room state is held. For full sync this starts a new sync-up with the existing room list staying intact. This also works from the offline at start.
- matrix_sdk_base::sync::SyncResponse is the internal representation that
we can update to account for sliding sync
- matrix_sdk::sync::SyncResponse is `/v3/`-specific and should not change
* Move swift build scripts into xtask (#1201)
* fix(ffi): use target_path from `cargo metadata` rather than guessing
* ci(ffi): install necessary target arch for build-framework test
* feat(xtask): copy to target without rsync.
As it appears, the first sync for a room might not include the
encryption state, so we cannot set the encryption state to be synced on
incoming syncs. That way, we always fetch the encryption state manually.
* split logger into platform specific implementations
* logging support for android
* use platform independent wrapper for uniffi:exports
* activate colors for ansi-terminals
It turns out that Google Chrome refuses to initialise the wasm via the
synchronous `WebAssembly.Module` constructor, complaining that it is too
big. To be fair, it has a point.
Anyway, that means we need to provide a way to load the wasm
asynchronously. So, we introduce an `initAsync()` function which applications
can call before they do anything else, to load the wasm in the background.
If the app *doesn't* call `initAsync()`, then we load the wasm synchronously
the first time a function that accesses the wasm is called.
This new `OlmMachine.close` forces to drop/close the `OlmMachine` without
waiting on the JavaScript garbage collector to collect it.
`wasm-bindgen` generates the following JS glue code:
```js
close() {
const ptr = this.__destroy_into_raw();
wasm.olmachine_close(ptr);
}
```
And, `__destroy_into_raw` looks like this:
```js
__destroy_into_raw() {
const ptr = this.ptr;
this.ptr = 0;
OlmMachineFinalization.unregister(this);
return ptr;
}
```
It unregisters itself from the `FinalizationRegistry` correctly. We are
protected from a double-free.
Surprisingly, `indexed_db_futures::IdbDatabase` is not closed when dropped.
Hopefully, there is a [`IdbDatabase::close(&self)`][close] method, which calls
`web_sys::IdbDatabase.close`, aka [`IDBDatabase.close`][websys-close], so let's
use it!
`IDBDatabase.close` returns immediately and closes the connection in a separate
thread. The connection isn't actually closed until all transactions created
using this connection are complete. No new transactions can be created for this
connection once this method is called. Methods that create transactions throw
an exception if a closing operation is pending.
[close]: https://github.com/Alorel/rust-indexed-db/blob/8c106eb418aecdba2f3fde80d91a4673a875fdf6/src/idb_database.rs#L73-L77
[websys-close]: https://developer.mozilla.org/en-US/docs/Web/API/IDBDatabase/close
I have tried for hours and have concluded it is probably not possible
even with GATs to have RawEvent borrow its inner value.
It is also not a commonly-used feature, so removing the unnecessary
clones is not that important.
This patch adds a couple more states to the SAS verification state
machine. Namely we add the following states:
* KeySent
* KeysExchanged
This prevents users from confirming that the short auth string matches
before they have sent out a m.key.verification.key event, which is
necessary for the other side to present the short auth string.
The KeysExchanged state functionally replaces the KeyReceived state.
Meaning that the short auth string can only presented once we
transitioned into the KeysExchanged state.
We can transition into the KeysExchanged state from the KeyReceived
state or from the KeySent state, depending on if we first receive a
m.key.verification.key event or if we first send out our own
m.key.verification.key event.
This means that, if the transition into the KeysExchanged state happens
through the KeySent state, users won't be able to tell if the transition
happened. In other words, listening to `m.key.verification` events
doesn't work anymore, users will need to use the new `Sas::changes()`
API to listen to a stream of state changes.
`wasm_bindgen::JsValue::(from|to)_serde` now emit warnings because of a
circular dependency. To solve this problem, this patch now uses `gloo-utils`,
see https://rustwasm.github.io/wasm-bindgen/reference/arbitrary-data-with-serde.html#an-alternative-approach---using-json.
Ideally, we would like to use `serde_wasm_bindgen` but the behaviour isn't
the same. `gloo-utils` serializes and deserialized through JSON, whilst
`serde_wasm_bindgen` manipulates the `JsValue` directly which can be better or
worst dependending of the case. This patch conserves the JSON approach as it
was the previous and tested one.
By asking `wasm-bindgen` to generate JS glue code for `WeakRef` support,
it removes the need to call `free` manually to free objects, thus we reduce
potential memory leaks inside Rust. See https:// rustwasm.github.io/docs/wasm-
bindgen/reference/weak-references.html to learn more.
It uses [`FinalizationRegistry`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/FinalizationRegistry)
under the hood, I reckon it's quite common and fits into Matrix's clients needs
in terms of browser support.
While technically a type constructor can return a `Promise`, it's not
considered as idiomatic JavaScript to do that. We are changing `new OlmMachine`
to `OlmMachine.initialize`. The rest of the code is strictly the same.
Add general extension framework for sliding sync, implementing the e2ee, to-device and account-data extensions as per existing proxy implementation. Add a new (ffi exposed) function to use activate the extensions.
Also extends jack-in to have permanent login and storage support now, rather than posting an access token and expose messages inside the sliding-sync layer to actually use the decrypted messages if given. Contains a lot of fixes around these aspects, too, like uploading any remaining messages from the olm-machine on every sliding-sync-request or processing even if no room data is present (which can happen now as processing only extensions might takes place).
This patch adds a way for users to listen to changes in the state of a
SAS verification.
This makes it much more pleasant to go through the verification flow and
incidentally easier to document it.
Now that we're not scoping the room keys by the Curve25519 sender key
we're opening the door of multiple devices trying to insert the same
room key into our store.
This patch changes our logic so we only store room keys from an
m.room_key event if we don't have one already or if the new key is
a better version of the one we already have.
This mostly assumes that the first room key with a given session id
is coming from the creator of the room key.
It was very confusing that matrix_sdk::store::make_store_config's first
argument was either a path or a database name depending on the build
configuration. ClientBuilder::{sled_store, indexeddb_store} are also
easier to use.
By default, the new `experimental-nodejs` feature is
disabled. However, when enabled, it uses our own fork of
`indexed_db_futures` that make it work on Node.js, and so this entire
`matrix-sdk-indexebdb` crate.
The crypto crate consumes to-device events, if needed decrypts them, and
in the end reserializes while zeroizing secrets.
It needs to do this reserialization cycle as loslessly as possible, this
is why we have a bunch of BTreeMaps lying in events and contents that
capture unknown fields.
The Deserialize implementation of ToDeviceEvent<C> put the event type
into this BTreeMap but the Serialize implementation figures it out from
the EventType trait of the content.
This patch simplifies things by putting the event type into
ToDeviceEvent<C>, this also means that we can just derive the Serialize
implementation for ToDeviceEvent<C>.
Inspired by the changes in #1013 I was thinking about the use case for `sync_*` and how we handle error cases. Most notably while we give the callback the option to stop the loop, we don't really give an indication to the outside, how to interpret that cancellation: was there a failure? should we restart?
Take e.g. a connectivity issue on the wire, we'd constantly loop and just `warn`, what you might or might not see. Even if you handle that in the `sync_with_result_callback` and thus break the loop, the outer caller now still doesn't know whether everything is honky dory or whether they should restart.
This Changes reworks that area by having all the `sync` return `Result<(), Error>`, where `()` means it was ended by the inner callback (which in `sync()` never occurs) or `Error` is the error either the inner `result_callback` found or the that was coming from the `send` in the first place. Thus allowing us to e.g. back down to sync as it was a dead wire or restart it if there was only a temporary problem. Making all that a just a bit more "rust-y".
… for matrix-sdk, matrix-sdk-base, matrix-sdk-common and matrix-sdk-crypto.
matrix-sdk-indexeddb as well as the JS bindings and wasm_command_bot are
left as-is because they will likely always require JS.
* starting with jack-in
* starting by flying tui
* connecting to real server, showing info
* add .env to gitignore
* infrastructure for tests
* display loading time for syncv2
* minor design updates
* initial sync
* finalise first edition of sliding sync
* directly link to sliding sync and show rooms list
* nicer UI, toggle logs
* passing through sliding sync homeserver
* separate syncs and disable v2 autostart
* selecting rooms
* nicer view
* configurable batches and more default needed events
* selecting rooms
* calculate and show status info per room
* precalculated room stats
* restructure code to allow for cancellation of streams
* finish up merge updates
* fix calculation error in room list len
* cleaning up system flow
* fixing sync up
* new multi-view API
* move sliding sync in separate module
* fixing format
* adding and clearning views
* expose filters and timeline limits
* renamed
* adding room subscriptions to sliding sync
* update summary
* live fetching and subscriptions in jack-in
* subscribe to selected room
* starting to switch to tuireal - using example
* status bar and first linkup
* re-adding rooms
* implementing port for customised update event
* showing details and timeline
* fix formatting
* cleaner UI, updating details quickly
* make it green
* implement other Ops
* proper handling of invalidation
* saving sliding sync results to db, too
* saving new prev_batch field if given
* split events and timeline
* cleaning up
* live updates
* upgrading to latest ruma and matrix-sdk
* Update tui-logger to fix the broken build
* fixing latest ruma sliding-sync-branch updates
* feat: first set of ffi sliding sync
* expose sliding sync via FFI
* implement un-/subscribe
* implement view state updates
* updating to latest JSON format and ruma update
* implementing room selecting for new data model
* fixing room selection
* fixing feature flag requirements for sliding-sync
* fixing style, clippy and docs
* style(sliding-sync): fixing rust format styles
* fix(ffi): fixing sliding sync merge failure
* fix(jack-in): update jack-in to latest ruma
* fix(sliding-sync): need to have a version set before polling proxy
* expose sliding sync builder over ffi
* add SlidingSyncViewBuilder
* add forgotten changes on sdk itself
* new file logger feature on jack for deeper logging
* fix(http-client): log the raw request we are sending out
* feat(sliding-sync): better logging
* fix(sliding-sync): switch to full-listen after reaching live state in full-sync and make sure we replace the correct entries
* feat(ffi): expose sliding sync view ranges
* fix(ffi): fixing sliding sync start_sync loop to actually loop
* feat(sliding-sync): allow lookup of room data
* feat(sliding-sync-ffi): fetching name of room
* feat(ffi): expose unread_notifications of rooms
* feat(ffi): stoppable spawn for sliding sync
* fix(ffi): expose has_unread_notifications on room
* feat(sliding-sync): latest room message
* fix(sliding-sync): update to latest ruma layout
* doc(sliding-sync): adding docs to builder fns
* feat(sliding-sync): extra signal on the view to inform about general updates
* fix(sync-v4-ffi): expose new callbacks via ffi
* fix(sliding-sync): reduce default required states to make things faster
* fix(sliding-sync): fix build params
* feat(jack-in): extended instrumentation
* fix(sliding-sync): unbreak faulty feature-gate
* fix(sliding-sync-ffi): mut where mut is due
* fix(sdk): allow separate homeserver on every request to unbreak using send on client while in sliding sync on a proxy
* fix(jack-in): update to latest dependencies, that work
* feat(ffi): helper to patch sliding sync room into regular room
* style(jack-in): cargo fmt
* fix(sliding-sync): Update to latest ruma changes
* fix(sliding-sync): fix missing FFI updates to latest ruma
* feat(sliding-sync)!: simplify stream cancellation, cancel ffi sync if already existing
* fix: timeline backwards pagination must work without synctoken
* fix(sliding-sync): clarify order of messaes in alive TL; pick correct last item
* fix: update view delegate api for clarity
* style(jack-in): fix cargo warnings
* feat(sliding-sync): update room details
* fix(sliding-sync): only update room info selectively when given
* fix(sliding-sync-ffi): convert and store counts as u32, check against 0 for has notificaitons
* style: cargo fmt, file endings and a few other minor style fixes
* docs(jack-in): improving CLI and Readme
* feat(sliding-sync): allow setting of required event_states on viewbuilder
* style(sliding-sync): docs and minor fixes
* style(sliding-sync): various clippy fixes
* style(jack-in): clippy suggestions applied
* fix(sliding-sync): Delegate becomes observer
* test(sdk): adding test for request config
* docs: Fixing copyright header
* style(ffi): Nicer naming of params for observer
* fix(ffi): sliding sync is not optional for now
* fix(sdk): remove superflusous tracing instrumentation
* fix(sdk): use structured logging
* fix(jack-in): removed unneded log import
* fix(jack-in): use server_name rather than deprecated user_id on ClientBuilder
* style: typo and clippy
* style(sliding-sync): clippy and formatting
* fix(sliding-sync): cleaning up minor syntax issues
* fix: remove unneded feature-definition section
* fix(sliding-sync): minor fixes as per review
* fix(sliding-sync): Make Builders owned
* fix(sliding-sync): more minor style improvements
* fix(sliding-sync): minor style improvements
* fix(sliding-sync): remove homeserver from RequestConfig, use specific internal fn instead
Co-authored-by: Stefan Ceriu <stefanc@matrix.org>
Merge pull request #1013 from matrix-org/ismail/logout
Add logout facilities to `client` and helpers to ffi for tracking the soft-logout issued by a server. Further more this adapts the `sync_with_callback` by adding a new `sync_with_result_callback` that hands the entire `Result` to the callback.
This patch removes the `decode_image` (default) feature for this
`matrix-sdk-qrcode` crate.
First reason is that `rqrr` panics for some particular QR codes, and
we don't want to panic.
Second reason is that it's not a feature that is used. For Element on
iOS and Android, it's very unlikely that every frame from the camera
will be sent to `matrix-sdk-qrcode` to see if it can be decoded. So
it's obvious that an external library is used to read the bytes from
the QR code, that are then sent to `matrix-sdk-qrcode`. For Element on
Web, it's basically the same argument.
This feature is actually used only in our tests to ensure the
generated QR code is valid, but it sometimes fails due to `rqrr`
(cf. First reason).
`AnyDecryptedOlmEvent::Custom` contains at least 440 bytes, while the
second-largest variant (`RoomKey`) contains at least 0 bytes. Let's
box `Custom` so that the size of `AnyDecryptedOlmEvent` stays low.
A room ID can no longer be invalid, it's just a string representing
either a `EventId` (we can validate that but…) or a `TransactionId`
(which is an opaque string, so it can be anything).
This patch updates `QrVerification::new_cross`, by passing a flow ID
as an owned `String` to `VerificationData`, thus removing a panic, and
allowing QR code verification to happen outside a room.
This patch updates `VerificationData` to receive a flow ID,
represented as an owned `String`, instead of an `OwnedEventId`. Why?
Because QR code verification can happen outside a room. In such
scenario, there is no event ID, but a transaction ID, unified behind
the `matrix_sdk_crypto::FlowId` enum. `VerificationData` doesn't
really care about that details. Proof is that `QrVerificationData`
receives an owned `String`, which is then casted into an
`OwnedEventId` to match this API properly; but at the top, it just
receives a string.
This patch brings also a little bit of clean up while editing code
around.
Merge pull request #979 from FlixCoder/cleanup
Fixes a problem, where non-stripped information of the room had predecence in all scenarios of the memory-, sled- and indexeddb-store, thought according to the spec, we prefer the full info only until we've received new stripped info (from another invite). This fixes that to be in line with the spec by removing stripped data when we see a full-event and keep stripped info preferred if found (so, after you joined, stripped data is removed and when you knocked or are invited again, it is preferred), without ever deleting full data. It also adds nice unit- and integration tests to ensure this works as intended.
This patch adds a method to format a list of emojis in a a terminal
friendly way.
This method was borrowed from weechat-matrix but it's also quite useful
in our own emoji verification example.
Since this example prints out messages to stdout, it becomes quite hard
to see when we ask for user input if all the logging is going on.
This patch adds a standard verbosity flag which disables all logging by
default.
Previously, when both a possibly-redacted timeline event handler and a
non-redacted timeline timeline event handler would apply to multiple
events in a sync response, they would individually run for every event
in order. With this change, they will instead both be called
for one event before the next is processed.
The use of `io::Read` wasn't helping since we had to buffer the whole
file in memory anyways, and we are unlikely to get around that in the
near future.
Inbound group sessions would fail to be deserialized if there was a
forwarding curve chain, this patch fixes it so we go through base64 when
we deserialize this field.
This patch introduces a new `pack` NPM script, which runs `wasm-pack
pack` behind the scene.
This patch modifies the `publish` NPM script to run the `pack` script
as a pre-script (so… in the `prepublish` script).
Finally, this patch no longer uses `$npm_execpath` as it doesn't work
on Windows. It should be `%npm_execpath%`. It's not obvious to make
scripts interoperable, so we will stick with `npm` for now.
The `OlmMachine` constructor now has 2 more optional arguments:
`store_name` and `store_passphrase`, to use Indexed DB as the backend
to store the Olm machine keys, rather than having an in-memory Olm
machine.
Node.js is used to test our binding. Indexed DB is absent of
Node.js. So, firstly, we are using the `fake-indexeddb` JavaScript
library to mimic the same API inside Node.js. And, secondly, we use
our own fork of the `indexed_db_futures` Rust crate to provide add
support for Node.js too ([PR is
here](https://github.com/Alorel/rust-indexed-db/pull/11)). It
basically looks in the Node.js global environment if the `indexedDB`
getter is present, just like it is already done for `Window` on any
browser, or `WorkerGlobalScope` for workers.
This patch updates the `publish` NPM script to explicitly run
`wasm-pack pack` to create the NPM package in the `pkg/`
directory. This patch also updates the Github Action workflow for the
`matrix-sdk-crypto-js` release, to create a new Github Release, with
the NPM package attached as an asset file.
This patch adds a new `npm run publish` script that:
1. Run `npm run prepublish` (which runs the `build` and `test` scripts),
2. Run `wasm-pack publish`.
Note 1: The `prepublish` script is using the `$npm_execpath`
environment variable instead of just “`npm`”, so that if someone is
using `yarn` or another JavaScript package manager, it _should_ work
(not tested yet).
Note 2: `wasm-pack publish` is run without running `wasm-pack login`
before that. _But_ we are updating the registry URL in the NPM
configuration file in the Github Action workflow, see below.
This patch then creates a new Github Action workflow that is triggered
when a new tag of the form `matrix-sdk-crypto-js-v[0_9]+.*` is
pushed. This workflow runs `npm run publish` basically, but before
that, it updates the NPM configuration file by setting a value for
`//registry.npmjs.org/:_authToken`. Thus, running `wasm-pack login` is
not necessary.
Implement the `Attachment.encrypt` and `Attachment.decrypt` methods,
along with its `EncryptedAttachment` companion.
The trick is to avoid copies as much as possible. Instead of dealing
with `Uint8Array` as I've initially done, `&[u8]` and `Vec<u8>` is
passed instead. `wasm-bindgen` is smart enough to do as few copies as
possible (from JavaScript to Wasm's memory is required, and we don't
want to do more), while typing everything as `Uint8Array`.
`EncryptedAttachment.encryptedData` returns a copy of the data
everytime it is called, and that's the last copy I'm not happy with.
… rather than deserializing at an enum type and taking the same branch
for getting a different enum variant¹ as failed deserialization.
¹ which should be impossible anyways
Initial state events aren't actually received through sync, they're sent from
the client to the server when creating a room.
This change makes it impossible to register event handlers for initial state
events. Previously, this was allowed, but the event handler was never called.
This patch adds some more stronger guarantees that received room key
events and other similarly security critical event types are only
handled if they have been received over a secure Olm channel.
… by waiting for the corresponding event confirming the action. Affects:
* Creating a room
* (Re-)Joining a room
* Leaving a room
* Accepting an invitation
* Rejecting an invitation
This patch delegates the decision making if a session is better to
vodozemac. Vodozemac has better insights if a group session can be
considered to be better.
This allows smoother development as a regular 'cargo check' will
validate most of the code. It also makes rust-analyzer work for the
crate without any configuration.
… instead of through `compile_error!` invocations. This helps avoid
unhelpful errors from the unsupported feature configuration by aborting
compilation earlier.
This patch ensures that the received secret has been sent by one of our
verified devices. We already ensure so for the secrets we import, since
we check that the cross signing keys match the public keys of our own
user idenity.
No other secrets get imported by this method but it could quite easily
become an issue if we start accepting more secret types.
This fixes a bug where we would lose already generated to-device
requests carrying an m.room_key when the client gets restarted.
This only happens if the request was generated after the initial
share of the room key. Such requests get generated if a new device
or member join the group.
This functionality is important to allow clients to log in using an OIDC server. The OIDC server returns only an access token, the client needs to fetch the user ID and device ID from the Matrix server separately.
This lives in the bindings for now since OIDC support in Matrix is being actively developed.
If the last sync was less then a 1s ago we wait for a 1s, it doesn't
make sense to wait an additional second on an error. Also the stream
sync api returns the error after a delay of 1s, which doesn't make
sense.
This patch creates one `inner` module for when `feature = "tracing"`,
and one for when `no(feature = "tracing")`. Then, let's expose
everything from `inner::*`.
This patch also replaces `Tracing.install` by `new Tracing`. In case
of `not(feature = "tracing")`, `new Tracing` raises an error.
The goal is to remove all the `#[cfg(…)]` annotations everywhere. Now
there is only 2 of them.
The patch updates the code to use `tracing_subscriber::reload`, so
that we get a `Handle` that can be used to modify the tracing at
runtime.
This patch also adds a new `tracing` feature.
This patch finally adds a test suite for the `Tracing` API.
The @matrix-org/rust team is happy to present to the wider public version 0.5 of the [`matrix-rust-sdk`][sdk], now available on crates.io for your convenience. This marks an important milestone after over half a year of work since the latest release (0.4) and with many new team members working on it. It comes with many bug fixes, improvements and updates, but a few we'd like to highlight here:
## Better, safer, native-er crypto
One of the biggest improvements under the hood is the replacement of the former crypto core written in C, libolm, with the fully rustic, fully audited [Vodozemac][vodozemac]. [Vodozemac][vodozemac] is a much more robust new implementation of the olm crypto primitives and engines with all the learning included and pitfalls from the previous C implementation avoided in a freshly baked, very robust library. With this release both matrix-sdk-crypto and matrix-sdk (when the `e2e-encryption` is enabled, which it is by default) use vodozemac to handle all crypto needs.
## Stores, stores, stores
This release has also seen a major refactoring around the state and crypto store primitives and implementations. From this release forward, the implementation of the storage layer is truly modular and pluggable, with the default implementation (based on `sled`) even being a separate crate. Furthermore this release has also additional support for the [`indexeddb`-storage][indexeddb] layer for in-browser WASM needs. As before, the base still ships with an in-memory store, but if none of them fits your needs, you are very much invited to build your own - we recommend looking at the implementation of the existing [`matrix-sdk-sled`][sled] to understand what is needed.
We've further extended and cleaned up the storage API and hardened the two existing implementations: both `sled` and `indexeddb` will now encrypt _all metadata_, including all keys, if a passphrase is given. Even a core dump of a database won't show any strings or other identifiable data. Both implementation also hold a database version string now, allowing future migrations of the database schema while we move forward.
## WebAssembly
It already came through in the previous paragraph: we now fully support `wasm` as a primary target for the main sdk and its dependencies (browser for now). While it was already possible to build the SDK for wasm before, if you were lucky and had the specific emscripten setup, even crypto didn't always fail to compile, our move to vodozemac frees us from these problems and our CI now makes sure all our PRs will continue to build on WebAssembly, too. And with the `indexeddb`-store, we also have a fully persistent storage layer implementation for all your in-browser matrix needs built in.
## More features
Of course a lot more has happened over the last few months as we've ramped up our work on the SDK. Most notably, you can now check whether a room has been created as a `space`, and we offer nice automatic thumbnailing and resizing support if you enable `image`. Additionally, there is a very early event-timeline API behind the `experimental-timeline`-feature-flag. Just to name a few. We really recommend you migrate from the older version to this one.
## Process and Workflow
We've also ramped up our CI, parallelized it a lot, while adding more tests. In general we don't merge anything breaking the tests, thus our `main` can also be considered `stable` to build again, though things might be changed without prior notice. We recommend keeping an eye out in our [team chat][chat] to stay current.
### Versions
This and all future releases of matrix-sdk adhere to [semver][semver] and we do our best to stick to it for the default-feature-set. As you will see only the main crate `matrix-sdk` is actually versioned at `0.5`, while other crates - especially newer ones - have different versions attached. We recommend to sticking to the high-level matrix-sdk-crate wherever reasonable as lower level crates might change more often, also in breaking fashion.
With this release all our crates are build with `rust v2021` and need a rust compiler of at least `1.60` (due to our need for weak-dependencies).
### Conventional Commit
We also want to make it easier for external parties to follow changes and stay up to date with what's-what, so, with this release, we are implementing the [conventional commits][cc] standard for our commits and will start putting in automatic changelog generation. Which makes it easier for us to create comprehensive changelogs for releases, but also for anyone following `main` to figure out what changed.
This is a rough migration guide to help you upgrade your code using matrix-sdk 0.5 to the newly released matrix-sdk 0.6 . While it won't cover all edge cases and problems, we are trying to get the most common issues covered. If you experience any other difficulties in upgrade or need support with using the matrix-sdk in general, please approach us in our [matrix-sdk channel on matrix.org][matrix-channel].
## Minimum Supported Rust Version Update: `1.60`
We have updated the minimal rust version you need in order to build `matrix-sdk`, as we require some new dependency resolving features from it:
> These crates are built with the Rust language version 2021 and require a minimum compiler version of 1.60
## Dependencies
Many dependencies have been upgraded. Most notably, we are using `ruma` at version `0.7.0` now. It has seen some renamings and restructurings since our last release, so you might find that some Types have new names now.
## Repo Structure Updates
If you are looking at the repository itself, you will find we've rearranged the code quite a bit: we have split out any bindings-specific and testing related crates (and other things) into respective folders, and we've moved all `examples` into its own top-level-folder with each example as their own crate (rendering them easier to find and copy as starting points), all in all slimming down the `crates` folder to the core aspects.
## Architecture Changes / API overall
### Builder Pattern
We are moving to the [builder pattern][] (familiar from e.g. `std::io:process:Command`) as the main configurable path for many aspects of the API, including to construct Matrix-Requests and workflows. This has been and is an on-going effort, and this release sees a lot of APIs transitioning to this pattern, you should already be familiar with from the `matrix_sdk::Client::builder()` in `0.5`. This pattern been extended onto:
- the [login configuration][login builder] and [login with sso][ssologin builder],
Most have fallback (though maybe with deprecation warning) support for an existing code path, but these are likely to be removed in upcoming releases.
### Splitting of concerns: Media
In an effort to declutter the `Client` API dedicated types have been created dealing with specific concerns in one place. In `0.5` we introduced `client.account()`, and `client.encryption()`, we are doing the same with `client.media()` to manage media and attachments in one place with the [`media::Media` type][media typ] now.
The signatures of media uploads, have also changed slightly: rather than expecting a reader `R: Read + Seek`, it now is a simple `&[u8]`. Which also means no more unnecessary `seek(0)` to reset the cursor, as we are just taking an immutable reference now.
### Event Handling & sync updaes
If you are using the `client.register_event_handler` function to receive updates on incoming sync events, you'll find yourself with a deprecation warning now. That is because we've refactored and redesigned the event handler logic to allowing `removing` of event handlers on the fly, too. For that the new `add_event_handler()` (and `add_room_event_handler`) will hand you an `EventHandlerHandle` (pardon the pun), which you can pass to `remove_event_handler`, or by using the convenient `client.event_handler_drop_guard` to create a `DropGuard` that will remove the handler when the guard is dropped. While the code still works, we recommend you switch to the new one, as we will be removing the `register_event_handler` and `register_event_handler_context` in a coming release.
Secondly, you will find a new [`sync_with_result_callback` sync function][sync with result]. Other than the previous sync functions, this will pass the entire `Result` to your callback, allowing you to handle errors or even raise some yourself to stop the loop. Further more, it will propagate any unhandled errors (it still handles retries as before) to the outer caller, allowing the higher level to decide how to handle that (e.g. in case of a network failure). This result-returning-behavior also punshes through the existing `sync` and `sync_with_callback`-API, allowing you to handle them on a higher level now (rather than the futures just resolving). If you find that warning, just adding a `?` to the `.await` of the call is probably the quickest way to move forward.
### Refresh Tokens
This release now [supports `refresh_token`s][refresh tokens PR] as part of the [`Session`][session]. It is implemented with a default-flag in serde so deserializing a previously serialized Session (e.g. in a store) will work as before. As part of `refresh_token` support, you can now configure the client via `ClientBuilder.request_refresh_token()` to refresh the access token automagically on certain failures or do it manually by calling `client.refresh_access_token()` yourself. Auto-refresh is _off_ by default.
You can stay informed about updates on the access token by listening to `client.session_tokens_signal()`.
### Further changes
- [`MessageOptions`][message options] has been updated to Matrix 1.3 by making the `from` parameter optional (and function signatures have been updated, too). You can now request the server sends you messages from the first one you are allowed to have received.
-`client.user_id()` is not a `future` anymore. Remove any `.await` you had behind it.
-`verified()`, `blacklisted()` and `deleted()` on `matrix_sdk::encryption::identities::Device` have been renamed with a `is_` prefix.
-`verified()` on `matrix_sdk::encryption::identities::UserIdentity`, too has been prefixed with `is_` and thus is onw called `is_verified()`.
- The top-level crypto and state-store types of Indexeddb and Sled have been renamed to unique types>
-`state_store` and `crypto_store` do not need to be boxed anymore when passed to the [`StoreConfig`][store config]
- Indexeddb's `SerializationError` is now `IndexedDBStoreError`
- Javascript specific features are now behind the `js` feature-gate
- The new experimental next generation of sync ("sliding sync"), with a totally revamped api, can be found behind the optional `sliding-sync`-feature-gate
## Quick Troubleshooting
You find yourself focussed with any of these, here are the steps to follow to upgrade your code accordingly:
### warning: use of deprecated associated function `matrix_sdk::Client::register_event_handler`: Use [`Client::add_event_handler`](#method.add_event_handler) instead
As it says on the tin: we have deprecated this function in favor of the newer removable handler approach (see above). You can still continue to use this `fn` for now, but it will be removed in a future release.
### warning: use of deprecated associated function `matrix_sdk::Client::login`: Replaced by [`Client::login_username`](#method.login_username)
We have replaced the login facilities with a `LoginBuilder` and recommend you use that from now on. This isn't an error yet, but the function will be removed in a future release.
### expected slice `[u8]`, found struct ...
We've updated the `send_attachment` and `Media` signatures to use `&[u8]` rather than `reader: Read + Seek` as it is more convenient and common place for most architectures anyways. If you are using `File::open(path)?` to get that handler, you can just replace that with `std::fs::read(path)?`
### no method named `verified` found for struct `matrix_sdk::encryption::identities::Device` in the current scope
Boolean flags like `verified`, `deleted`, `blacklisted`, etc have been renamed with a `is_` prefix. So, just follow the cargo suggestion:
```
|
69 | device.verified()
| ^^^^^^^^ help: there is an associated function with a similar name: `is_verified`
Ruma has been updated to `0.7.0`, you will find some ruma Events names have changed, most notably, the `AnySyncRoomEvent` is now named `AnySyncTimelineEvent` (and not `AnySyncStateEvent`, which cargo wrongly suggests). Just rename the import and usage of it.
### `std::option::Option<&matrix_sdk::ruma::UserId>` is not a future
You are seeing something along the lines of:
```
19 | if room_member.state_key != client.user_id().await.unwrap() {
| ^^^^^^ `std::option::Option<&matrix_sdk::ruma::UserId>` is not a future
|
= help: the trait `Future` is not implemented for `std::option::Option<&matrix_sdk::ruma::UserId>`
= note: std::option::Option<&matrix_sdk::ruma::UserId> must be a future or must implement `IntoFuture` to be awaited
= note: required because of the requirements on the impl of `IntoFuture` for `std::option::Option<&matrix_sdk::ruma::UserId>`
help: remove the `.await`
|
19 - if room_member.state_key != client.user_id().await.unwrap() {
19 + if room_member.state_key != client.user_id().unwrap() {
```
You are using `client.user_id().await` but `user_id()` is no longer `async`. Just follow the cargo suggestion and remove the `.await`, it is not necessary any longer.
**matrix-rust-sdk** leverages [UniFFI](https://mozilla.github.io/uniffi-rs/) to generate bindings for host languages (eg. Swift and Kotlin).
Rust code related with bindings live in the [matrix-rust-sdk/bindings](https://github.com/matrix-org/matrix-rust-sdk/tree/main/bindings) folder.
Developers can expose Rust code to UniFFI using two different approaches:
- Using an `.udl` file. When a crate has one, you find it under the `src` folder (an example is [here](https://github.com/matrix-org/matrix-rust-sdk/blob/main/bindings/matrix-sdk-ffi/src/api.udl)).
- Add UniFFI directivies as Rust attributes. In this case Rust source files (`.rs`) contain attributes related to UniFFI (e.g. `#[uniffi::export]`). Attributes are preferred, where applicable.
## Expose Rust definitions to UniFFI
### Check if the API is already on UniFFI
First of all check if the Rust definition you are looking for exists on UniFFI already. Most of exposed matrix definitions are collected in the crate [matrix-sdk-ffi](https://github.com/matrix-org/matrix-rust-sdk/tree/main/bindings/matrix-sdk-ffi).
This crate contains mainly small Rust wrappers around the actual Rust SDK (e.g. the crate [matrix-sdk](https://github.com/matrix-org/matrix-rust-sdk/tree/main/crates/matrix-sdk))
If the Rust definition is on UniFFI already, you either:
- find it in a `.udl` file like [matrix-sdk-ffi/src/api.udl](https://github.com/matrix-org/matrix-rust-sdk/blob/main/bindings/matrix-sdk-ffi/src/api.udl)
- see it marked with a proper UniFFI Rust attribute like this `#[uniffi::export]`
### Adding a missing matrix API
1. Unless you want to contribute on the crypto side, you probably need to add some code in the [matrix-sdk-ffi](https://github.com/matrix-org/matrix-rust-sdk/tree/main/bindings/matrix-sdk-ffi) crate. After you find the crate you need to understand which file is best to contain the new Rust definition. When exposing new matrix API often (but not always) you need to touch the file [client.rs](https://github.com/matrix-org/matrix-rust-sdk/blob/main/bindings/matrix-sdk-ffi/src/client.rs)
2. Identify the API to expose in the target Rust crate (typically in [matrix-sdk](https://github.com/matrix-org/matrix-rust-sdk/tree/main/crates/matrix-sdk). If you can’t find it, you probably need to touch the actual Rust SDK as well. In this case you typically just need to write some code around [ruma](https://github.com/ruma/ruma) (a Rust SDK’s dependency) which already implements most of the matrix protocol
3. After you got (by finding or writing) the required Rust code, you need to expose to UniFFI. To do that just write a small Rust wrapper in the related UniFFI crate (most of the time is **matrix-sdk-ffi**) you found on step 1.
4. When your new (wrapping) Rust definition is ready, remember to expose it to UniFFI.
It’s best to do it using UniFFI Rust attributes (e.g. `#[uniffi::export]`). Otherwise add the new definition in the crate’s `.udl` file. For the **matrix-sdk-ffi** crate the definition file is [api.udl](https://github.com/matrix-org/matrix-rust-sdk/blob/main/bindings/matrix-sdk-ffi/src/api.udl). **Remember**: the language inside a `.udl` file isn’t Rust. To learn more about how map Rust into UDL read [here](https://mozilla.github.io/uniffi-rs/udl_file_spec.html)
## FAQ
**Q**: I wrote my Rust code and exposed it to UniFFI. How can I check if the compiler is happy?\
**A**: Run `cargo build` in the crate you touched (e.g. matrix-sdk-ffi). The compiler will complain if the Rust code and/or the `.udl` is wrong.
**Q**: The compiler is happy with my code but the CI is failing on GitHub. How can I fix it?\
**A**: The CI may fail for different reasons, you need to have a look on the failing GitHub workflow. One common reason though is that the linter ([Clippy](https://github.com/rust-lang/rust-clippy)) isn’t happy with your code. If this is the case, you can run `cargo clippy` in the crate you touched to see what’s wrong and fix it accordingly.
This project and build script demonstrate how to create an XCFramework that can be imported into an Xcode project and run on Apple platforms. It can compile and bundle an [entire SDK](#Building-the-SDK), or only a smaller [Crypto module](#Building-only-the-Crypto-SDK) that provides end-to-end encryption for clients that already depend on an SDK (e.g. [Matrix iOS SDK](https://github.com/matrix-org/matrix-ios-sdk))
## Prerequisites for building universal frameworks
## Building
* the Rust toolchain
* UniFFI - `cargo install uniffi_bindgen`
* Apple targets (e.g. `rustup target add aarch64-apple-ios`)
*`xcodebuild` command line tool from [Apple](https://developer.apple.com/library/archive/technotes/tn2339/_index.html)
*`lipo` for creating the fat static libs
### Prerequisites for building universal frameworks
-`xcodebuild` command line tool from [Apple](https://developer.apple.com/library/archive/technotes/tn2339/_index.html)
-`lipo` for creating the fat static libs
### Building the SDK
```
sh build_xcframework.sh
cargo xtask swift build-framework
```
The `build_xcframework.sh` script will go through all the steps required to generate a fully usable `.xcframework`:
The `build-framework` task will go through all the steps required to generate a fully usable `.xcframework`:
1. compile `matrix-sdk-ffi` libraries for iOS, the iOS simulator, MacOS, and Mac Catalyst under `/target`. Some targets are not part of the standard library and they will be built using the nightly toolchain.
1. compile `matrix-sdk-ffi` libraries for iOS, the iOS simulator and macOS under `/target`. Some targets are not part of the standard library and they will be built using the nightly toolchain.
2.`lipo` together the libraries for the same platform under `/generated`
3. run `uniffi` and generate the C header, module map and swift files
4.`xcodebuild` an `xcframework` from the fat static libs and the original iOS one, and add the header and module map to it under `generated/MatrixSDKFFI.xcframework`
5. cleanup and delete the generated files except the .xcframework and the swift sources (that aren't part of the framework)
## Building only the Crypto SDK
For development purposes, it will additionally generate a `Package.swift` file in the root of the repo that can be used to add the framework to your project and enable debugging through the use of [rust-xcode-plugin](https://github.com/BrainiumLLC/rust-xcode-plugin) (make sure to run the task with the argument `--profile=reldbg`).
When building the SDK for release you should pass the `--release` argument to the task, which will strip away any symbols and optimise the created binary.
### Building only the Crypto SDK
```
sh build_crypto_xcframework.sh
build_crypto_xcframework.sh
```
The `build_crypto_xcframework.sh` script will go through all the steps required to generate a fully usable `.xcframework`:
@@ -38,16 +43,31 @@ The `build_crypto_xcframework.sh` script will go through all the steps required
4.`xcodebuild` an `xcframework` from the fat static libs and the original iOS one, and add the header and module map to it under `generated/MatrixSDKCryptoFFI.xcframework`
5. cleanup and delete the generated files except the .xcframework and the swift sources (that aren't part of the framework)
## Running the Xcode project
### Building & testing the Swift package
The Xcode project is meant to provide a simple example on how to integrate everything together but also a place to run unit and integration tests from.
The `Package.swift` file in this directory provides a simple example on how to integrate everything together but also a place to run unit and integration tests from.
It's pre-configured to link to the generated .xcframework and .swift files so successfully running the script first is necessary for it to compile.
It makes the compiled code available to swift by importing the C header through its bridging header.
Once all the generated components are available running it should be as easy as choosing a platform and clicking run.
It's pre-configured to link to the generated static lib and .swift files so successfully running `cargo xtask swift build-library` first is necessary for it to compile. Afterwards you can execute the tests with `swift test`. Note that for the moment this only works on macOS but we're planning to add Linux support in the future.
## Distribution
The generated framework and Swift code can be distributed and integrated directly but in order to make things simpler we bundle them together as a Swift package available [TBD](here) in the case of SDK, and as CocoaPods podspec in the case of Crypto SDK.
The generated framework and Swift code can be distributed and integrated directly but in order to make things simpler we bundle them together as a [Swift package](https://github.com/matrix-org/matrix-rust-components-swift/) in the case of SDK, and as a CocoaPods podspec in the case of Crypto SDK.
### Publishing MatrixSDKCrypto
1. Navigate into `bindings/apple` and run [`build_crypto_xcframework.sh`](#building-only-the-crypto-sdk) script which generates a .zip file with the framework in `./generated` folder
Note that whilst you can run this command from any git branch, if you want to make a public release, ensure you are building from latest `main`
2. Tag the commit you just used to build the release and push the tag to GitHub
Use `matrix-sdk-crypto-ffi-<major>.<minor>.<patch>` tag name
3. Create a new [GitHub release](https://github.com/matrix-org/matrix-rust-sdk/releases) using this tag (see [example](https://github.com/matrix-org/matrix-rust-sdk/releases/tag/matrix-sdk-crypto-ffi-0.3.4))
4. Upload the previously generated .zip file to this release
5. Increment the version in [`MatrixSDKCrypto.podspec`](./MatrixSDKCrypto.podspec)
Note that this is not automated and is a local-only change. To determine the most recent published version, run `pod repo update && pod search MatrixSDKCrypto`
or check git tags via `git tag | grep matrix-sdk-crypto-ffi`
6. Push new Podspec version to Cocoapods via `pod trunk push MatrixSDKCrypto.podspec --allow-warnings`
// base64 decoder, based on the code at https://developer.mozilla.org/en-US/docs/Glossary/Base64#solution_2_%E2%80%93_rewriting_atob_and_btoa_using_typedarrays_and_utf-8
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Florian Duros