v15.1.1

Prepare changelog for v15.1.1
v15.1.1-rc.1
2021-11-22 13:28:39 +00:00 · 2021-11-22 13:28:39 +00:00 · 2021-11-17 13:52:47 +00:00 · 2021-11-17 13:52:47 +00:00 · 2021-11-15 18:08:20 +00:00 · 2021-11-15 14:47:21 +00:00
23 changed files with 1149 additions and 715 deletions
@@ -1,3 +1,79 @@
+Changes in [15.1.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v15.1.1) (2021-11-22)
+==================================================================================================
+
+## 🐛 Bug Fixes
+ * Fix edit history being broken after editing an unencrypted event with an encrypted event ([\#2013](https://github.com/matrix-org/matrix-js-sdk/pull/2013)). Fixes vector-im/element-web#19651 and vector-im/element-web#19651. Contributed by @aaronraimist.
+ * Make events pagination responses parse threads ([\#2011](https://github.com/matrix-org/matrix-js-sdk/pull/2011)). Fixes vector-im/element-web#19587 and vector-im/element-web#19587.
+
+Changes in [15.1.1-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v15.1.1-rc.1) (2021-11-17)
+============================================================================================================
+
+## 🐛 Bug Fixes
+ * Fix edit history being broken after editing an unencrypted event with an encrypted event ([\#2013](https://github.com/matrix-org/matrix-js-sdk/pull/2013)). Fixes vector-im/element-web#19651 and vector-im/element-web#19651. Contributed by @aaronraimist.
+ * Make events pagination responses parse threads ([\#2011](https://github.com/matrix-org/matrix-js-sdk/pull/2011)). Fixes vector-im/element-web#19587 and vector-im/element-web#19587.
+
+Changes in [15.1.0](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v15.1.0) (2021-11-08)
+==================================================================================================
+
+## 🦖 Deprecations
+ * Mark old verification methods as deprecated ([\#1994](https://github.com/matrix-org/matrix-js-sdk/pull/1994)).
+
+## ✨ Features
+ * Try to set a sender on search result events if possible ([\#2004](https://github.com/matrix-org/matrix-js-sdk/pull/2004)).
+ * Port some changes from group calls branch to develop ([\#2001](https://github.com/matrix-org/matrix-js-sdk/pull/2001)). Contributed by @SimonBrandner.
+ * Fetch room membership from server rather than relying on stored data ([\#1998](https://github.com/matrix-org/matrix-js-sdk/pull/1998)).
+ * Add method to fetch the MSC3266 Room Summary of a Room ([\#1988](https://github.com/matrix-org/matrix-js-sdk/pull/1988)).
+
+## 🐛 Bug Fixes
+ * Don't show `Unable to access microphone` when cancelling screensharing dialog ([\#2005](https://github.com/matrix-org/matrix-js-sdk/pull/2005)). Fixes vector-im/element-web#19533 and vector-im/element-web#19533. Contributed by @SimonBrandner.
+ * Strip direction override characters from display names ([\#1992](https://github.com/matrix-org/matrix-js-sdk/pull/1992)). Fixes vector-im/element-web#1712 and vector-im/element-web#1712.
+
+Changes in [15.1.0-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v15.1.0-rc.1) (2021-11-02)
+============================================================================================================
+
+## 🦖 Deprecations
+ * Mark old verification methods as deprecated ([\#1994](https://github.com/matrix-org/matrix-js-sdk/pull/1994)).
+
+## ✨ Features
+ * Try to set a sender on search result events if possible ([\#2004](https://github.com/matrix-org/matrix-js-sdk/pull/2004)).
+ * Port some changes from group calls branch to develop ([\#2001](https://github.com/matrix-org/matrix-js-sdk/pull/2001)). Contributed by @SimonBrandner.
+ * Fetch room membership from server rather than relying on stored data ([\#1998](https://github.com/matrix-org/matrix-js-sdk/pull/1998)).
+ * Add method to fetch the MSC3266 Room Summary of a Room ([\#1988](https://github.com/matrix-org/matrix-js-sdk/pull/1988)).
+
+## 🐛 Bug Fixes
+ * Don't show `Unable to access microphone` when cancelling screensharing dialog ([\#2005](https://github.com/matrix-org/matrix-js-sdk/pull/2005)). Fixes vector-im/element-web#19533 and vector-im/element-web#19533. Contributed by @SimonBrandner.
+ * Strip direction override characters from display names ([\#1992](https://github.com/matrix-org/matrix-js-sdk/pull/1992)). Fixes vector-im/element-web#1712 and vector-im/element-web#1712.
+
+Changes in [15.0.0](https://github.com/vector-im/element-desktop/releases/tag/v15.0.0) (2021-10-25)
+===================================================================================================
+
+## 🚨 BREAKING CHANGES
+ * Use `ICallFeedOpts` in the `CallFeed` constructor. To construct a new `CallFeed` object you have to pass `ICallFeedOpts` e.g. `const callFeed = new CallFeed({client ([\#1964](https://github.com/matrix-org/matrix-js-sdk/pull/1964)). Contributed by [SimonBrandner](https://github.com/SimonBrandner).
+
+## ✨ Features
+ * Make threads use 'm.thread' relation ([\#1980](https://github.com/matrix-org/matrix-js-sdk/pull/1980)).
+ * Try to answer a call without video if we can't access the camera  ([\#1972](https://github.com/matrix-org/matrix-js-sdk/pull/1972)). Fixes vector-im/element-web#17975 and vector-im/element-web#17975. Contributed by [SimonBrandner](https://github.com/SimonBrandner).
+ * Make `opts` in `importRoomKeys()` optional ([\#1974](https://github.com/matrix-org/matrix-js-sdk/pull/1974)). Contributed by [SimonBrandner](https://github.com/SimonBrandner).
+ * Enable TypeScript declaration maps ([\#1966](https://github.com/matrix-org/matrix-js-sdk/pull/1966)). Contributed by [Alexendoo](https://github.com/Alexendoo).
+
+## 🐛 Bug Fixes
+ * Fix `requestVerificationDM` with chronological `pendingEventOrdering` ([\#1943](https://github.com/matrix-org/matrix-js-sdk/pull/1943)). Contributed by [freaktechnik](https://github.com/freaktechnik).
+
+Changes in [15.0.0-rc.1](https://github.com/vector-im/element-desktop/releases/tag/v15.0.0-rc.1) (2021-10-19)
+=============================================================================================================
+
+## 🚨 BREAKING CHANGES
+ * Use `ICallFeedOpts` in the `CallFeed` constructor. To construct a new `CallFeed` object you have to pass `ICallFeedOpts` e.g. `const callFeed = new CallFeed({client ([\#1964](https://github.com/matrix-org/matrix-js-sdk/pull/1964)). Contributed by [SimonBrandner](https://github.com/SimonBrandner).
+
+## ✨ Features
+ * Make threads use 'm.thread' relation ([\#1980](https://github.com/matrix-org/matrix-js-sdk/pull/1980)).
+ * Try to answer a call without video if we can't access the camera  ([\#1972](https://github.com/matrix-org/matrix-js-sdk/pull/1972)). Fixes vector-im/element-web#17975 and vector-im/element-web#17975. Contributed by [SimonBrandner](https://github.com/SimonBrandner).
+ * Make `opts` in `importRoomKeys()` optional ([\#1974](https://github.com/matrix-org/matrix-js-sdk/pull/1974)). Contributed by [SimonBrandner](https://github.com/SimonBrandner).
+ * Enable TypeScript declaration maps ([\#1966](https://github.com/matrix-org/matrix-js-sdk/pull/1966)). Contributed by [Alexendoo](https://github.com/Alexendoo).
+
+## 🐛 Bug Fixes
+ * Fix `requestVerificationDM` with chronological `pendingEventOrdering` ([\#1943](https://github.com/matrix-org/matrix-js-sdk/pull/1943)). Contributed by [freaktechnik](https://github.com/freaktechnik).
+
 Changes in [14.0.1](https://github.com/vector-im/element-desktop/releases/tag/v14.0.1) (2021-10-12)
 ===================================================================================================

@@ -16,8 +92,6 @@ Changes in [14.0.1](https://github.com/vector-im/element-desktop/releases/tag/v1
 * Fix event partitioning from non threading ready clients ([\#1948](https://github.com/matrix-org/matrix-js-sdk/pull/1948)).
 * Ensure unencrypted fields get exposed by getEffectiveEvent() ([\#1938](https://github.com/matrix-org/matrix-js-sdk/pull/1938)). Fixes vector-im/element-web#19062 and vector-im/element-web#19062.

-Changes in [14.0.0-rc.2](https://github.com/vector-im/element-desktop/releases/tag/v14.0.0-rc.2) (2021-10-07)
-=============================================================================================================

 Changes in [14.0.0-rc.1](https://github.com/vector-im/element-desktop/releases/tag/v14.0.0-rc.1) (2021-10-04)
 =============================================================================================================
@@ -1,6 +1,6 @@
 {
  "name": "matrix-js-sdk",
-  "version": "14.0.1",
+  "version": "15.1.1",
  "description": "Matrix Client-Server SDK for Javascript",
  "scripts": {
    "prepublishOnly": "yarn build",
@@ -81,7 +81,7 @@
    "@types/request": "^2.48.5",
    "@typescript-eslint/eslint-plugin": "^4.17.0",
    "@typescript-eslint/parser": "^4.17.0",
-    "allchange": "^1.0.2",
+    "allchange": "^1.0.5",
    "babel-jest": "^26.6.3",
    "babelify": "^10.0.0",
    "better-docs": "^2.4.0-beta.9",
@@ -94,6 +94,17 @@ class MockMediaStream {
    addEventListener() {}
 }

+class MockMediaDeviceInfo {
+    constructor(
+        public kind: "audio" | "video",
+    ) {}
+}
+
+class MockMediaHandler {
+    getUserMediaStream() { return new MockMediaStream("mock_stream_from_media_handler"); }
+    stopUserMediaStream() {}
+}
+
 describe('Call', function() {
    let client;
    let call;
@@ -110,6 +121,8 @@ describe('Call', function() {
            mediaDevices: {
                // @ts-ignore Mock
                getUserMedia: () => new MockMediaStream("local_stream"),
+                // @ts-ignore Mock
+                enumerateDevices: async () => [new MockMediaDeviceInfo("audio"), new MockMediaDeviceInfo("video")],
            },
        };

@@ -129,6 +142,8 @@ describe('Call', function() {
        // We just stub out sendEvent: we're not interested in testing the client's
        // event sending code here
        client.client.sendEvent = () => {};
+        client.client.mediaHandler = new MockMediaHandler;
+        client.client.getMediaHandler = () => client.client.mediaHandler;
        client.httpBackend.when("GET", "/voip/turnServer").respond(200, {});
        call = new MatrixCall({
            client: client.client,
@@ -368,4 +383,16 @@ describe('Call', function() {
        call.setScreensharingEnabled(true);
        expect(call.setScreensharingEnabledWithoutMetadataSupport).toHaveBeenCalled();
    });
+
+    it("should fallback to answering with no video", async () => {
+        await client.httpBackend.flush();
+
+        call.shouldAnswerWithMediaType = (wantedValue: boolean) => wantedValue;
+        client.client.mediaHandler.getUserMediaStream = jest.fn().mockRejectedValue("reject");
+
+        await call.answer(true, true);
+
+        expect(client.client.mediaHandler.getUserMediaStream).toHaveBeenNthCalledWith(1, true, true);
+        expect(client.client.mediaHandler.getUserMediaStream).toHaveBeenNthCalledWith(2, true, false);
+    });
 });
@@ -92,6 +92,14 @@ export enum EventType {
 export enum RelationType {
    Annotation = "m.annotation",
    Replace = "m.replace",
+    /**
+     * Note, "io.element.thread" is hardcoded
+     * Should be replaced with "m.thread" once MSC3440 lands
+     * Can not use `UnstableValue` as TypeScript does not
+     * allow computed values in enums
+     * https://github.com/microsoft/TypeScript/issues/27976
+     */
+    Thread = "io.element.thread",
 }

 export enum MsgType {
@@ -168,11 +176,6 @@ export const UNSTABLE_ELEMENT_FUNCTIONAL_USERS = new UnstableValue(
    "io.element.functional_members",
    "io.element.functional_members");

-export const UNSTABLE_ELEMENT_REPLY_IN_THREAD = new UnstableValue(
-    "m.in_thread",
-    "io.element.in_thread",
-);
-
 export interface IEncryptedFile {
    url: string;
    mimetype?: string;
@@ -671,6 +671,12 @@ interface IThirdPartyUser {
    protocol: string;
    fields: object;
 }
+
+interface IRoomSummary extends Omit<IPublicRoomsChunkRoom, "canonical_alias" | "aliases"> {
+    room_type?: RoomType;
+    membership?: string;
+    is_encrypted: boolean;
+}
 /* eslint-enable camelcase */

 /**
@@ -1749,6 +1755,7 @@ export class MatrixClient extends EventEmitter {
     * @param {string} deviceId the device to verify
     *
     * @returns {Verification} a verification object
+     * @deprecated Use `requestVerification` instead.
     */
    public beginKeyVerification(method: string, userId: string, deviceId: string): Verification {
        if (!this.crypto) {
@@ -1911,6 +1918,7 @@ export class MatrixClient extends EventEmitter {
        return this.crypto.checkCrossSigningPrivateKey(privateKey, expectedPublicKey);
    }

+    // deprecated: use requestVerification instead
    public legacyDeviceVerification(
        userId: string,
        deviceId: string,
@@ -2374,7 +2382,7 @@ export class MatrixClient extends EventEmitter {
     * @return {Promise} a promise which resolves when the keys
     *    have been imported
     */
-    public importRoomKeys(keys: IMegolmSessionData[], opts: IImportRoomKeysOpts): Promise<void> {
+    public importRoomKeys(keys: IMegolmSessionData[], opts?: IImportRoomKeysOpts): Promise<void> {
        if (!this.crypto) {
            throw new Error("End-to-end encryption disabled");
        }
@@ -3397,7 +3405,7 @@ export class MatrixClient extends EventEmitter {
    /**
     * @param {string} roomId
     * @param {object} eventObject An object with the partial structure of an event, to which event_id, user_id, room_id and origin_server_ts will be added.
-     * @param {string} txnId the txnId.
+     * @param {string} txnId Optional.
     * @param {module:client.callback} callback Optional.
     * @return {Promise} Resolves: to an empty object {}
     * @return {module:http-api.MatrixError} Rejects: with an error response.
@@ -3405,7 +3413,7 @@ export class MatrixClient extends EventEmitter {
    private sendCompleteEvent(
        roomId: string,
        eventObject: any,
-        txnId: string,
+        txnId?: string,
        callback?: Callback,
    ): Promise<ISendEventResponse> {
        if (utils.isFunction(txnId)) {
@@ -4043,10 +4051,8 @@ export class MatrixClient extends EventEmitter {
        // Work backwards first, looking at create events.
        let createEvent = currentRoom.currentState.getStateEvents(EventType.RoomCreate, "");
        while (createEvent) {
-            logger.log(`Looking at ${createEvent.getId()}`);
            const predecessor = createEvent.getContent()['predecessor'];
            if (predecessor && predecessor['room_id']) {
-                logger.log(`Looking at predecessor ${predecessor['room_id']}`);
                const refRoom = this.getRoom(predecessor['room_id']);
                if (!refRoom) break; // end of the chain

@@ -4572,7 +4578,12 @@ export class MatrixClient extends EventEmitter {
                    const stateEvents = res.state.map(this.getEventMapper());
                    room.currentState.setUnknownStateEvents(stateEvents);
                }
-                room.addEventsToTimeline(matrixEvents, true, room.getLiveTimeline());
+
+                const [timelineEvents, threadedEvents] = this.partitionThreadedEvents(matrixEvents);
+
+                room.addEventsToTimeline(timelineEvents, true, room.getLiveTimeline());
+                this.processThreadEvents(room, threadedEvents);
+
                room.oldState.paginationToken = res.end;
                if (res.chunk.length === 0) {
                    room.oldState.paginationToken = null;
@@ -4678,7 +4689,11 @@ export class MatrixClient extends EventEmitter {
                const stateEvents = res.state.map(this.getEventMapper());
                timeline.getState(EventTimeline.BACKWARDS).setUnknownStateEvents(stateEvents);
            }
-            timelineSet.addEventsToTimeline(matrixEvents, true, timeline, res.start);
+
+            const [timelineEvents, threadedEvents] = this.partitionThreadedEvents(matrixEvents);
+
+            timelineSet.addEventsToTimeline(timelineEvents, true, timeline, res.start);
+            this.processThreadEvents(timelineSet.room, threadedEvents);

            // there is no guarantee that the event ended up in "timeline" (we
            // might have switched to a neighbouring timeline) - so check the
@@ -4811,8 +4826,11 @@ export class MatrixClient extends EventEmitter {
                    matrixEvents[i] = event;
                }

-                eventTimeline.getTimelineSet()
-                    .addEventsToTimeline(matrixEvents, backwards, eventTimeline, token);
+                const [timelineEvents, threadedEvents] = this.partitionThreadedEvents(matrixEvents);
+
+                const timelineSet = eventTimeline.getTimelineSet();
+                timelineSet.addEventsToTimeline(timelineEvents, backwards, eventTimeline, token);
+                this.processThreadEvents(timelineSet.room, threadedEvents);

                // if we've hit the end of the timeline, we need to stop trying to
                // paginate. We need to keep the 'forwards' token though, to make sure
@@ -4845,8 +4863,12 @@ export class MatrixClient extends EventEmitter {
                }
                const token = res.end;
                const matrixEvents = res.chunk.map(this.getEventMapper());
+
+                const [timelineEvents, threadedEvents] = this.partitionThreadedEvents(matrixEvents);
+
                eventTimeline.getTimelineSet()
-                    .addEventsToTimeline(matrixEvents, backwards, eventTimeline, token);
+                    .addEventsToTimeline(timelineEvents, backwards, eventTimeline, token);
+                this.processThreadEvents(room, threadedEvents);

                // if we've hit the end of the timeline, we need to stop trying to
                // paginate. We need to keep the 'forwards' token though, to make sure
@@ -5416,6 +5438,14 @@ export class MatrixClient extends EventEmitter {
        const resultsLength = roomEvents.results ? roomEvents.results.length : 0;
        for (let i = 0; i < resultsLength; i++) {
            const sr = SearchResult.fromJson(roomEvents.results[i], this.getEventMapper());
+            const room = this.getRoom(sr.context.getEvent().getRoomId());
+            if (room) {
+                // Copy over a known event sender if we can
+                for (const ev of sr.context.getTimeline()) {
+                    const sender = room.getMember(ev.getSender());
+                    if (!ev.sender && sender) ev.sender = sender;
+                }
+            }
            searchResults.results.push(sr);
        }
        return searchResults;
@@ -5985,7 +6015,9 @@ export class MatrixClient extends EventEmitter {
        if (fetchedEventType === EventType.RoomMessageEncrypted) {
            const allEvents = originalEvent ? events.concat(originalEvent) : events;
            await Promise.all(allEvents.map(e => {
-                return new Promise(resolve => e.once("Event.decrypted", resolve));
+                if (e.isEncrypted()) {
+                    return new Promise(resolve => e.once("Event.decrypted", resolve));
+                }
            }));
            events = events.filter(e => e.getType() === eventType);
        }
@@ -8523,6 +8555,74 @@ export class MatrixClient extends EventEmitter {
    public supportsExperimentalThreads(): boolean {
        return this.clientOpts?.experimentalThreadSupport || false;
    }
+
+    /**
+     * Fetches the summary of a room as defined by an initial version of MSC3266 and implemented in Synapse
+     * Proposed at https://github.com/matrix-org/matrix-doc/pull/3266
+     * @param {string} roomIdOrAlias The ID or alias of the room to get the summary of.
+     * @param {string[]?} via The list of servers which know about the room if only an ID was provided.
+     */
+    public async getRoomSummary(roomIdOrAlias: string, via?: string[]): Promise<IRoomSummary> {
+        const path = utils.encodeUri("/rooms/$roomid/summary", { $roomid: roomIdOrAlias });
+        return this.http.authedRequest(undefined, "GET", path, { via }, null, {
+            qsStringifyOptions: { arrayFormat: 'repeat' },
+            prefix: "/_matrix/client/unstable/im.nheko.summary",
+        });
+    }
+
+    public partitionThreadedEvents(events: MatrixEvent[]): [MatrixEvent[], MatrixEvent[]] {
+        // Indices to the events array, for readibility
+        const ROOM = 0;
+        const THREAD = 1;
+        const threadRoots = new Set<string>();
+        if (this.supportsExperimentalThreads()) {
+            return events.reduce((memo, event: MatrixEvent) => {
+                const room = this.getRoom(event.getRoomId());
+                // An event should live in the thread timeline if
+                // - It's a reply in thread event
+                // - It's related to a reply in thread event
+                let shouldLiveInThreadTimeline = event.isThreadRelation;
+                if (shouldLiveInThreadTimeline) {
+                    threadRoots.add(event.relationEventId);
+                } else {
+                    const parentEventId = event.parentEventId;
+                    const parentEvent = room?.findEventById(parentEventId) || events.find((mxEv: MatrixEvent) => {
+                        return mxEv.getId() === parentEventId;
+                    });
+                    shouldLiveInThreadTimeline = parentEvent?.isThreadRelation;
+
+                    // Copy all the reactions and annotations to the root event
+                    // to the thread timeline. They will end up living in both
+                    // timelines at the same time
+                    const targetingThreadRoot = parentEvent?.isThreadRoot || threadRoots.has(event.relationEventId);
+                    if (targetingThreadRoot && !event.isThreadRelation && event.relationEventId) {
+                        memo[THREAD].push(event);
+                    }
+                }
+                const targetTimeline = shouldLiveInThreadTimeline ? THREAD : ROOM;
+                memo[targetTimeline].push(event);
+                return memo;
+            }, [[], []]);
+        } else {
+            // When `experimentalThreadSupport` is disabled
+            // treat all events as timelineEvents
+            return [
+                events,
+                [],
+            ];
+        }
+    }
+
+    /**
+     * @experimental
+     */
+    public processThreadEvents(room: Room, threadedEvents: MatrixEvent[]): void {
+        threadedEvents
+            .sort((a, b) => a.getTs() - b.getTs())
+            .forEach(event => {
+                room.addThreadedEvent(event);
+            });
+    }
 }

 /**
@@ -341,12 +341,15 @@ class MegolmEncryption extends EncryptionAlgorithm {
            await Promise.all([
                (async () => {
                    // share keys with devices that we already have a session for
-                    logger.debug(`Sharing keys with existing Olm sessions in ${this.roomId}`);
+                    logger.debug(`Sharing keys with existing Olm sessions in ${this.roomId}`, olmSessions);
                    await this.shareKeyWithOlmSessions(session, key, payload, olmSessions);
                    logger.debug(`Shared keys with existing Olm sessions in ${this.roomId}`);
                })(),
                (async () => {
-                    logger.debug(`Sharing keys (start phase 1) with new Olm sessions in ${this.roomId}`);
+                    logger.debug(
+                        `Sharing keys (start phase 1) with new Olm sessions in ${this.roomId}`,
+                        devicesWithoutSession,
+                    );
                    const errorDevices = [];

                    // meanwhile, establish olm sessions for devices that we don't
@@ -403,8 +406,11 @@ class MegolmEncryption extends EncryptionAlgorithm {
                    logger.debug(`Shared keys (all phases done) with new Olm sessions in ${this.roomId}`);
                })(),
                (async () => {
-                    logger.debug(`Notifying blocked devices in ${this.roomId}`);
-                    // also, notify blocked devices that they're blocked
+                    logger.debug(`There are ${Object.entries(blocked).length} blocked devices in ${this.roomId}`,
+                        Object.entries(blocked));
+
+                    // also, notify newly blocked devices that they're blocked
+                    logger.debug(`Notifying newly blocked devices in ${this.roomId}`);
                    const blockedMap: Record<string, Record<string, { device: IBlockedDevice }>> = {};
                    let blockedCount = 0;
                    for (const [userId, userBlockedDevices] of Object.entries(blocked)) {
@@ -421,7 +427,7 @@ class MegolmEncryption extends EncryptionAlgorithm {
                    }

                    await this.notifyBlockedDevices(session, blockedMap);
-                    logger.debug(`Notified ${blockedCount} blocked devices in ${this.roomId}`);
+                    logger.debug(`Notified ${blockedCount} newly blocked devices in ${this.roomId}`, blockedMap);
                })(),
            ]);
        };
@@ -697,7 +703,7 @@ class MegolmEncryption extends EncryptionAlgorithm {

        await this.baseApis.sendToDevice("org.matrix.room_key.withheld", contentMap);

-        // store that we successfully uploaded the keys of the current slice
+        // record the fact that we notified these blocked devices
        for (const userId of Object.keys(contentMap)) {
            for (const deviceId of Object.keys(contentMap[userId])) {
                session.markNotifiedBlockedDevice(userId, deviceId);
@@ -846,9 +852,9 @@ class MegolmEncryption extends EncryptionAlgorithm {

        this.getDevicesWithoutSessions(devicemap, devicesByUser, errorDevices);

-        logger.debug(`Sharing keys with Olm sessions in ${this.roomId}`);
+        logger.debug(`Sharing keys with newly created Olm sessions in ${this.roomId}`);
        await this.shareKeyWithOlmSessions(session, key, payload, devicemap);
-        logger.debug(`Shared keys with Olm sessions in ${this.roomId}`);
+        logger.debug(`Shared keys with newly created Olm sessions in ${this.roomId}`);
    }

    private async shareKeyWithOlmSessions(
@@ -864,7 +870,8 @@ class MegolmEncryption extends EncryptionAlgorithm {
                `megolm keys for ${session.sessionId} ` +
                `in ${this.roomId} (slice ${i + 1}/${userDeviceMaps.length})`;
            try {
-                logger.debug(`Sharing ${taskDetail}`);
+                logger.debug(`Sharing ${taskDetail}`,
+                    userDeviceMaps[i].map((d) => `${d.userId}/${d.deviceInfo.deviceId}`));
                await this.encryptAndSendKeysToDevices(
                    session, key.chain_index, userDeviceMaps[i], payload,
                );
@@ -906,14 +913,16 @@ class MegolmEncryption extends EncryptionAlgorithm {
            );
        }

-        const filteredFailedDevices =
-            await this.olmDevice.filterOutNotifiedErrorDevices(failedDevices);
+        const unnotifiedFailedDevices =
+            await this.olmDevice.filterOutNotifiedErrorDevices(
+                failedDevices,
+            );
        logger.debug(
-            `Filtered down to ${filteredFailedDevices.length} error devices ` +
-            `in ${this.roomId}`,
+            `Need to notify ${unnotifiedFailedDevices.length} failed devices ` +
+            `which haven't been notified before in ${this.roomId}`,
        );
        const blockedMap: Record<string, Record<string, { device: IBlockedDevice }>> = {};
-        for (const { userId, deviceInfo } of filteredFailedDevices) {
+        for (const { userId, deviceInfo } of unnotifiedFailedDevices) {
            blockedMap[userId] = blockedMap[userId] || {};
            // we use a similar format to what
            // olmlib.ensureOlmSessionsForDevices returns, so that
@@ -930,7 +939,7 @@ class MegolmEncryption extends EncryptionAlgorithm {
        // send the notifications
        await this.notifyBlockedDevices(session, blockedMap);
        logger.debug(
-            `Notified ${filteredFailedDevices.length} devices we failed to ` +
+            `Notified ${unnotifiedFailedDevices.length} devices we failed to ` +
            `create Olm sessions in ${this.roomId}`,
        );
    }
@@ -142,7 +142,7 @@ export interface IImportOpts {
 }

 export interface IImportRoomKeysOpts {
-    progressCallback: (stage: IImportOpts) => void;
+    progressCallback?: (stage: IImportOpts) => void;
    untrusted?: boolean;
    source?: string; // TODO: Enum
 }
@@ -40,7 +40,7 @@ import {
    ISecretRequest,
    SecretStorageKeyObject,
 } from './SecretStorage';
-import { IAddSecretStorageKeyOpts, ISecretStorageKeyInfo } from "./api";
+import { IAddSecretStorageKeyOpts, IImportRoomKeysOpts, ISecretStorageKeyInfo } from "./api";
 import { OutgoingRoomKeyRequestManager } from './OutgoingRoomKeyRequestManager';
 import { IndexedDBCryptoStore } from './store/indexeddb-crypto-store';
 import { ReciprocateQRCode, SCAN_QR_CODE_METHOD, SHOW_QR_CODE_METHOD } from './verification/QRCode';
@@ -58,7 +58,7 @@ import { BackupManager } from "./backup";
 import { IStore } from "../store";
 import { Room } from "../models/room";
 import { RoomMember } from "../models/room-member";
-import { MatrixEvent } from "../models/event";
+import { MatrixEvent, EventStatus } from "../models/event";
 import { MatrixClient, IKeysUploadResponse, SessionStore, ISignedKey } from "../client";
 import type { EncryptionAlgorithm, DecryptionAlgorithm } from "./algorithms/base";
 import type { IRoomEncryption, RoomList } from "./RoomList";
@@ -2658,7 +2658,7 @@ export class Crypto extends EventEmitter {
     * @param {Function} opts.progressCallback called with an object which has a stage param
     * @return {Promise} a promise which resolves once the keys have been imported
     */
-    public importRoomKeys(keys: IMegolmSessionData[], opts: any = {}): Promise<any> { // TODO types
+    public importRoomKeys(keys: IMegolmSessionData[], opts: IImportRoomKeysOpts = {}): Promise<any> { // TODO types
        let successes = 0;
        let failures = 0;
        const total = keys.length;
@@ -3201,6 +3201,29 @@ export class Crypto extends EventEmitter {
        createRequest: any, // TODO types
        isLiveEvent = true,
    ): Promise<void> {
+        // Wait for event to get its final ID with pendingEventOrdering: "chronological", since DM channels depend on it.
+        if (event.isSending() && event.status != EventStatus.SENT) {
+            let eventIdListener;
+            let statusListener;
+            try {
+                await new Promise((resolve, reject) => {
+                    eventIdListener = resolve;
+                    statusListener = () => {
+                        if (event.status == EventStatus.CANCELLED) {
+                            reject(new Error("Event status set to CANCELLED."));
+                        }
+                    };
+                    event.once("Event.localEventIdReplaced", eventIdListener);
+                    event.on("Event.status", statusListener);
+                });
+            } catch (err) {
+                logger.error("error while waiting for the verification event to be sent: " + err.message);
+                return;
+            } finally {
+                event.removeListener("Event.localEventIdReplaced", eventIdListener);
+                event.removeListener("Event.status", statusListener);
+            }
+        }
        let request = requestsMap.getRequest(event);
        let isNewRequest = false;
        if (!request) {
@@ -61,6 +61,7 @@ export enum AuthType {
    Sso = "m.login.sso",
    SsoUnstable = "org.matrix.login.sso",
    Dummy = "m.login.dummy",
+    RegistrationToken = "org.matrix.msc3231.login.registration_token",
 }

 export interface IAuthDict {
@@ -449,6 +450,7 @@ export class InteractiveAuth {
            } catch (e) {
                this.attemptAuthDeferred.reject(e);
                this.attemptAuthDeferred = null;
+                return;
            }

            if (
@@ -0,0 +1,51 @@
+/*
+Copyright 2021 The Matrix.org Foundation C.I.C.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+import { EventEmitter } from "events";
+
+/**
+ * Typed Event Emitter class which can act as a Base Model for all our model
+ * and communication events.
+ * This makes it much easier for us to distinguish between events, as we now need
+ * to properly type this, so that our events are not stringly-based and prone
+ * to silly typos.
+ */
+export abstract class BaseModel<Events extends string> extends EventEmitter {
+    public on(event: Events, listener: (...args: any[]) => void): this {
+        super.on(event, listener);
+        return this;
+    }
+
+    public once(event: Events, listener: (...args: any[]) => void): this {
+        super.once(event, listener);
+        return this;
+    }
+
+    public off(event: Events, listener: (...args: any[]) => void): this {
+        super.off(event, listener);
+        return this;
+    }
+
+    public addListener(event: Events, listener: (...args: any[]) => void): this {
+        super.addListener(event, listener);
+        return this;
+    }
+
+    public removeListener(event: Events, listener: (...args: any[]) => void): this {
+        super.removeListener(event, listener);
+        return this;
+    }
+}
@@ -44,6 +44,7 @@ interface IOpts {
    timelineSupport?: boolean;
    filter?: Filter;
    unstableClientRelationAggregation?: boolean;
+    pendingEvents?: boolean;
 }

 export enum DuplicateStrategy {
@@ -54,6 +55,7 @@ export enum DuplicateStrategy {
 export class EventTimelineSet extends EventEmitter {
    private readonly timelineSupport: boolean;
    private unstableClientRelationAggregation: boolean;
+    private displayPendingEvents: boolean;
    private liveTimeline: EventTimeline;
    private timelines: EventTimeline[];
    private _eventIdToTimeline: Record<string, EventTimeline>;
@@ -102,6 +104,7 @@ export class EventTimelineSet extends EventEmitter {
        this.timelineSupport = Boolean(opts.timelineSupport);
        this.liveTimeline = new EventTimeline(this);
        this.unstableClientRelationAggregation = !!opts.unstableClientRelationAggregation;
+        this.displayPendingEvents = opts.pendingEvents !== false;

        // just a list - *not* ordered.
        this.timelines = [this.liveTimeline];
@@ -151,7 +154,7 @@ export class EventTimelineSet extends EventEmitter {
     * @throws If <code>opts.pendingEventOrdering</code> was not 'detached'
     */
    public getPendingEvents(): MatrixEvent[] {
-        if (!this.room) {
+        if (!this.room || !this.displayPendingEvents) {
            return [];
        }

@@ -28,7 +28,6 @@ import {
    EventType,
    MsgType,
    RelationType,
-    UNSTABLE_ELEMENT_REPLY_IN_THREAD,
 } from "../@types/event";
 import { Crypto } from "../crypto";
 import { deepSortedObjectEntries } from "../utils";
@@ -119,7 +118,7 @@ interface IAggregatedRelation {
    key?: string;
 }

-interface IEventRelation {
+export interface IEventRelation {
    rel_type: RelationType | string;
    event_id: string;
    key?: string;
@@ -419,38 +418,45 @@ export class MatrixEvent extends EventEmitter {

    /**
     * @experimental
-     * Get the event ID of the replied event
+     * Get the event ID of the thread head
     */
+    public get threadRootId(): string {
+        const relatesTo = this.getWireContent()?.["m.relates_to"];
+        if (relatesTo?.rel_type === RelationType.Thread) {
+            return relatesTo.event_id;
+        }
+    }
+
+    /**
+     * @experimental
+     */
+    public get isThreadRelation(): boolean {
+        return !!this.threadRootId;
+    }
+
+    /**
+     * @experimental
+     */
+    public get isThreadRoot(): boolean {
+        // TODO, change the inner working of this getter for it to use the
+        // bundled relationship return on the event, view MSC3440
+        const thread = this.getThread();
+        return thread?.id === this.getId();
+    }
+
+    public get parentEventId(): string {
+        return this.replyEventId || this.relationEventId;
+    }
+
    public get replyEventId(): string {
        const relations = this.getWireContent()["m.relates_to"];
        return relations?.["m.in_reply_to"]?.["event_id"];
    }

-    /**
-     * @experimental
-     * Determines whether a reply should be rendered in a thread
-     * or in the main room timeline
-     */
-    public get replyInThread(): boolean {
-        /**
-         * UNSTABLE_ELEMENT_REPLY_IN_THREAD can live either
-         * at the m.relates_to and m.in_reply_to level
-         * This will likely change once we settle on a
-         * way to achieve threads
-         * TODO: Clean this up once we have a clear way forward
-         */
-
-        const relatesTo = this.getWireContent()?.["m.relates_to"];
-        const replyTo = relatesTo?.["m.in_reply_to"];
-
-        return relatesTo?.[UNSTABLE_ELEMENT_REPLY_IN_THREAD.name]
-            || (this.replyEventId && replyTo[UNSTABLE_ELEMENT_REPLY_IN_THREAD.name])
-            || this.thread instanceof Thread;
-    }
-
-    public get parentEventId(): string {
-        return this.replyEventId
-            || this.getWireContent()["m.relates_to"]?.event_id;
+    public get relationEventId(): string {
+        return this.getWireContent()
+            ?.["m.relates_to"]
+            ?.event_id;
    }

    /**
@@ -131,7 +131,9 @@ export class RoomMember extends EventEmitter {
            this.disambiguate,
        );

-        this.rawDisplayName = event.getDirectionalContent().displayname;
+        // not quite raw: we strip direction override chars so it can safely be inserted into
+        // blocks of text without breaking the text direction
+        this.rawDisplayName = utils.removeDirectionOverrideChars(event.getDirectionalContent().displayname);
        if (!this.rawDisplayName || !utils.removeHiddenChars(this.rawDisplayName)) {
            this.rawDisplayName = this.userId;
        }
@@ -355,7 +357,7 @@ function calculateDisplayName(
    roomState: RoomState,
    disambiguate: boolean,
 ): string {
-    if (disambiguate) return displayName + " (" + selfUserId + ")";
+    if (disambiguate) return utils.removeDirectionOverrideChars(displayName) + " (" + selfUserId + ")";

    if (!displayName || displayName === selfUserId) return selfUserId;

@@ -363,7 +365,18 @@ function calculateDisplayName(
    // after stripping it of zero width characters and padding spaces
    if (!utils.removeHiddenChars(displayName)) return selfUserId;

-    return displayName;
+    // We always strip the direction override characters (LRO and RLO).
+    // These override the text direction for all subsequent characters
+    // in the paragraph so if display names contained these, they'd
+    // need to be wrapped in something to prevent this from leaking out
+    // (which we can do in HTML but not text) or we'd need to add
+    // control characters to the string to reset any overrides (eg.
+    // adding PDF characters at the end). As far as we can see,
+    // there should be no reason these would be necessary - rtl display
+    // names should flip into the correct direction automatically based on
+    // the characters, and you can still embed rtl in ltr or vice versa
+    // with the embed chars or marker chars.
+    return utils.removeDirectionOverrideChars(displayName);
 }

 /**
@@ -149,7 +149,7 @@ export class Room extends EventEmitter {
    /**
     * @experimental
     */
-    public threads = new Set<Thread>();
+    public threads = new Map<string, Thread>();

    /**
     * Construct a new Room.
@@ -672,7 +672,13 @@ export class Room extends EventEmitter {
        // were the members loaded from the server?
        let fromServer = false;
        let rawMembersEvents = await this.client.store.getOutOfBandMembers(this.roomId);
-        if (rawMembersEvents === null) {
+        // If the room is encrypted, we always fetch members from the server at
+        // least once, in case the latest state wasn't persisted properly.  Note
+        // that this function is only called once (unless loading the members
+        // fails), since loadMembersIfNeeded always returns this.membersPromise
+        // if set, which will be the result of the first (successful) call.
+        if (rawMembersEvents === null ||
+            (this.client.isCryptoEnabled() && this.client.isRoomEncrypted(this.roomId))) {
            fromServer = true;
            rawMembersEvents = await this.loadMembersFromServer();
            logger.log(`LL: got ${rawMembersEvents.length} ` +
@@ -1068,19 +1074,6 @@ export class Room extends EventEmitter {
        );
    }

-    /**
-     * @experimental
-     */
-    public addThread(thread: Thread): Set<Thread> {
-        this.threads.add(thread);
-        if (!thread.ready) {
-            thread.once(ThreadEvent.Ready, this.dedupeThreads);
-            this.emit(ThreadEvent.Update, thread);
-            this.reEmitter.reEmit(thread, [ThreadEvent.Update, ThreadEvent.Ready]);
-        }
-        return this.threads;
-    }
-
    /**
     * @experimental
     */
@@ -1097,26 +1090,6 @@ export class Room extends EventEmitter {
        return Array.from(this.threads.values());
    }

-    /**
-     * Two threads starting from a different child event can end up
-     * with the same event root. This method ensures that the duplicates
-     * are removed
-     * @experimental
-     */
-    private dedupeThreads = (readyThread): void => {
-        const deduped = Array.from(this.threads).reduce((dedupedThreads, thread) => {
-            if (dedupedThreads.has(thread.id)) {
-                dedupedThreads.get(thread.id).merge(thread);
-            } else {
-                dedupedThreads.set(thread.id, thread);
-            }
-
-            return dedupedThreads;
-        }, new Map<string, Thread>());
-
-        this.threads = new Set<Thread>(deduped.values());
-    };
-
    /**
     * Get a member from the current room state.
     * @param {string} userId The user ID of the member.
@@ -1293,21 +1266,45 @@ export class Room extends EventEmitter {
        }
    }

+    public findThreadForEvent(event: MatrixEvent): Thread {
+        if (!event) {
+            return null;
+        }
+
+        if (event.isThreadRelation) {
+            return this.threads.get(event.threadRootId);
+        } else if (event.isThreadRoot) {
+            return this.threads.get(event.getId());
+        } else {
+            const parentEvent = this.findEventById(event.parentEventId);
+            return this.findThreadForEvent(parentEvent);
+        }
+    }
+
    /**
     * Add an event to a thread's timeline. Will fire "Thread.update"
     * @experimental
     */
-    public addThreadedEvent(event: MatrixEvent): void {
-        let thread = this.findEventById(event.parentEventId)?.getThread();
+    public async addThreadedEvent(event: MatrixEvent): Promise<void> {
+        let thread = this.findThreadForEvent(event);
        if (thread) {
            thread.addEvent(event);
        } else {
-            thread = new Thread([event], this, this.client);
-        }
-
-        if (!this.threads.has(thread)) {
-            this.addThread(thread);
+            const events = [event];
+            let rootEvent = this.findEventById(event.threadRootId);
+            // If the rootEvent does not exist in the current sync, then look for
+            // it over the network
+            if (!rootEvent) {
+                const eventData = await this.client.fetchRoomEvent(this.roomId, event.threadRootId);
+                rootEvent = new MatrixEvent(eventData);
+            }
+            events.unshift(rootEvent);
+            thread = new Thread(events, this, this.client);
+            this.threads.set(thread.id, thread);
+            this.reEmitter.reEmit(thread, [ThreadEvent.Update, ThreadEvent.Ready]);
+            this.emit(ThreadEvent.New, thread);
        }
+        this.emit(ThreadEvent.Update, thread);
    }

    /**
@@ -1409,7 +1406,7 @@ export class Room extends EventEmitter {
        // TODO: Enable "pending events" for threads
        // There's a fair few things to update to make them work with Threads
        // Will get back to it when the plan is to build a more polished UI ready for production
-        if (this.client?.supportsExperimentalThreads() && event.replyInThread) {
+        if (this.client?.supportsExperimentalThreads() && event.threadRootId) {
            return;
        }

@@ -1585,14 +1582,6 @@ export class Room extends EventEmitter {
            oldEventId, oldStatus);
    }

-    public findThreadByEventId(eventId: string): Thread {
-        for (const thread of this.threads) {
-            if (thread.has(eventId)) {
-                return thread;
-            }
-        }
-    }
-
    /**
     * Update the status / event id on a pending event, to reflect its transmission
     * progress.
@@ -1733,6 +1722,10 @@ export class Room extends EventEmitter {
            // TODO: We should have a filter to say "only add state event
            // types X Y Z to the timeline".
            this.addLiveEvent(events[i], duplicateStrategy, fromCache);
+            const thread = this.threads.get(events[i].getId());
+            if (thread && !thread.ready) {
+                thread.addEvent(events[i], true);
+            }
        }
    }

@@ -14,27 +14,23 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */

-import { EventEmitter } from "events";
 import { MatrixClient } from "../matrix";
 import { MatrixEvent } from "./event";
 import { EventTimeline } from "./event-timeline";
 import { EventTimelineSet } from './event-timeline-set';
 import { Room } from './room';
+import { BaseModel } from "./base-model";

 export enum ThreadEvent {
+    New = "Thread.new",
    Ready = "Thread.ready",
-    Update = "Thread.update"
-}
-
-interface ISerialisedThread {
-    id: string;
-    tails: string[];
+    Update = "Thread.update",
 }

 /**
 * @experimental
 */
-export class Thread extends EventEmitter {
+export class Thread extends BaseModel<ThreadEvent> {
    /**
     * A reference to the event ID at the top of the thread
     */
@@ -42,8 +38,7 @@ export class Thread extends EventEmitter {
    /**
     * A reference to all the events ID at the bottom of the threads
     */
-    public readonly tail = new Set<string>();
-    public readonly timelineSet: EventTimelineSet;
+    public readonly timelineSet;

    constructor(
        events: MatrixEvent[] = [],
@@ -51,9 +46,14 @@ export class Thread extends EventEmitter {
        public readonly client: MatrixClient,
    ) {
        super();
-        this.timelineSet = new EventTimelineSet(room, {
+        if (events.length === 0) {
+            throw new Error("Can't create an empty thread");
+        }
+
+        this.timelineSet = new EventTimelineSet(this.room, {
            unstableClientRelationAggregation: true,
            timelineSupport: true,
+            pendingEvents: false,
        });
        events.forEach(event => this.addEvent(event));
    }
@@ -69,13 +69,12 @@ export class Thread extends EventEmitter {
            return;
        }

-        if (this.tail.has(event.replyEventId)) {
-            this.tail.delete(event.replyEventId);
-        }
-        this.tail.add(event.getId());
-
-        if (!event.replyEventId || !this.timelineSet.findEventById(event.replyEventId)) {
-            this.root = event.getId();
+        if (!this.root) {
+            if (event.isThreadRelation) {
+                this.root = event.threadRootId;
+            } else {
+                this.root = event.getId();
+            }
        }

        // all the relevant membership info to hydrate events with a sender
@@ -99,51 +98,6 @@ export class Thread extends EventEmitter {
        this.emit(ThreadEvent.Update, this);
    }

-    /**
-     * Completes the reply chain with all events
-     * missing from the current sync data
-     * Will fire "Thread.ready"
-     */
-    public async fetchReplyChain(): Promise<void> {
-        if (!this.ready) {
-            let mxEvent = this.room.findEventById(this.rootEvent.replyEventId);
-            if (!mxEvent) {
-                mxEvent = await this.fetchEventById(
-                    this.rootEvent.getRoomId(),
-                    this.rootEvent.replyEventId,
-                );
-            }
-
-            this.addEvent(mxEvent, true);
-            if (mxEvent.replyEventId) {
-                await this.fetchReplyChain();
-            } else {
-                await this.decryptEvents();
-                this.emit(ThreadEvent.Ready, this);
-            }
-        }
-    }
-
-    private async decryptEvents(): Promise<void> {
-        await Promise.allSettled(
-            Array.from(this.timelineSet.getLiveTimeline().getEvents()).map(event => {
-                return this.client.decryptEventIfNeeded(event, {});
-            }),
-        );
-    }
-
-    /**
-     * Fetches an event over the network
-     */
-    private async fetchEventById(roomId: string, eventId: string): Promise<MatrixEvent> {
-        const response = await this.client.http.authedRequest(
-            undefined,
-            "GET",
-            `/rooms/${roomId}/event/${eventId}`,
-        );
-        return new MatrixEvent(response);
-    }
-
    /**
     * Finds an event by ID in the current thread
     */
@@ -151,11 +105,20 @@ export class Thread extends EventEmitter {
        return this.timelineSet.findEventById(eventId);
    }

+    /**
+     * Return last reply to the thread
+     */
+    public get lastReply(): MatrixEvent {
+        const threadReplies = this.events
+            .filter(event => event.isThreadRelation);
+        return threadReplies[threadReplies.length - 1];
+    }
+
    /**
     * Determines thread's ready status
     */
    public get ready(): boolean {
-        return this.rootEvent.replyEventId === undefined;
+        return this.rootEvent !== undefined;
    }

    /**
@@ -178,9 +141,13 @@ export class Thread extends EventEmitter {

    /**
     * The number of messages in the thread
+     * Only count rel_type=m.thread as we want to
+     * exclude annotations from that number
     */
    public get length(): number {
-        return this.events.length;
+        return this.events
+            .filter(event => event.isThreadRelation)
+            .length;
    }

    /**
@@ -216,32 +183,4 @@ export class Thread extends EventEmitter {
    public has(eventId: string): boolean {
        return this.timelineSet.findEventById(eventId) instanceof MatrixEvent;
    }
-
-    public toJson(): ISerialisedThread {
-        return {
-            id: this.id,
-            tails: Array.from(this.tail),
-        };
-    }
-
-    public on(event: ThreadEvent, listener: (...args: any[]) => void): this {
-        super.on(event, listener);
-        return this;
-    }
-    public once(event: ThreadEvent, listener: (...args: any[]) => void): this {
-        super.once(event, listener);
-        return this;
-    }
-    public off(event: ThreadEvent, listener: (...args: any[]) => void): this {
-        super.off(event, listener);
-        return this;
-    }
-    public addListener(event: ThreadEvent, listener: (...args: any[]) => void): this {
-        super.addListener(event, listener);
-        return this;
-    }
-    public removeListener(event: ThreadEvent, listener: (...args: any[]) => void): this {
-        super.removeListener(event, listener);
-        return this;
-    }
 }
@@ -0,0 +1,37 @@
+/*
+Copyright 2021 The Matrix.org Foundation C.I.C.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+import { BaseModel } from "../models/base-model";
+
+export enum LocalStorageErrors {
+    Global = 'Global',
+    SetItemError = 'setItem',
+    GetItemError = 'getItem',
+    RemoveItemError = 'removeItem',
+    ClearError = 'clear',
+    QuotaExceededError = 'QuotaExceededError'
+}
+
+/**
+ * Used in element-web as a temporary hack to handle all the localStorage errors on the highest level possible
+ * As of 15.11.2021 (DD/MM/YYYY) we're not properly handling local storage exceptions anywhere.
+ * This store, as an event emitter, is used to re-emit local storage exceptions so that we can handle them
+ * and show some kind of a "It's dead Jim" modal to the users, telling them that hey,
+ * maybe you should check out your disk, as it's probably dying and your session may die with it.
+ * See: https://github.com/vector-im/element-web/issues/18423
+ */
+class LocalStorageErrorsEventsEmitter extends BaseModel<LocalStorageErrors> {}
+export const localStorageErrorsEventsEmitter = new LocalStorageErrorsEventsEmitter();
@@ -285,7 +285,7 @@ export class SyncApi {
                }
                leaveObj.timeline = leaveObj.timeline || {};
                const events = this.mapSyncEventsFormat(leaveObj.timeline, room);
-                const [timelineEvents, threadedEvents] = this.partitionThreadedEvents(events);
+                const [timelineEvents, threadedEvents] = this.client.partitionThreadedEvents(events);

                const stateEvents = this.mapSyncEventsFormat(leaveObj.state, room);

@@ -307,39 +307,6 @@ export class SyncApi {
        });
    }

-    /**
-     * Split events between the ones that will end up in the main
-     * room timeline versus the one that need to be processed in a thread
-     * @experimental
-     */
-    public partitionThreadedEvents(events: MatrixEvent[]): [MatrixEvent[], MatrixEvent[]] {
-        if (this.opts.experimentalThreadSupport) {
-            return events.reduce((memo, event: MatrixEvent) => {
-                const room = this.client.getRoom(event.getRoomId());
-                // An event should live in the thread timeline if
-                // - It's a reply in thread event
-                // - It's related to a reply in thread event
-                let shouldLiveInThreadTimeline = event.replyInThread;
-                if (!shouldLiveInThreadTimeline) {
-                    const parentEventId = event.parentEventId;
-                    const parentEvent = room?.findEventById(parentEventId) || events.find((mxEv: MatrixEvent) => {
-                        return mxEv.getId() === parentEventId;
-                    });
-                    shouldLiveInThreadTimeline = parentEvent?.replyInThread;
-                }
-                memo[shouldLiveInThreadTimeline ? 1 : 0].push(event);
-                return memo;
-            }, [[], []]);
-        } else {
-            // When `experimentalThreadSupport` is disabled
-            // treat all events as timelineEvents
-            return [
-                events,
-                [],
-            ];
-        }
-    }
-
    /**
     * Peek into a room. This will result in the room in question being synced so it
     * is accessible via getRooms(). Live updates for the room will be provided.
@@ -1320,7 +1287,7 @@ export class SyncApi {
                }
            }

-            const [timelineEvents, threadedEvents] = this.partitionThreadedEvents(events);
+            const [timelineEvents, threadedEvents] = this.client.partitionThreadedEvents(events);

            this.processRoomEvents(room, stateEvents, timelineEvents, syncEventData.fromCache);
            this.processThreadEvents(room, threadedEvents);
@@ -1388,7 +1355,7 @@ export class SyncApi {
            const events = this.mapSyncEventsFormat(leaveObj.timeline, room);
            const accountDataEvents = this.mapSyncEventsFormat(leaveObj.account_data);

-            const [timelineEvents, threadedEvents] = this.partitionThreadedEvents(events);
+            const [timelineEvents, threadedEvents] = this.client.partitionThreadedEvents(events);

            this.processRoomEvents(room, stateEvents, timelineEvents);
            this.processThreadEvents(room, threadedEvents);
@@ -1734,11 +1701,7 @@ export class SyncApi {
     * @experimental
     */
    private processThreadEvents(room: Room, threadedEvents: MatrixEvent[]): void {
-        threadedEvents
-            .sort((a, b) => a.getTs() - b.getTs())
-            .forEach(event => {
-                room.addThreadedEvent(event);
-            });
+        return this.client.processThreadEvents(room, threadedEvents);
    }

    // extractRelatedEvents(event: MatrixEvent, events: MatrixEvent[], relatedEvents: MatrixEvent[] = []): MatrixEvent[] {
@@ -387,6 +387,18 @@ export function removeHiddenChars(str: string): string {
    return "";
 }

+/**
+ * Removes the direction override characters from a string
+ * @param {string} input
+ * @returns string with chars removed
+ */
+export function removeDirectionOverrideChars(str: string): string {
+    if (typeof str === "string") {
+        return str.replace(/[\u202d-\u202e]/g, '');
+    }
+    return "";
+}
+
 export function normalize(str: string): string {
    // Note: we have to match the filter with the removeHiddenChars() because the
    // function strips spaces and other characters (M becomes RN for example, in lowercase).
@@ -416,10 +416,26 @@ export class MatrixCall extends EventEmitter {
        return this.localUsermediaFeed?.stream;
    }

-    private get localScreensharingStream(): MediaStream {
+    public get localScreensharingStream(): MediaStream {
        return this.localScreensharingFeed?.stream;
    }

+    public get remoteUsermediaFeed(): CallFeed {
+        return this.getRemoteFeeds().find((feed) => feed.purpose === SDPStreamMetadataPurpose.Usermedia);
+    }
+
+    public get remoteScreensharingFeed(): CallFeed {
+        return this.getRemoteFeeds().find((feed) => feed.purpose === SDPStreamMetadataPurpose.Screenshare);
+    }
+
+    public get remoteUsermediaStream(): MediaStream {
+        return this.remoteUsermediaFeed?.stream;
+    }
+
+    public get remoteScreensharingStream(): MediaStream {
+        return this.remoteScreensharingFeed?.stream;
+    }
+
    private getFeedByStreamId(streamId: string): CallFeed {
        return this.getFeeds().find((feed) => feed.stream.id === streamId);
    }
@@ -497,7 +513,15 @@ export class MatrixCall extends EventEmitter {
        if (existingFeed) {
            existingFeed.setNewStream(stream);
        } else {
-            this.feeds.push(new CallFeed(stream, userId, purpose, this.client, this.roomId, audioMuted, videoMuted));
+            this.feeds.push(new CallFeed({
+                client: this.client,
+                roomId: this.roomId,
+                userId,
+                stream,
+                purpose,
+                audioMuted,
+                videoMuted,
+            }));
            this.emit(CallEvent.FeedsChanged, this.feeds);
        }

@@ -528,50 +552,102 @@ export class MatrixCall extends EventEmitter {
        if (feed) {
            feed.setNewStream(stream);
        } else {
-            this.feeds.push(new CallFeed(stream, userId, purpose, this.client, this.roomId, false, false));
+            this.feeds.push(new CallFeed({
+                client: this.client,
+                roomId: this.roomId,
+                audioMuted: false,
+                videoMuted: false,
+                userId,
+                stream,
+                purpose,
+            }));
            this.emit(CallEvent.FeedsChanged, this.feeds);
        }

        logger.info(`Pushed remote stream (id="${stream.id}", active="${stream.active}")`);
    }

-    private pushLocalFeed(stream: MediaStream, purpose: SDPStreamMetadataPurpose, addToPeerConnection = true): void {
+    private pushNewLocalFeed(stream: MediaStream, purpose: SDPStreamMetadataPurpose, addToPeerConnection = true): void {
        const userId = this.client.getUserId();

+        // TODO: Find out what is going on here
+        // why do we enable audio (and only audio) tracks here? -- matthew
+        setTracksEnabled(stream.getAudioTracks(), true);
+
        // We try to replace an existing feed if there already is one with the same purpose
        const existingFeed = this.getLocalFeeds().find((feed) => feed.purpose === purpose);
        if (existingFeed) {
            existingFeed.setNewStream(stream);
        } else {
-            this.feeds.push(new CallFeed(stream, userId, purpose, this.client, this.roomId, false, false));
+            this.pushLocalFeed(
+                new CallFeed({
+                    client: this.client,
+                    roomId: this.roomId,
+                    audioMuted: stream.getAudioTracks().length === 0,
+                    videoMuted: stream.getVideoTracks().length === 0,
+                    userId,
+                    stream,
+                    purpose,
+                }),
+                addToPeerConnection,
+            );
            this.emit(CallEvent.FeedsChanged, this.feeds);
        }
+    }

-        // TODO: Find out what is going on here
-        // why do we enable audio (and only audio) tracks here? -- matthew
-        setTracksEnabled(stream.getAudioTracks(), true);
+    /**
+     * Pushes supplied feed to the call
+     * @param {CallFeed} callFeed to push
+     * @param {boolean} addToPeerConnection whether to add the tracks to the peer connection
+     */
+    public pushLocalFeed(callFeed: CallFeed, addToPeerConnection = true): void {
+        this.feeds.push(callFeed);

        if (addToPeerConnection) {
-            const senderArray = purpose === SDPStreamMetadataPurpose.Usermedia ?
+            const senderArray = callFeed.purpose === SDPStreamMetadataPurpose.Usermedia ?
                this.usermediaSenders : this.screensharingSenders;
            // Empty the array
            senderArray.splice(0, senderArray.length);

-            this.emit(CallEvent.FeedsChanged, this.feeds);
-            for (const track of stream.getTracks()) {
+            for (const track of callFeed.stream.getTracks()) {
                logger.info(
                    `Adding track (` +
                    `id="${track.id}", ` +
                    `kind="${track.kind}", ` +
-                    `streamId="${stream.id}", ` +
-                    `streamPurpose="${purpose}"` +
+                    `streamId="${callFeed.stream.id}", ` +
+                    `streamPurpose="${callFeed.purpose}"` +
                    `) to peer connection`,
                );
-                senderArray.push(this.peerConn.addTrack(track, stream));
+                senderArray.push(this.peerConn.addTrack(track, callFeed.stream));
            }
        }

-        logger.info(`Pushed local stream (id="${stream.id}", active="${stream.active}", purpose="${purpose}")`);
+        logger.info(
+            `Pushed local stream ` +
+            `(id="${callFeed.stream.id}", ` +
+            `active="${callFeed.stream.active}", ` +
+            `purpose="${callFeed.purpose}")`,
+        );
+
+        this.emit(CallEvent.FeedsChanged, this.feeds);
+    }
+
+    /**
+     * Removes local call feed from the call and its tracks from the peer
+     * connection
+     * @param callFeed to remove
+     */
+    public removeLocalFeed(callFeed: CallFeed): void {
+        const senderArray = callFeed.purpose === SDPStreamMetadataPurpose.Usermedia
+            ? this.usermediaSenders
+            : this.screensharingSenders;
+
+        for (const sender of senderArray) {
+            this.peerConn.removeTrack(sender);
+        }
+        // Empty the array
+        senderArray.splice(0, senderArray.length);
+        this.deleteFeedByStream(callFeed.stream);
    }

    private deleteAllFeeds(): void {
@@ -728,25 +804,60 @@ export class MatrixCall extends EventEmitter {
        logger.debug(`Answering call ${this.callId}`);

        if (!this.localUsermediaStream && !this.waitForLocalAVStream) {
+            const prevState = this.state;
+            const answerWithAudio = this.shouldAnswerWithMediaType(audio, this.hasRemoteUserMediaAudioTrack, "audio");
+            const answerWithVideo = this.shouldAnswerWithMediaType(video, this.hasRemoteUserMediaVideoTrack, "video");
+
            this.setState(CallState.WaitLocalMedia);
            this.waitForLocalAVStream = true;

            try {
-                const mediaStream = await this.client.getMediaHandler().getUserMediaStream(
-                    this.shouldAnswerWithMediaType(audio, this.hasRemoteUserMediaAudioTrack, "audio"),
-                    this.shouldAnswerWithMediaType(video, this.hasRemoteUserMediaVideoTrack, "video"),
+                const stream = await this.client.getMediaHandler().getUserMediaStream(
+                    answerWithAudio, answerWithVideo,
                );
                this.waitForLocalAVStream = false;
-                this.gotUserMediaForAnswer(mediaStream);
+                const usermediaFeed = new CallFeed({
+                    client: this.client,
+                    roomId: this.roomId,
+                    userId: this.client.getUserId(),
+                    stream,
+                    purpose: SDPStreamMetadataPurpose.Usermedia,
+                    audioMuted: stream.getAudioTracks().length === 0,
+                    videoMuted: stream.getVideoTracks().length === 0,
+                });
+
+                const feeds = [usermediaFeed];
+
+                if (this.localScreensharingFeed) {
+                    feeds.push(this.localScreensharingFeed);
+                }
+
+                this.answerWithCallFeeds(feeds);
            } catch (e) {
-                this.getUserMediaFailed(e);
-                return;
+                if (answerWithVideo) {
+                    // Try to answer without video
+                    logger.warn("Failed to getUserMedia(), trying to getUserMedia() without video");
+                    this.setState(prevState);
+                    this.waitForLocalAVStream = false;
+                    await this.answer(answerWithAudio, false);
+                } else {
+                    this.getUserMediaFailed(e);
+                    return;
+                }
            }
        } else if (this.waitForLocalAVStream) {
            this.setState(CallState.WaitLocalMedia);
        }
    }

+    public answerWithCallFeeds(callFeeds: CallFeed[]): void {
+        if (this.inviteOrAnswerSent) return;
+
+        logger.debug(`Answering call ${this.callId}`);
+
+        this.gotCallFeedsForAnswer(callFeeds);
+    }
+
    /**
     * Replace this call with a new call, e.g. for glare resolution. Used by
     * MatrixClient.
@@ -758,7 +869,7 @@ export class MatrixCall extends EventEmitter {
            newCall.waitForLocalAVStream = true;
        } else if ([CallState.CreateOffer, CallState.InviteSent].includes(this.state)) {
            logger.debug("Handing local stream to new call");
-            newCall.gotUserMediaForAnswer(this.localUsermediaStream);
+            newCall.gotCallFeedsForAnswer(this.getLocalFeeds());
        }
        this.successor = newCall;
        this.emit(CallEvent.Replaced, newCall);
@@ -830,7 +941,7 @@ export class MatrixCall extends EventEmitter {
                if (this.hasLocalUserMediaAudioTrack) return;
                if (this.hasLocalUserMediaVideoTrack) return;

-                this.pushLocalFeed(stream, SDPStreamMetadataPurpose.Usermedia);
+                this.pushNewLocalFeed(stream, SDPStreamMetadataPurpose.Usermedia);
            } else if (upgradeAudio) {
                if (this.hasLocalUserMediaAudioTrack) return;

@@ -896,12 +1007,10 @@ export class MatrixCall extends EventEmitter {
            try {
                const stream = await this.client.getMediaHandler().getScreensharingStream(desktopCapturerSourceId);
                if (!stream) return false;
-                this.pushLocalFeed(stream, SDPStreamMetadataPurpose.Screenshare);
+                this.pushNewLocalFeed(stream, SDPStreamMetadataPurpose.Screenshare);
                return true;
            } catch (err) {
-                this.emit(CallEvent.Error,
-                    new CallError(CallErrorCode.NoUserMedia, "Failed to get screen-sharing stream: ", err),
-                );
+                logger.error("Failed to get screen-sharing stream:", err);
                return false;
            }
        } else {
@@ -938,13 +1047,11 @@ export class MatrixCall extends EventEmitter {
                });
                sender.replaceTrack(track);

-                this.pushLocalFeed(stream, SDPStreamMetadataPurpose.Screenshare, false);
+                this.pushNewLocalFeed(stream, SDPStreamMetadataPurpose.Screenshare, false);

                return true;
            } catch (err) {
-                this.emit(CallEvent.Error,
-                    new CallError(CallErrorCode.NoUserMedia, "Failed to get screen-sharing stream: ", err),
-                );
+                logger.error("Failed to get screen-sharing stream:", err);
                return false;
            }
        } else {
@@ -969,6 +1076,10 @@ export class MatrixCall extends EventEmitter {
     * @returns the new mute state
     */
    public async setLocalVideoMuted(muted: boolean): Promise<boolean> {
+        if (!await this.client.getMediaHandler().hasVideoDevice()) {
+            return this.isLocalVideoMuted();
+        }
+
        if (!this.hasLocalUserMediaVideoTrack && !muted) {
            await this.upgradeCall(false, true);
            return this.isLocalVideoMuted();
@@ -997,6 +1108,10 @@ export class MatrixCall extends EventEmitter {
     * @returns the new mute state
     */
    public async setMicrophoneMuted(muted: boolean): Promise<boolean> {
+        if (!await this.client.getMediaHandler().hasAudioDevice()) {
+            return this.isMicrophoneMuted();
+        }
+
        if (!this.hasLocalUserMediaAudioTrack && !muted) {
            await this.upgradeCall(true, false);
            return this.isMicrophoneMuted();
@@ -1090,13 +1205,9 @@ export class MatrixCall extends EventEmitter {
        setTracksEnabled(this.localUsermediaStream.getVideoTracks(), !vidShouldBeMuted);
    }

-    /**
-     * Internal
-     * @param {Object} stream
-     */
-    private gotUserMediaForInvite = async (stream: MediaStream): Promise<void> => {
+    private gotCallFeedsForInvite(callFeeds: CallFeed[]): void {
        if (this.successor) {
-            this.successor.gotUserMediaForAnswer(stream);
+            this.successor.gotCallFeedsForAnswer(callFeeds);
            return;
        }
        if (this.callHasEnded()) {
@@ -1104,12 +1215,15 @@ export class MatrixCall extends EventEmitter {
            return;
        }

-        this.pushLocalFeed(stream, SDPStreamMetadataPurpose.Usermedia);
+        for (const feed of callFeeds) {
+            this.pushLocalFeed(feed);
+        }
+
        this.setState(CallState.CreateOffer);

        logger.debug("gotUserMediaForInvite");
        // Now we wait for the negotiationneeded event
-    };
+    }

    private async sendAnswer(): Promise<void> {
        const answerContent = {
@@ -1158,12 +1272,15 @@ export class MatrixCall extends EventEmitter {
        this.sendCandidateQueue();
    }

-    private gotUserMediaForAnswer = async (stream: MediaStream): Promise<void> => {
-        if (this.callHasEnded()) {
-            return;
+    private async gotCallFeedsForAnswer(callFeeds: CallFeed[]): Promise<void> {
+        if (this.callHasEnded()) return;
+
+        this.waitForLocalAVStream = false;
+
+        for (const feed of callFeeds) {
+            this.pushLocalFeed(feed);
        }

-        this.pushLocalFeed(stream, SDPStreamMetadataPurpose.Usermedia);
        this.setState(CallState.CreateAnswer);

        let myAnswer;
@@ -1191,7 +1308,7 @@ export class MatrixCall extends EventEmitter {
            this.terminate(CallParty.Local, CallErrorCode.SetLocalDescription, true);
            return;
        }
-    };
+    }

    /**
     * Internal
@@ -1933,15 +2050,41 @@ export class MatrixCall extends EventEmitter {
     * @throws if have passed audio=false.
     */
    public async placeCall(audio: boolean, video: boolean): Promise<void> {
-        logger.debug(`placeCall audio=${audio} video=${video}`);
        if (!audio) {
            throw new Error("You CANNOT start a call without audio");
        }
+        this.setState(CallState.WaitLocalMedia);
+
+        try {
+            const stream = await this.client.getMediaHandler().getUserMediaStream(audio, video);
+            const callFeed = new CallFeed({
+                client: this.client,
+                roomId: this.roomId,
+                userId: this.client.getUserId(),
+                stream,
+                purpose: SDPStreamMetadataPurpose.Usermedia,
+                audioMuted: stream.getAudioTracks().length === 0,
+                videoMuted: stream.getVideoTracks().length === 0,
+            });
+            await this.placeCallWithCallFeeds([callFeed]);
+        } catch (e) {
+            this.getUserMediaFailed(e);
+            return;
+        }
+    }
+
+    /**
+     * Place a call to this room with call feed.
+     * @param {CallFeed[]} callFeeds to use
+     * @throws if you have not specified a listener for 'error' events.
+     * @throws if have passed audio=false.
+     */
+    public async placeCallWithCallFeeds(callFeeds: CallFeed[]): Promise<void> {
        this.checkForErrorListener();
+        this.direction = CallDirection.Outbound;
+
        // XXX Find a better way to do this
        this.client.callEventHandler.calls.set(this.callId, this);
-        this.setState(CallState.WaitLocalMedia);
-        this.direction = CallDirection.Outbound;

        // make sure we have valid turn creds. Unless something's gone wrong, it should
        // poll and keep the credentials valid so this should be instant.
@@ -1953,14 +2096,7 @@ export class MatrixCall extends EventEmitter {
        // create the peer connection now so it can be gathering candidates while we get user
        // media (assuming a candidate pool size is configured)
        this.peerConn = this.createPeerConnection();
-
-        try {
-            const mediaStream = await this.client.getMediaHandler().getUserMediaStream(audio, video);
-            this.gotUserMediaForInvite(mediaStream);
-        } catch (e) {
-            this.getUserMediaFailed(e);
-            return;
-        }
+        this.gotCallFeedsForInvite(callFeeds);
    }

    private createPeerConnection(): RTCPeerConnection {
@@ -19,8 +19,19 @@ import { SDPStreamMetadataPurpose } from "./callEventTypes";
 import { MatrixClient } from "../client";
 import { RoomMember } from "../models/room-member";

-const POLLING_INTERVAL = 250; // ms
-const SPEAKING_THRESHOLD = -60; // dB
+const POLLING_INTERVAL = 200; // ms
+export const SPEAKING_THRESHOLD = -60; // dB
+const SPEAKING_SAMPLE_COUNT = 8; // samples
+
+export interface ICallFeedOpts {
+    client: MatrixClient;
+    roomId: string;
+    userId: string;
+    stream: MediaStream;
+    purpose: SDPStreamMetadataPurpose;
+    audioMuted: boolean;
+    videoMuted: boolean;
+}

 export enum CallFeedEvent {
    NewStream = "new_stream",
@@ -30,6 +41,15 @@ export enum CallFeedEvent {
 }

 export class CallFeed extends EventEmitter {
+    public stream: MediaStream;
+    public userId: string;
+    public purpose: SDPStreamMetadataPurpose;
+    public speakingVolumeSamples: number[];
+
+    private client: MatrixClient;
+    private roomId: string;
+    private audioMuted: boolean;
+    private videoMuted: boolean;
    private measuringVolumeActivity = false;
    private audioContext: AudioContext;
    private analyser: AnalyserNode;
@@ -38,17 +58,18 @@ export class CallFeed extends EventEmitter {
    private speaking = false;
    private volumeLooperTimeout: number;

-    constructor(
-        public stream: MediaStream,
-        public userId: string,
-        public purpose: SDPStreamMetadataPurpose,
-        private client: MatrixClient,
-        private roomId: string,
-        private audioMuted: boolean,
-        private videoMuted: boolean,
-    ) {
+    constructor(opts: ICallFeedOpts) {
        super();
-        this.updateStream(null, stream);
+
+        this.client = opts.client;
+        this.roomId = opts.roomId;
+        this.userId = opts.userId;
+        this.purpose = opts.purpose;
+        this.audioMuted = opts.audioMuted;
+        this.videoMuted = opts.videoMuted;
+        this.speakingVolumeSamples = new Array(SPEAKING_SAMPLE_COUNT).fill(-Infinity);
+
+        this.updateStream(null, opts.stream);

        if (this.hasAudioTrack) {
            this.initVolumeMeasuring();
@@ -60,6 +81,8 @@ export class CallFeed extends EventEmitter {
    }

    private updateStream(oldStream: MediaStream, newStream: MediaStream): void {
+        if (newStream === oldStream) return;
+
        if (oldStream) {
            oldStream.removeEventListener("addtrack", this.onAddTrack);
            this.measureVolumeActivity(false);
@@ -134,6 +157,10 @@ export class CallFeed extends EventEmitter {
        return this.stream.getVideoTracks().length === 0 || this.videoMuted;
    }

+    public isSpeaking(): boolean {
+        return this.speaking;
+    }
+
    /**
     * Replaces the current MediaStream with a new one.
     * This method should be only used by MatrixCall.
@@ -149,6 +176,7 @@ export class CallFeed extends EventEmitter {
     */
    public setAudioMuted(muted: boolean): void {
        this.audioMuted = muted;
+        this.speakingVolumeSamples.fill(-Infinity);
        this.emit(CallFeedEvent.MuteStateChanged, this.audioMuted, this.videoMuted);
    }

@@ -173,6 +201,7 @@ export class CallFeed extends EventEmitter {
            this.volumeLooper();
        } else {
            this.measuringVolumeActivity = false;
+            this.speakingVolumeSamples.fill(-Infinity);
            this.emit(CallFeedEvent.VolumeChanged, -Infinity);
        }
    }
@@ -181,31 +210,43 @@ export class CallFeed extends EventEmitter {
        this.speakingThreshold = threshold;
    }

-    private volumeLooper(): void {
+    private volumeLooper = () => {
        if (!this.analyser) return;

-        this.volumeLooperTimeout = setTimeout(() => {
-            if (!this.measuringVolumeActivity) return;
+        if (!this.measuringVolumeActivity) return;

-            this.analyser.getFloatFrequencyData(this.frequencyBinCount);
+        this.analyser.getFloatFrequencyData(this.frequencyBinCount);

-            let maxVolume = -Infinity;
-            for (let i = 0; i < this.frequencyBinCount.length; i++) {
-                if (this.frequencyBinCount[i] > maxVolume) {
-                    maxVolume = this.frequencyBinCount[i];
-                }
+        let maxVolume = -Infinity;
+        for (let i = 0; i < this.frequencyBinCount.length; i++) {
+            if (this.frequencyBinCount[i] > maxVolume) {
+                maxVolume = this.frequencyBinCount[i];
            }
+        }

-            this.emit(CallFeedEvent.VolumeChanged, maxVolume);
-            const newSpeaking = maxVolume > this.speakingThreshold;
-            if (this.speaking !== newSpeaking) {
-                this.speaking = newSpeaking;
-                this.emit(CallFeedEvent.Speaking, this.speaking);
+        this.speakingVolumeSamples.shift();
+        this.speakingVolumeSamples.push(maxVolume);
+
+        this.emit(CallFeedEvent.VolumeChanged, maxVolume);
+
+        let newSpeaking = false;
+
+        for (let i = 0; i < this.speakingVolumeSamples.length; i++) {
+            const volume = this.speakingVolumeSamples[i];
+
+            if (volume > this.speakingThreshold) {
+                newSpeaking = true;
+                break;
            }
+        }

-            this.volumeLooper();
-        }, POLLING_INTERVAL);
-    }
+        if (this.speaking !== newSpeaking) {
+            this.speaking = newSpeaking;
+            this.emit(CallFeedEvent.Speaking, this.speaking);
+        }
+
+        this.volumeLooperTimeout = setTimeout(this.volumeLooper, POLLING_INTERVAL);
+    };

    public dispose(): void {
        clearTimeout(this.volumeLooperTimeout);
@@ -43,23 +43,37 @@ export class MediaHandler {
        this.videoInput = deviceId;
    }

+    public async hasAudioDevice(): Promise<boolean> {
+        const devices = await navigator.mediaDevices.enumerateDevices();
+        return devices.filter(device => device.kind === "audioinput").length > 0;
+    }
+
+    public async hasVideoDevice(): Promise<boolean> {
+        const devices = await navigator.mediaDevices.enumerateDevices();
+        return devices.filter(device => device.kind === "videoinput").length > 0;
+    }
+
    /**
     * @returns {MediaStream} based on passed parameters
     */
    public async getUserMediaStream(audio: boolean, video: boolean): Promise<MediaStream> {
+        const shouldRequestAudio = audio && await this.hasAudioDevice();
+        const shouldRequestVideo = video && await this.hasVideoDevice();
+
        let stream: MediaStream;

        // Find a stream with matching tracks
        const matchingStream = this.userMediaStreams.find((stream) => {
-            if (audio !== (stream.getAudioTracks().length > 0)) return false;
-            if (video !== (stream.getVideoTracks().length > 0)) return false;
+            if (shouldRequestAudio !== (stream.getAudioTracks().length > 0)) return false;
+            if (shouldRequestVideo !== (stream.getVideoTracks().length > 0)) return false;
+            return true;
        });

        if (matchingStream) {
            logger.log("Cloning user media stream", matchingStream.id);
            stream = matchingStream.clone();
        } else {
-            const constraints = this.getUserMediaContraints(audio, video);
+            const constraints = this.getUserMediaContraints(shouldRequestAudio, shouldRequestVideo);
            logger.log("Getting user media with constraints", constraints);
            stream = await navigator.mediaDevices.getUserMedia(constraints);
        }
@@ -1,6 +1,7 @@
 {
  "extends": "./tsconfig.json",
  "compilerOptions": {
+    "declarationMap": true,
    "sourceMap": true,
    "noEmit": false,
    "emitDecoratorMetadata": true,
Author	SHA1	Message	Date
RiotRobot	6fff3d6db5	v15.1.1	2021-11-22 13:28:39 +00:00
RiotRobot	8043d83921	Prepare changelog for v15.1.1	2021-11-22 13:28:39 +00:00
RiotRobot	861023b3f6	v15.1.1-rc.1	2021-11-17 13:52:47 +00:00
RiotRobot	1f5a83994b	Prepare changelog for v15.1.1-rc.1	2021-11-17 13:52:47 +00:00
Dariusz Niemczyk	af523522de	Add LocalStorageErrorEventListener (#2019 ) Add a way to listen to LocalStorage error events from element-web and matrix-react-sdk in the same manner. Related: https://github.com/vector-im/element-web/issues/18423	2021-11-15 18:08:20 +00:00
Dariusz Niemczyk	c3a266b3e7	Implement TypedEventEmitter for better TS support (#2018 ) We're using stringly typed events everywhere, this is the first step for better typescript support with our event emitters before we replace it with something much better for React support.	2021-11-15 14:47:21 +00:00
Callum Brown	f41d815aa6	Add registration token UIA type (#2020 )	2021-11-15 09:16:23 +00:00
Germain	ad8a93dde8	Fix check supportExperimentalThreads (#2017 )	2021-11-10 10:54:13 +00:00
Germain	b07d44a6c0	Getter for last thread reply (#2015 )	2021-11-09 14:46:48 +00:00
RiotRobot	1dc899ba6e	Resetting package fields for development	2021-11-08 17:37:26 +00:00
RiotRobot	0b1e3edaff	Merge branch 'master' into develop	2021-11-08 17:37:25 +00:00
RiotRobot	9fb4fed044	v15.1.0	2021-11-08 17:34:08 +00:00
RiotRobot	730b0b121d	Prepare changelog for v15.1.0	2021-11-08 17:34:07 +00:00
Aaron R	9d9d9e2cfa	Fix edit history being broken after editing an unencrypted event with an encrypted event (#2013 )	2021-11-08 08:55:41 +00:00
Germain	43bc09f392	Copy relations to thread root in the thread timeline (#2012 )	2021-11-05 14:16:45 +00:00
Germain	195498e9db	Make events pagination responses parse threads (#2011 )	2021-11-04 11:29:14 +00:00
Faye Duxovni	50332c4999	End authentication attempt immediately if we couldn't find an appropriate flow (#2008 ) If `doRequest()` in `interactive-auth.ts` fails to obtain an appropriate authentication flow, it should return immediately after rejecting the promise. If it continues, it'll attempt to check `chosenFlow.stages`, which will cause an error because `chosenFlow` is `null`. This was breaking the interactive auth spec tests with Node 16.	2021-11-02 10:40:46 -04:00
RiotRobot	9e46646b16	v15.1.0-rc.1	2021-11-02 14:09:01 +00:00
RiotRobot	a74a03e414	Prepare changelog for v15.1.0-rc.1	2021-11-02 14:09:00 +00:00
David Baker	fe8b9eca8e	Update allchange to 1.0.5 (#2010 )	2021-11-02 13:30:49 +00:00
Germain	a7cc1ae630	Fetch thread root event if it's missing from the sync (#2009 )	2021-11-02 11:56:43 +00:00
Šimon Brandner	14e008bbad	Don't emit CallError when we fail to get screen-sharing stream (#2005 ) Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>	2021-10-28 16:23:06 +01:00
Travis Ralston	6a12c265cf	PSFD-455: Try to set a sender on search result events if possible (#2004 ) This is to ensure that search results have aesthetic information such as display name and avatar. Though the membership event won't be in context for when the event was sent, it'll at least be something better than a bare user ID.	2021-10-28 07:48:27 -06:00
Šimon Brandner	c35cb57a79	Port some changes from group calls branch to develop (#2001 ) * Add some useful getters Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> Co-authored-by: Robert Long <robert@robertlong.me> * Add removeLocalFeed() Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> Co-authored-by: Robert Long <robert@robertlong.me> * Don't updateStream() if they're the same Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> Co-authored-by: Robert Long <robert@robertlong.me> * Add isSpeaking() Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> Co-authored-by: Robert Long <robert@robertlong.me> * Improve speaking detection using history Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> Co-authored-by: Robert Long <robert@robertlong.me> * Make code for placing and answering calls more flexible Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> Co-authored-by: Robert Long <robert@robertlong.me> * Correctly log stream id Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Remove mistaken parameter Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Add a unit Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> Co-authored-by: Robert Long <robert@robertlong.me>	2021-10-26 11:59:15 -07:00
Christian Paul	51c776f51e	Correct TypeScript signature of client.sendCompleteEvent (#1997 ) * Correct TypeScript signature of client.sendCompleteEvent * Revert renaming txnId to txnIdOrCallback	2021-10-25 14:25:38 -06:00
Hubert Chathi	7ce0eb26d4	Fetch room membership from server rather than relying on stored data (#1998 ) To work around the possibility that the persisted sync data may be out of date.	2021-10-25 10:29:46 -04:00
RiotRobot	f0091fa811	Resetting package fields for development	2021-10-25 11:25:30 +01:00
RiotRobot	25c88bc986	Merge branch 'master' into develop	2021-10-25 11:25:29 +01:00
RiotRobot	ab06f57965	v15.0.0	2021-10-25 11:22:22 +01:00
RiotRobot	c46209a159	Prepare changelog for v15.0.0	2021-10-25 11:22:21 +01:00
Germain	0c47e27f9f	Add explicit options to disable pending events (#1993 )	2021-10-22 09:23:12 +01:00
Hubert Chathi	0385f265e8	Mark old verification methods as deprecated (#1994 ) The old verification methods start a verification by sending an `m.key.verification.start` message, rather than an `m.key.verification.request` message. This behaviour is deprecated in the spec (https://github.com/matrix-org/matrix-doc/pull/3122), so these functions should be deprecated too.	2021-10-21 16:10:13 -04:00
David Baker	59b3960a42	Strip direction override characters from display names (#1992 ) Strip RLO and LRO characters from name and rawDisplayName so they can safely be embedded into other text without messing up the text ordering. Fixes https://github.com/vector-im/element-web/issues/1712	2021-10-20 17:08:40 +01:00
Germain	d5ab4ba2ef	Update thread replies count to only include m.thread (#1991 )	2021-10-20 16:49:54 +01:00
Travis Ralston	f59b81f46b	Update allchange (#1989 )	2021-10-19 18:21:52 +01:00
Michael Telatynski	4c39ce8c96	Add method to fetch the MSC3266 Room Summary of a Room (#1988 )	2021-10-19 16:13:29 +01:00
Germain Souquet	94ff0ea4cd	Upgrade yarn dependencies	2021-10-19 11:24:53 +01:00
RiotRobot	526758a07f	v15.0.0-rc.1	2021-10-19 10:50:42 +01:00
RiotRobot	9686fba81e	Prepare changelog for v15.0.0-rc.1	2021-10-19 10:50:41 +01:00
Germain	bbc547fd7f	Merge pull request #1982 from matrix-org/gsouquet/thread-event-new	2021-10-15 14:32:19 +01:00
Germain	7d5dbeaf2d	Merge pull request #1981 from matrix-org/gsouquet/threads-root-guard	2021-10-15 14:32:13 +01:00
Germain Souquet	8c19cf45e2	Dispatch a Thread.New event	2021-10-15 11:09:14 +01:00
Germain Souquet	c9277de8ee	Remove console statement	2021-10-15 10:31:43 +01:00
Germain Souquet	b5baca51a6	Null-guard for thread root event not loaded yet	2021-10-15 10:22:32 +01:00
Germain	fc800821f9	Merge pull request #1980 from matrix-org/gsouquet/threads-relations	2021-10-15 08:53:42 +01:00
Germain Souquet	efbc46937f	Null-guard for event that do not have a thread	2021-10-14 17:37:33 +01:00
Germain Souquet	9ceeb2d220	Delete UnstableValue and use hardcoded value for m.thread	2021-10-14 17:09:00 +01:00
Germain Souquet	7b89862ed0	Make threads use 'm.thread' relation	2021-10-14 16:56:48 +01:00
David Baker	6804e4290a	Merge pull request #1972 from SimonBrandner/feature/answer-no-cam Try to answer a call without video if we can't access the camera	2021-10-14 12:15:35 +01:00
David Baker	0a3d820fda	Merge pull request #1943 from freaktechnik/develop Fix `requestVerificationDM` with chronological `pendingEventOrdering`	2021-10-14 12:14:20 +01:00
Michael Telatynski	e2a9c95ea8	Merge pull request #1974 from SimonBrandner/task/security-ts	2021-10-14 10:02:14 +01:00
Šimon Brandner	3aefc9f02e	Add tests for falling back to answering with no video Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>	2021-10-14 08:46:36 +02:00
Šimon Brandner	0042cb5292	Fix importRoomKeys() definition Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>	2021-10-13 16:50:27 +02:00
Denis Kasak	b288c97372	Merge pull request #1975 from matrix-org/dkasak/more-detailed-room-key-sharing-logging More detailed logging when sharing room keys	2021-10-12 13:26:43 +00:00
Denis Kasak	a0de1740a4	Add comma per review Co-authored-by: Michael Telatynski <7t3chguy@gmail.com>	2021-10-12 13:23:22 +00:00
Denis Kasak	8e2e3018d8	Log device objects for existing/new devices when sharing room keys.	2021-10-12 13:48:41 +02:00
Denis Kasak	60e93024e7	Fix comment regarding marking blocked devices as notified.	2021-10-12 13:43:28 +02:00
Denis Kasak	d89aa365f9	More detailed logging of blocked devices. Separately log how many blocked devices there are in total and list them out. Then, log how many devices were notified because they were newly blocked and list those out too.	2021-10-12 13:43:28 +02:00
Denis Kasak	4241263f2c	Make log message regarding notifying unnotified failed devices clearer.	2021-10-12 13:43:26 +02:00
Denis Kasak	b009f9c21b	Log more clearly when we're sharing room keys with new Olm sessions.	2021-10-12 13:41:04 +02:00
Denis Kasak	04ccec6de3	Log which devices room keys were shared with.	2021-10-12 13:41:04 +02:00
RiotRobot	e7b41fadd0	Merge branch 'master' into develop	2021-10-12 08:50:43 +01:00
RiotRobot	2dc1acef2e	Resetting package fields for development	2021-10-12 08:36:13 +01:00
RiotRobot	b977b629f4	Merge branch 'release-v14.0.1'	2021-10-12 08:35:42 +01:00
RiotRobot	8f245af317	Merge branch 'master' into develop	2021-10-11 11:48:22 +01:00
RiotRobot	9b2bfbe7d0	v14.0.0	2021-10-11 11:44:54 +01:00
RiotRobot	f69ea85e55	Prepare changelog for v14.0.0	2021-10-11 11:44:53 +01:00
Šimon Brandner	7559a10526	Make opts in importRoomKeys optional Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>	2021-10-10 08:36:11 +02:00
Šimon Brandner	68dbe959bb	Add a logline Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>	2021-10-10 08:19:29 +02:00
Šimon Brandner	74a875aa23	Add MockMediaDeviceInfo Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>	2021-10-10 08:19:29 +02:00
Šimon Brandner	4b679c5056	Try to answer without video if we can't access it Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>	2021-10-10 08:19:22 +02:00
Šimon Brandner	cb8a260b9c	Improve handling when we don't have a camera Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>	2021-10-09 08:36:19 +02:00
Robert Long	7d7975b8bb	Fix connecting to a call without a webcam Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>	2021-10-09 08:05:59 +02:00
Travis Ralston	f12513da96	Merge pull request #1971 from matrix-org/travis/spam-log Remove excess logging from getRoomUpgradeHistory()	2021-10-08 10:33:07 -06:00
Travis Ralston	785a3e3381	Remove excess logging from getRoomUpgradeHistory() On large accounts this can now cause a massive amount of spam. It's not really providing any value in day-to-day debugging either.	2021-10-08 10:28:20 -06:00
Michael Telatynski	29bc83f4a7	Merge pull request #1968 from matrix-org/t3chguy/fix/19276	2021-10-06 13:27:12 +01:00
Michael Telatynski	13c8774d45	Tweak Room Hierarchy root visibility and add loading getter	2021-10-06 10:25:27 +01:00
Travis Ralston	c9212e770d	Merge pull request #1966 from Alexendoo/declaration-map Enable TypeScript declaration maps	2021-10-05 13:39:47 -06:00
Germain	08aa7803e3	Merge pull request #1964 from SimonBrandner/feature/call-feed-opts	2021-10-04 15:02:49 +01:00
Alex Macleod	770858c255	Enable TypeScript declaration maps This allows go-to-definition to reach the original .ts files for consumers of the package. It emits .d.ts.map files alongside the regular .d.ts files e.g. in sdk.createClient go-to-definition on createClient will place the cursor at the function in src/matrix.ts rather than inside a definition file lib/matrix.d.ts Signed-off-by: Alex Macleod <alex@macleod.io>	2021-10-04 14:36:50 +01:00
Germain Souquet	c73bb7d408	Update dependencies	2021-10-04 12:30:14 +01:00
Šimon Brandner	c63abe9988	Use `ICallFeedOpts` in the `CallFeed` constructor Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>	2021-10-02 08:12:43 +02:00
Martin Giger	482e3c4cb7	Skip waiting if already sent and abort on cancel	2021-09-30 16:58:08 +02:00
Martin Giger	3c97d17770	Fix requestVerificationDM with chronological pendingEventOrdering Signed-off-by: Martin Giger <martin@humanoids.be>	2021-09-21 20:04:07 +02:00