v1.1.0

Prepare changelog for v1.1.0
Merge branch 'develop' into release-v1.1.0
2019-05-07 15:23:32 +01:00 · 2019-05-07 15:23:32 +01:00 · 2019-04-30 11:50:24 +01:00 · 2019-04-30 11:48:38 +01:00 · 2019-04-30 11:43:35 +01:00 · 2019-04-30 11:43:34 +01:00
23 changed files with 863 additions and 323 deletions
@@ -22,3 +22,13 @@ steps:
    plugins:
      - docker#v3.0.1:
          image: "node:10"
+
+  - wait
+
+  - label: "🐴 Trigger matrix-react-sdk"
+    trigger: "matrix-react-sdk"
+    branches: "develop"
+    build:
+        branch: "develop"
+        message: "[js-sdk] ${BUILDKITE_MESSAGE}"
+    async: true
@@ -1,3 +1,78 @@
+Changes in [1.1.0](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.1.0) (2019-05-07)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.1.0-rc.1...v1.1.0)
+
+ * No Changes since rc.1
+
+Changes in [1.1.0-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.1.0-rc.1) (2019-04-30)
+==========================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.4...v1.1.0-rc.1)
+
+ * use the release version of olm 3.1.0
+   [\#903](https://github.com/matrix-org/matrix-js-sdk/pull/903)
+ * Use new Olm repo link in README
+   [\#901](https://github.com/matrix-org/matrix-js-sdk/pull/901)
+ * Support being fed a .well-known config object for validation
+   [\#897](https://github.com/matrix-org/matrix-js-sdk/pull/897)
+ * emit self-membership event at end of handling sync update
+   [\#900](https://github.com/matrix-org/matrix-js-sdk/pull/900)
+ * Use packages.matrix.org for Olm
+   [\#898](https://github.com/matrix-org/matrix-js-sdk/pull/898)
+ * Fix tests on develop
+   [\#899](https://github.com/matrix-org/matrix-js-sdk/pull/899)
+ * Stop syncing when the token is invalid
+   [\#895](https://github.com/matrix-org/matrix-js-sdk/pull/895)
+ * change event redact,  POST request to PUT request
+   [\#887](https://github.com/matrix-org/matrix-js-sdk/pull/887)
+ * Expose better autodiscovery error messages
+   [\#894](https://github.com/matrix-org/matrix-js-sdk/pull/894)
+ * Explicitly guard store usage during sync startup
+   [\#892](https://github.com/matrix-org/matrix-js-sdk/pull/892)
+ * Flag v3 rooms as safe
+   [\#893](https://github.com/matrix-org/matrix-js-sdk/pull/893)
+ * Cache failed capabilities lookups for shorter amounts of time
+   [\#890](https://github.com/matrix-org/matrix-js-sdk/pull/890)
+ * Fix highlight notifications for unencrypted rooms
+   [\#891](https://github.com/matrix-org/matrix-js-sdk/pull/891)
+ * Document checking crypto state before using `hasUnverifiedDevices`
+   [\#889](https://github.com/matrix-org/matrix-js-sdk/pull/889)
+ * Add logging to sync startup path
+   [\#888](https://github.com/matrix-org/matrix-js-sdk/pull/888)
+ * Track e2e highlights better, particularly in 'Mentions Only' rooms
+   [\#886](https://github.com/matrix-org/matrix-js-sdk/pull/886)
+ * support both the incorrect and correct MAC methods
+   [\#882](https://github.com/matrix-org/matrix-js-sdk/pull/882)
+ * Refuse to set forwards pagination token on live timeline
+   [\#885](https://github.com/matrix-org/matrix-js-sdk/pull/885)
+ * Degrade `IndexedDBStore` back to memory only on failure
+   [\#884](https://github.com/matrix-org/matrix-js-sdk/pull/884)
+ * Refuse to link live timelines into the forwards/backwards position when
+   either is invalid
+   [\#877](https://github.com/matrix-org/matrix-js-sdk/pull/877)
+ * Key backup logging improvements
+   [\#883](https://github.com/matrix-org/matrix-js-sdk/pull/883)
+ * Don't assume aborts are always from txn.abort()
+   [\#880](https://github.com/matrix-org/matrix-js-sdk/pull/880)
+ * Add a bunch of logging
+   [\#878](https://github.com/matrix-org/matrix-js-sdk/pull/878)
+ * Refuse splicing the live timeline into a broken position
+   [\#873](https://github.com/matrix-org/matrix-js-sdk/pull/873)
+ * Add existence check to local storage based crypto store
+   [\#872](https://github.com/matrix-org/matrix-js-sdk/pull/872)
+
+Changes in [1.0.4](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.0.4) (2019-04-08)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.3...v1.0.4)
+
+ * Hotfix: more logging and potential fixes for timeline corruption issue, see ticket https://github.com/vector-im/riot-web/issues/8593.
+
+Changes in [1.0.3](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.0.3) (2019-04-01)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.3-rc.1...v1.0.3)
+
+ * Add existence check to local storage based crypto store
+   [\#874](https://github.com/matrix-org/matrix-js-sdk/pull/874)
+
 Changes in [1.0.3-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.0.3-rc.1) (2019-03-27)
 ==========================================================================================================
 [Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.2...v1.0.3-rc.1)
@@ -204,7 +279,7 @@ Changes in [0.14.0-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/ta

 BREAKING CHANGE
 ----------------
- 
+
 * js-sdk now uses Olm 3.0. Apps using Olm must update to 3.0 to
   continue using Olm with the js-sdk. The js-sdk will call Olm's
   init() method when the client is started.
@@ -297,9 +297,9 @@ Then visit ``http://localhost:8005`` to see the API docs.
 End-to-end encryption support
 =============================

-The SDK supports end-to-end encryption via the Olm and Megolm protocols, using
-[libolm](http://matrix.org/git/olm). It is left up to the application to make
-libolm available, via the ``Olm`` global.
+The SDK supports end-to-end encryption via the and Megolm protocols, using
+[libolm](https://gitlab.matrix.org/matrix-org/olm). It is left up to the 
+application to make libolm available, via the ``Olm`` global.

 It is also necessry to call ``matrixClient.initCrypto()`` after creating a new
 ``MatrixClient`` (but **before** calling ``matrixClient.startClient()``) to
@@ -318,18 +318,18 @@ specification.

 To provide the Olm library in a browser application:

- * download the transpiled libolm (from https://matrix.org/packages/npm/olm/).
+ * download the transpiled libolm (from https://packages.matrix.org/npm/olm/).
 * load ``olm.js`` as a ``<script>`` *before* ``browser-matrix.js``.
 
 To provide the Olm library in a node.js application:

- * ``yarn add https://matrix.org/packages/npm/olm/olm-3.0.0.tgz``
+ * ``yarn add https://packages.matrix.org/npm/olm/olm-3.0.0.tgz``
   (replace the URL with the latest version you want to use from
-    https://matrix.org/packages/npm/olm/)
+    https://packages.matrix.org/npm/olm/)
 * ``global.Olm = require('olm');`` *before* loading ``matrix-js-sdk``.

 If you want to package Olm as dependency for your node.js application, you can
-use ``yarn add https://matrix.org/packages/npm/olm/olm-3.0.0.tgz``. If your
+use ``yarn add https://packages.matrix.org/npm/olm/olm-3.0.0.tgz``. If your
 application also works without e2e crypto enabled, add ``--optional`` to mark it
 as an optional dependency.

@@ -1,6 +1,6 @@
 {
  "name": "matrix-js-sdk",
-  "version": "1.0.3-rc.1",
+  "version": "1.1.0",
  "description": "Matrix Client-Server SDK for Javascript",
  "main": "index.js",
  "scripts": {
@@ -82,7 +82,7 @@
    "matrix-mock-request": "^1.2.3",
    "mocha": "^5.2.0",
    "mocha-jenkins-reporter": "^0.4.0",
-    "olm": "https://matrix.org/packages/npm/olm/olm-3.1.0-pre1.tgz",
+    "olm": "https://packages.matrix.org/npm/olm/olm-3.1.0.tgz",
    "rimraf": "^2.5.4",
    "source-map-support": "^0.4.11",
    "sourceify": "^0.1.0",
@@ -157,7 +157,7 @@ describe("MatrixClient events", function() {
                    return;
                }

-                expect(event.event).toEqual(SYNC_DATA.presence.events[0]);
+                expect(event.event).toMatch(SYNC_DATA.presence.events[0]);
                expect(user.presence).toEqual(
                    SYNC_DATA.presence.events[0].content.presence,
                );
@@ -94,7 +94,7 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_PROMPT",
-                        error: "Invalid homeserver discovery response",
+                        error: AutoDiscovery.ERROR_INVALID,
                        base_url: null,
                    },
                    "m.identity_server": {
@@ -117,7 +117,7 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_PROMPT",
-                        error: "Invalid homeserver discovery response",
+                        error: AutoDiscovery.ERROR_INVALID,
                        base_url: null,
                    },
                    "m.identity_server": {
@@ -140,7 +140,7 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_PROMPT",
-                        error: "Invalid homeserver discovery response",
+                        error: AutoDiscovery.ERROR_INVALID,
                        base_url: null,
                    },
                    "m.identity_server": {
@@ -163,7 +163,7 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_PROMPT",
-                        error: "Invalid homeserver discovery response",
+                        error: AutoDiscovery.ERROR_INVALID,
                        base_url: null,
                    },
                    "m.identity_server": {
@@ -191,7 +191,7 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_PROMPT",
-                        error: "Invalid homeserver discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_HS_BASE_URL,
                        base_url: null,
                    },
                    "m.identity_server": {
@@ -217,7 +217,7 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_PROMPT",
-                        error: "Invalid homeserver discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_HS_BASE_URL,
                        base_url: null,
                    },
                    "m.identity_server": {
@@ -245,7 +245,7 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_ERROR",
-                        error: "Invalid homeserver discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_HS_BASE_URL,
                        base_url: null,
                    },
                    "m.identity_server": {
@@ -274,7 +274,7 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_ERROR",
-                        error: "Invalid homeserver discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_HOMESERVER,
                        base_url: null,
                    },
                    "m.identity_server": {
@@ -303,7 +303,7 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_ERROR",
-                        error: "Invalid homeserver discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_HOMESERVER,
                        base_url: null,
                    },
                    "m.identity_server": {
@@ -334,7 +334,7 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_ERROR",
-                        error: "Invalid homeserver discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_HOMESERVER,
                        base_url: null,
                    },
                    "m.identity_server": {
@@ -439,14 +439,14 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_ERROR",
-                        error: "Invalid identity server discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_IS,

                        // We still expect the base_url to be here for debugging purposes.
                        base_url: "https://chat.example.org",
                    },
                    "m.identity_server": {
                        state: "FAIL_ERROR",
-                        error: "Invalid identity server discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_IS_BASE_URL,
                        base_url: null,
                    },
                };
@@ -479,14 +479,14 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_ERROR",
-                        error: "Invalid identity server discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_IS,

                        // We still expect the base_url to be here for debugging purposes.
                        base_url: "https://chat.example.org",
                    },
                    "m.identity_server": {
                        state: "FAIL_ERROR",
-                        error: "Invalid identity server discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_IS_BASE_URL,
                        base_url: null,
                    },
                };
@@ -520,14 +520,14 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_ERROR",
-                        error: "Invalid identity server discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_IS,

                        // We still expect the base_url to be here for debugging purposes.
                        base_url: "https://chat.example.org",
                    },
                    "m.identity_server": {
                        state: "FAIL_ERROR",
-                        error: "Invalid identity server discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_IDENTITY_SERVER,
                        base_url: null,
                    },
                };
@@ -561,14 +561,14 @@ describe("AutoDiscovery", function() {
                const expected = {
                    "m.homeserver": {
                        state: "FAIL_ERROR",
-                        error: "Invalid identity server discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_IS,

                        // We still expect the base_url to be here for debugging purposes
                        base_url: "https://chat.example.org",
                    },
                    "m.identity_server": {
                        state: "FAIL_ERROR",
-                        error: "Invalid identity server discovery response",
+                        error: AutoDiscovery.ERROR_INVALID_IDENTITY_SERVER,
                        base_url: null,
                    },
                };
@@ -58,105 +58,170 @@ describe("SAS verification", function() {
        expect(spy).toHaveBeenCalled();
    });

-    it("should verify a key", async function() {
-        const [alice, bob] = await makeTestClients(
-            [
-                {userId: "@alice:example.com", deviceId: "Osborne2"},
-                {userId: "@bob:example.com", deviceId: "Dynabook"},
-            ],
-            {
-                verificationMethods: [verificationMethods.SAS],
-            },
-        );
-
-        alice.setDeviceVerified = expect.createSpy();
-        alice.getDeviceEd25519Key = () => {
-            return "alice+base64+ed25519+key";
-        };
-        alice.getStoredDevice = () => {
-            return DeviceInfo.fromStorage(
-                {
-                    keys: {
-                        "ed25519:Dynabook": "bob+base64+ed25519+key",
-                    },
-                },
-                "Dynabook",
-            );
-        };
-        alice.downloadKeys = () => {
-            return Promise.resolve();
-        };
-
-        bob.setDeviceVerified = expect.createSpy();
-        bob.getStoredDevice = () => {
-            return DeviceInfo.fromStorage(
-                {
-                    keys: {
-                        "ed25519:Osborne2": "alice+base64+ed25519+key",
-                    },
-                },
-                "Osborne2",
-            );
-        };
-        bob.getDeviceEd25519Key = () => {
-            return "bob+base64+ed25519+key";
-        };
-        bob.downloadKeys = () => {
-            return Promise.resolve();
-        };
-
+    describe("verification", function() {
+        let alice;
+        let bob;
        let aliceSasEvent;
        let bobSasEvent;
+        let aliceVerifier;
+        let bobPromise;

-        const bobPromise = new Promise((resolve, reject) => {
-            bob.on("crypto.verification.start", (verifier) => {
-                verifier.on("show_sas", (e) => {
-                    if (!e.sas.emoji || !e.sas.decimal) {
-                        e.cancel();
-                    } else if (!aliceSasEvent) {
-                        bobSasEvent = e;
-                    } else {
-                        try {
-                            expect(e.sas).toEqual(aliceSasEvent.sas);
-                            e.confirm();
-                            aliceSasEvent.confirm();
-                        } catch (error) {
-                            e.mismatch();
-                            aliceSasEvent.mismatch();
+        beforeEach(async function() {
+            [alice, bob] = await makeTestClients(
+                [
+                    {userId: "@alice:example.com", deviceId: "Osborne2"},
+                    {userId: "@bob:example.com", deviceId: "Dynabook"},
+                ],
+                {
+                    verificationMethods: [verificationMethods.SAS],
+                },
+            );
+
+            alice.setDeviceVerified = expect.createSpy();
+            alice.getDeviceEd25519Key = () => {
+                return "alice+base64+ed25519+key";
+            };
+            alice.getStoredDevice = () => {
+                return DeviceInfo.fromStorage(
+                    {
+                        keys: {
+                            "ed25519:Dynabook": "bob+base64+ed25519+key",
+                        },
+                    },
+                    "Dynabook",
+                );
+            };
+            alice.downloadKeys = () => {
+                return Promise.resolve();
+            };
+
+            bob.setDeviceVerified = expect.createSpy();
+            bob.getStoredDevice = () => {
+                return DeviceInfo.fromStorage(
+                    {
+                        keys: {
+                            "ed25519:Osborne2": "alice+base64+ed25519+key",
+                        },
+                    },
+                    "Osborne2",
+                );
+            };
+            bob.getDeviceEd25519Key = () => {
+                return "bob+base64+ed25519+key";
+            };
+            bob.downloadKeys = () => {
+                return Promise.resolve();
+            };
+
+            aliceSasEvent = null;
+            bobSasEvent = null;
+
+            bobPromise = new Promise((resolve, reject) => {
+                bob.on("crypto.verification.start", (verifier) => {
+                    verifier.on("show_sas", (e) => {
+                        if (!e.sas.emoji || !e.sas.decimal) {
+                            e.cancel();
+                        } else if (!aliceSasEvent) {
+                            bobSasEvent = e;
+                        } else {
+                            try {
+                                expect(e.sas).toEqual(aliceSasEvent.sas);
+                                e.confirm();
+                                aliceSasEvent.confirm();
+                            } catch (error) {
+                                e.mismatch();
+                                aliceSasEvent.mismatch();
+                            }
                        }
-                    }
+                    });
+                    resolve(verifier);
                });
-                resolve(verifier);
+            });
+
+            aliceVerifier = alice.beginKeyVerification(
+                verificationMethods.SAS, bob.getUserId(), bob.deviceId,
+            );
+            aliceVerifier.on("show_sas", (e) => {
+                if (!e.sas.emoji || !e.sas.decimal) {
+                    e.cancel();
+                } else if (!bobSasEvent) {
+                    aliceSasEvent = e;
+                } else {
+                    try {
+                        expect(e.sas).toEqual(bobSasEvent.sas);
+                        e.confirm();
+                        bobSasEvent.confirm();
+                    } catch (error) {
+                        e.mismatch();
+                        bobSasEvent.mismatch();
+                    }
+                }
            });
        });

-        const aliceVerifier = alice.beginKeyVerification(
-            verificationMethods.SAS, bob.getUserId(), bob.deviceId,
-        );
-        aliceVerifier.on("show_sas", (e) => {
-            if (!e.sas.emoji || !e.sas.decimal) {
-                e.cancel();
-            } else if (!bobSasEvent) {
-                aliceSasEvent = e;
-            } else {
-                try {
-                    expect(e.sas).toEqual(bobSasEvent.sas);
-                    e.confirm();
-                    bobSasEvent.confirm();
-                } catch (error) {
-                    e.mismatch();
-                    bobSasEvent.mismatch();
+        it("should verify a key", async function() {
+            let macMethod;
+            const origSendToDevice = alice.sendToDevice;
+            bob.sendToDevice = function(type, map) {
+                if (type === "m.key.verification.accept") {
+                    macMethod = map[alice.getUserId()][alice.deviceId]
+                        .message_authentication_code;
                }
-            }
+                return origSendToDevice.call(this, type, map);
+            };
+
+            await Promise.all([
+                aliceVerifier.verify(),
+                bobPromise.then((verifier) => verifier.verify()),
+            ]);
+
+            // make sure that it uses the preferred method
+            expect(macMethod).toBe("hkdf-hmac-sha256");
+
+            // make sure Alice and Bob verified each other
+            expect(alice.setDeviceVerified)
+                .toHaveBeenCalledWith(bob.getUserId(), bob.deviceId);
+            expect(bob.setDeviceVerified)
+                .toHaveBeenCalledWith(alice.getUserId(), alice.deviceId);
+        });
+
+        it("should be able to verify using the old MAC", async function() {
+            // pretend that Alice can only understand the old (incorrect) MAC,
+            // and make sure that she can still verify with Bob
+            let macMethod;
+            const origSendToDevice = alice.sendToDevice;
+            alice.sendToDevice = function(type, map) {
+                if (type === "m.key.verification.start") {
+                    // Note: this modifies not only the message that Bob
+                    // receives, but also the copy of the message that Alice
+                    // has, since it is the same object.  If this does not
+                    // happen, the verification will fail due to a hash
+                    // commitment mismatch.
+                    map[bob.getUserId()][bob.deviceId]
+                        .message_authentication_codes = ['hmac-sha256'];
+                }
+                return origSendToDevice.call(this, type, map);
+            };
+            bob.sendToDevice = function(type, map) {
+                if (type === "m.key.verification.accept") {
+                    macMethod = map[alice.getUserId()][alice.deviceId]
+                        .message_authentication_code;
+                }
+                return origSendToDevice.call(this, type, map);
+            };
+
+            await Promise.all([
+                aliceVerifier.verify(),
+                bobPromise.then((verifier) => verifier.verify()),
+            ]);
+
+            expect(macMethod).toBe("hmac-sha256");
+
+            expect(alice.setDeviceVerified)
+                .toHaveBeenCalledWith(bob.getUserId(), bob.deviceId);
+            expect(bob.setDeviceVerified)
+                .toHaveBeenCalledWith(alice.getUserId(), alice.deviceId);
        });
-        await Promise.all([
-            aliceVerifier.verify(),
-            bobPromise.then((verifier) => verifier.verify()),
-        ]);
-        expect(alice.setDeviceVerified)
-            .toHaveBeenCalledWith(bob.getUserId(), bob.deviceId);
-        expect(bob.setDeviceVerified)
-            .toHaveBeenCalledWith(alice.getUserId(), alice.deviceId);
    });

    it("should send a cancellation message on error", async function() {
@@ -1318,6 +1318,9 @@ describe("Room", function() {
                    // events should already be MatrixEvents
                    return function(event) {return event;};
                },
+                isCryptoEnabled() {
+                    return true;
+                },
                isRoomEncrypted: function() {
                    return false;
                },
@@ -102,6 +102,56 @@ export class AutoDiscovery {
    // translate the meaning of the states in the spec, but also
    // support our own if needed.

+    static get ERROR_INVALID() {
+        return "Invalid homeserver discovery response";
+    }
+
+    static get ERROR_GENERIC_FAILURE() {
+        return "Failed to get autodiscovery configuration from server";
+    }
+
+    static get ERROR_INVALID_HS_BASE_URL() {
+        return "Invalid base_url for m.homeserver";
+    }
+
+    static get ERROR_INVALID_HOMESERVER() {
+        return "Homeserver URL does not appear to be a valid Matrix homeserver";
+    }
+
+    static get ERROR_INVALID_IS_BASE_URL() {
+        return "Invalid base_url for m.identity_server";
+    }
+
+    static get ERROR_INVALID_IDENTITY_SERVER() {
+        return "Identity server URL does not appear to be a valid identity server";
+    }
+
+    static get ERROR_INVALID_IS() {
+        return "Invalid identity server discovery response";
+    }
+
+    static get ERROR_MISSING_WELLKNOWN() {
+        return "No .well-known JSON file found";
+    }
+
+    static get ERROR_INVALID_JSON() {
+        return "Invalid JSON";
+    }
+
+    static get ALL_ERRORS() {
+        return [
+            AutoDiscovery.ERROR_INVALID,
+            AutoDiscovery.ERROR_GENERIC_FAILURE,
+            AutoDiscovery.ERROR_INVALID_HS_BASE_URL,
+            AutoDiscovery.ERROR_INVALID_HOMESERVER,
+            AutoDiscovery.ERROR_INVALID_IS_BASE_URL,
+            AutoDiscovery.ERROR_INVALID_IDENTITY_SERVER,
+            AutoDiscovery.ERROR_INVALID_IS,
+            AutoDiscovery.ERROR_MISSING_WELLKNOWN,
+            AutoDiscovery.ERROR_INVALID_JSON,
+        ];
+    }
+
    /**
     * The auto discovery failed. The client is expected to communicate
     * the error to the user and refuse logging in.
@@ -137,6 +187,166 @@ export class AutoDiscovery {
     */
    static get SUCCESS() { return "SUCCESS"; }

+    /**
+     * Validates and verifies client configuration information for purposes
+     * of logging in. Such information includes the homeserver URL
+     * and identity server URL the client would want. Additional details
+     * may also be included, and will be transparently brought into the
+     * response object unaltered.
+     * @param {string} wellknown The configuration object itself, as returned
+     * by the .well-known auto-discovery endpoint.
+     * @return {Promise<DiscoveredClientConfig>} Resolves to the verified
+     * configuration, which may include error states. Rejects on unexpected
+     * failure, not when verification fails.
+     */
+    static async fromDiscoveryConfig(wellknown) {
+        // Step 1 is to get the config, which is provided to us here.
+
+        // We default to an error state to make the first few checks easier to
+        // write. We'll update the properties of this object over the duration
+        // of this function.
+        const clientConfig = {
+            "m.homeserver": {
+                state: AutoDiscovery.FAIL_ERROR,
+                error: AutoDiscovery.ERROR_INVALID,
+                base_url: null,
+            },
+            "m.identity_server": {
+                // Technically, we don't have a problem with the identity server
+                // config at this point.
+                state: AutoDiscovery.PROMPT,
+                error: null,
+                base_url: null,
+            },
+        };
+
+        if (!wellknown || !wellknown["m.homeserver"]) {
+            logger.error("No m.homeserver key in config");
+
+            clientConfig["m.homeserver"].state = AutoDiscovery.FAIL_PROMPT;
+            clientConfig["m.homeserver"].error = AutoDiscovery.ERROR_INVALID;
+
+            return Promise.resolve(clientConfig);
+        }
+
+        if (!wellknown["m.homeserver"]["base_url"]) {
+            logger.error("No m.homeserver base_url in config");
+
+            clientConfig["m.homeserver"].state = AutoDiscovery.FAIL_PROMPT;
+            clientConfig["m.homeserver"].error = AutoDiscovery.ERROR_INVALID_HS_BASE_URL;
+
+            return Promise.resolve(clientConfig);
+        }
+
+        // Step 2: Make sure the homeserver URL is valid *looking*. We'll make
+        // sure it points to a homeserver in Step 3.
+        const hsUrl = this._sanitizeWellKnownUrl(
+            wellknown["m.homeserver"]["base_url"],
+        );
+        if (!hsUrl) {
+            logger.error("Invalid base_url for m.homeserver");
+            clientConfig["m.homeserver"].error = AutoDiscovery.ERROR_INVALID_HS_BASE_URL;
+            return Promise.resolve(clientConfig);
+        }
+
+        // Step 3: Make sure the homeserver URL points to a homeserver.
+        const hsVersions = await this._fetchWellKnownObject(
+            `${hsUrl}/_matrix/client/versions`,
+        );
+        if (!hsVersions || !hsVersions.raw["versions"]) {
+            logger.error("Invalid /versions response");
+            clientConfig["m.homeserver"].error = AutoDiscovery.ERROR_INVALID_HOMESERVER;
+            return Promise.resolve(clientConfig);
+        }
+
+        // Step 4: Now that the homeserver looks valid, update our client config.
+        clientConfig["m.homeserver"] = {
+            state: AutoDiscovery.SUCCESS,
+            error: null,
+            base_url: hsUrl,
+        };
+
+        // Step 5: Try to pull out the identity server configuration
+        let isUrl = "";
+        if (wellknown["m.identity_server"]) {
+            // We prepare a failing identity server response to save lines later
+            // in this branch. Note that we also fail the homeserver check in the
+            // object because according to the spec we're supposed to FAIL_ERROR
+            // if *anything* goes wrong with the IS validation, including invalid
+            // format. This means we're supposed to stop discovery completely.
+            const failingClientConfig = {
+                "m.homeserver": {
+                    state: AutoDiscovery.FAIL_ERROR,
+                    error: AutoDiscovery.ERROR_INVALID_IS,
+
+                    // We'll provide the base_url that was previously valid for
+                    // debugging purposes.
+                    base_url: clientConfig["m.homeserver"].base_url,
+                },
+                "m.identity_server": {
+                    state: AutoDiscovery.FAIL_ERROR,
+                    error: AutoDiscovery.ERROR_INVALID_IS,
+                    base_url: null,
+                },
+            };
+
+            // Step 5a: Make sure the URL is valid *looking*. We'll make sure it
+            // points to an identity server in Step 5b.
+            isUrl = this._sanitizeWellKnownUrl(
+                wellknown["m.identity_server"]["base_url"],
+            );
+            if (!isUrl) {
+                logger.error("Invalid base_url for m.identity_server");
+                failingClientConfig["m.identity_server"].error =
+                    AutoDiscovery.ERROR_INVALID_IS_BASE_URL;
+                return Promise.resolve(failingClientConfig);
+            }
+
+            // Step 5b: Verify there is an identity server listening on the provided
+            // URL.
+            const isResponse = await this._fetchWellKnownObject(
+                `${isUrl}/_matrix/identity/api/v1`,
+            );
+            if (!isResponse || !isResponse.raw || isResponse.action !== "SUCCESS") {
+                logger.error("Invalid /api/v1 response");
+                failingClientConfig["m.identity_server"].error =
+                    AutoDiscovery.ERROR_INVALID_IDENTITY_SERVER;
+                return Promise.resolve(failingClientConfig);
+            }
+        }
+
+        // Step 6: Now that the identity server is valid, or never existed,
+        // populate the IS section.
+        if (isUrl && isUrl.length > 0) {
+            clientConfig["m.identity_server"] = {
+                state: AutoDiscovery.SUCCESS,
+                error: null,
+                base_url: isUrl,
+            };
+        }
+
+        // Step 7: Copy any other keys directly into the clientConfig. This is for
+        // things like custom configuration of services.
+        Object.keys(wellknown)
+            .map((k) => {
+                if (k === "m.homeserver" || k === "m.identity_server") {
+                    // Only copy selected parts of the config to avoid overwriting
+                    // properties computed by the validation logic above.
+                    const notProps = ["error", "state", "base_url"];
+                    for (const prop of Object.keys(wellknown[k])) {
+                        if (notProps.includes(prop)) continue;
+                        clientConfig[k][prop] = wellknown[k][prop];
+                    }
+                } else {
+                    // Just copy the whole thing over otherwise
+                    clientConfig[k] = wellknown[k];
+                }
+            });
+
+        // Step 8: Give the config to the caller (finally)
+        return Promise.resolve(clientConfig);
+    }
+
    /**
     * Attempts to automatically discover client configuration information
     * prior to logging in. Such information includes the homeserver URL
@@ -171,7 +381,7 @@ export class AutoDiscovery {
        const clientConfig = {
            "m.homeserver": {
                state: AutoDiscovery.FAIL_ERROR,
-                error: "Invalid homeserver discovery response",
+                error: AutoDiscovery.ERROR_INVALID,
                base_url: null,
            },
            "m.identity_server": {
@@ -188,10 +398,8 @@ export class AutoDiscovery {
        const wellknown = await this._fetchWellKnownObject(
            `https://${domain}/.well-known/matrix/client`,
        );
-        if (!wellknown || wellknown.action !== "SUCCESS"
-            || !wellknown.raw["m.homeserver"]
-            || !wellknown.raw["m.homeserver"]["base_url"]) {
-            logger.error("No m.homeserver key in well-known response");
+        if (!wellknown || wellknown.action !== "SUCCESS") {
+            logger.error("No response or error when parsing .well-known");
            if (wellknown.reason) logger.error(wellknown.reason);
            if (wellknown.action === "IGNORE") {
                clientConfig["m.homeserver"] = {
@@ -202,99 +410,13 @@ export class AutoDiscovery {
            } else {
                // this can only ever be FAIL_PROMPT at this point.
                clientConfig["m.homeserver"].state = AutoDiscovery.FAIL_PROMPT;
+                clientConfig["m.homeserver"].error = AutoDiscovery.ERROR_INVALID;
            }
            return Promise.resolve(clientConfig);
        }

-        // Step 2: Make sure the homeserver URL is valid *looking*. We'll make
-        // sure it points to a homeserver in Step 3.
-        const hsUrl = this._sanitizeWellKnownUrl(
-            wellknown.raw["m.homeserver"]["base_url"],
-        );
-        if (!hsUrl) {
-            logger.error("Invalid base_url for m.homeserver");
-            return Promise.resolve(clientConfig);
-        }
-
-        // Step 3: Make sure the homeserver URL points to a homeserver.
-        const hsVersions = await this._fetchWellKnownObject(
-            `${hsUrl}/_matrix/client/versions`,
-        );
-        if (!hsVersions || !hsVersions.raw["versions"]) {
-            logger.error("Invalid /versions response");
-            return Promise.resolve(clientConfig);
-        }
-
-        // Step 4: Now that the homeserver looks valid, update our client config.
-        clientConfig["m.homeserver"] = {
-            state: AutoDiscovery.SUCCESS,
-            error: null,
-            base_url: hsUrl,
-        };
-
-        // Step 5: Try to pull out the identity server configuration
-        let isUrl = "";
-        if (wellknown.raw["m.identity_server"]) {
-            // We prepare a failing identity server response to save lines later
-            // in this branch. Note that we also fail the homeserver check in the
-            // object because according to the spec we're supposed to FAIL_ERROR
-            // if *anything* goes wrong with the IS validation, including invalid
-            // format. This means we're supposed to stop discovery completely.
-            const failingClientConfig = {
-                "m.homeserver": {
-                    state: AutoDiscovery.FAIL_ERROR,
-                    error: "Invalid identity server discovery response",
-
-                    // We'll provide the base_url that was previously valid for
-                    // debugging purposes.
-                    base_url: clientConfig["m.homeserver"].base_url,
-                },
-                "m.identity_server": {
-                    state: AutoDiscovery.FAIL_ERROR,
-                    error: "Invalid identity server discovery response",
-                    base_url: null,
-                },
-            };
-
-            // Step 5a: Make sure the URL is valid *looking*. We'll make sure it
-            // points to an identity server in Step 5b.
-            isUrl = this._sanitizeWellKnownUrl(
-                wellknown.raw["m.identity_server"]["base_url"],
-            );
-            if (!isUrl) {
-                logger.error("Invalid base_url for m.identity_server");
-                return Promise.resolve(failingClientConfig);
-            }
-
-            // Step 5b: Verify there is an identity server listening on the provided
-            // URL.
-            const isResponse = await this._fetchWellKnownObject(
-                `${isUrl}/_matrix/identity/api/v1`,
-            );
-            if (!isResponse || !isResponse.raw || isResponse.action !== "SUCCESS") {
-                logger.error("Invalid /api/v1 response");
-                return Promise.resolve(failingClientConfig);
-            }
-        }
-
-        // Step 6: Now that the identity server is valid, or never existed,
-        // populate the IS section.
-        if (isUrl && isUrl.length > 0) {
-            clientConfig["m.identity_server"] = {
-                state: AutoDiscovery.SUCCESS,
-                error: null,
-                base_url: isUrl,
-            };
-        }
-
-        // Step 7: Copy any other keys directly into the clientConfig. This is for
-        // things like custom configuration of services.
-        Object.keys(wellknown.raw)
-            .filter((k) => k !== "m.homeserver" && k !== "m.identity_server")
-            .map((k) => clientConfig[k] = wellknown.raw[k]);
-
-        // Step 8: Give the config to the caller (finally)
-        return Promise.resolve(clientConfig);
+        // Step 2: Validate and parse the config
+        return AutoDiscovery.fromDiscoveryConfig(wellknown.raw);
    }

    /**
@@ -365,7 +487,7 @@ export class AutoDiscovery {
                        let reason = (err ? err.message : null) || "General failure";
                        if (response.statusCode === 404) {
                            action = "IGNORE";
-                            reason = "No .well-known JSON file found";
+                            reason = AutoDiscovery.ERROR_MISSING_WELLKNOWN;
                        }
                        resolve({raw: {}, action: action, reason: reason, error: err});
                        return;
@@ -374,12 +496,15 @@ export class AutoDiscovery {
                    try {
                        resolve({raw: JSON.parse(body), action: "SUCCESS"});
                    } catch (e) {
-                        let reason = "General failure";
-                        if (e.name === "SyntaxError") reason = "Invalid JSON";
+                        let reason = AutoDiscovery.ERROR_INVALID;
+                        if (e.name === "SyntaxError") {
+                            reason = AutoDiscovery.ERROR_INVALID_JSON;
+                        }
                        resolve({
                            raw: {},
                            action: "FAIL_PROMPT",
-                            reason: reason, error: e,
+                            reason: reason,
+                            error: e,
                        });
                    }
                },
@@ -891,18 +891,29 @@ MatrixBaseApis.prototype.sendStateEvent = function(roomId, eventType, content, s
 /**
 * @param {string} roomId
 * @param {string} eventId
+ * @param {string} [txnId]  transaction id. One will be made up if not
+ *    supplied.
 * @param {module:client.callback} callback Optional.
 * @return {module:client.Promise} Resolves: TODO
 * @return {module:http-api.MatrixError} Rejects: with an error response.
 */
-MatrixBaseApis.prototype.redactEvent = function(roomId, eventId, callback) {
-    const path = utils.encodeUri("/rooms/$roomId/redact/$eventId", {
+MatrixBaseApis.prototype.redactEvent = function(
+    roomId, eventId, txnId, callback,
+) {
+    if (arguments.length === 3) {
+        callback = txnId;
+    }
+
+    const path = utils.encodeUri("/rooms/$roomId/redact/$eventId/$tnxId", {
        $roomId: roomId,
        $eventId: eventId,
+        $txnId: txnId ? txnId : this.makeTxnId(),
    });
-    return this._http.authedRequest(callback, "POST", path, undefined, {});
+
+    return this._http.authedRequest(callback, "PUT", path, undefined, {});
 };

+
 /**
 * @param {string} roomId
 * @param {Number} limit
@@ -242,19 +242,32 @@ function MatrixClient(opts) {
        const actions = this._pushProcessor.actionsForEvent(event);
        event.setPushActions(actions); // Might as well while we're here

+        const room = this.getRoom(event.getRoomId());
+        if (!room) return;
+
+        const currentCount = room.getUnreadNotificationCount("highlight");
+
        // Ensure the unread counts are kept up to date if the event is encrypted
+        // We also want to make sure that the notification count goes up if we already
+        // have encrypted events to avoid other code from resetting 'highlight' to zero.
        const oldHighlight = oldActions && oldActions.tweaks
            ? !!oldActions.tweaks.highlight : false;
        const newHighlight = actions && actions.tweaks
            ? !!actions.tweaks.highlight : false;
-        if (oldHighlight !== newHighlight) {
-            const room = this.getRoom(event.getRoomId());
+        if (oldHighlight !== newHighlight || currentCount > 0) {
            // TODO: Handle mentions received while the client is offline
            // See also https://github.com/vector-im/riot-web/issues/9069
-            if (room && !room.hasUserReadEvent(this.getUserId(), event.getId())) {
-                const current = room.getUnreadNotificationCount("highlight");
-                const newCount = newHighlight ? current + 1 : current - 1;
+            if (!room.hasUserReadEvent(this.getUserId(), event.getId())) {
+                let newCount = currentCount;
+                if (newHighlight && !oldHighlight) newCount++;
+                if (!newHighlight && oldHighlight) newCount--;
                room.setUnreadNotificationCount("highlight", newCount);
+
+                // Fix 'Mentions Only' rooms from not having the right badge count
+                const totalCount = room.getUnreadNotificationCount('total');
+                if (totalCount < newCount) {
+                    room.setUnreadNotificationCount('total', newCount);
+                }
            }
        }
    });
@@ -432,9 +445,10 @@ MatrixClient.prototype.setNotifTimelineSet = function(notifTimelineSet) {
 * @return {module:http-api.MatrixError} Rejects: with an error response.
 */
 MatrixClient.prototype.getCapabilities = function() {
+    const now = new Date().getTime();
+
    if (this._cachedCapabilities) {
-        const now = new Date().getTime();
-        if (now - this._cachedCapabilities.lastUpdated <= CAPABILITIES_CACHE_MS) {
+        if (now < this._cachedCapabilities.expiration) {
            console.log("Returning cached capabilities");
            return Promise.resolve(this._cachedCapabilities.capabilities);
        }
@@ -443,12 +457,22 @@ MatrixClient.prototype.getCapabilities = function() {
    // We swallow errors because we need a default object anyhow
    return this._http.authedRequest(
        undefined, "GET", "/capabilities",
-    ).catch(() => null).then((r) => {
+    ).catch((e) => {
+        console.error(e);
+        return null; // otherwise consume the error
+    }).then((r) => {
        if (!r) r = {};
        const capabilities = r["capabilities"] || {};
+
+        // If the capabilities missed the cache, cache it for a shorter amount
+        // of time to try and refresh them later.
+        const cacheMs = Object.keys(capabilities).length
+            ? CAPABILITIES_CACHE_MS
+            : 60000 + (Math.random() * 5000);
+
        this._cachedCapabilities = {
            capabilities: capabilities,
-            lastUpdated: new Date().getTime(),
+            expiration: now + cacheMs,
        };

        console.log("Caching capabilities: ", capabilities);
@@ -491,6 +515,7 @@ MatrixClient.prototype.initCrypto = async function() {
    }

    // initialise the list of encrypted rooms (whether or not crypto is enabled)
+    console.log("Crypto: initialising roomlist...");
    await this._roomList.init();

    const userId = this.getUserId();
@@ -525,6 +550,7 @@ MatrixClient.prototype.initCrypto = async function() {
        "crypto.warning",
    ]);

+    console.log("Crypto: initialising crypto object...");
    await crypto.init();

    this.olmVersion = Crypto.getOlmVersion();
@@ -197,8 +197,11 @@ utils.inherits(Crypto, EventEmitter);
 * Returns a promise which resolves once the crypto module is ready for use.
 */
 Crypto.prototype.init = async function() {
+    console.log("Crypto: initialising Olm...");
    await global.Olm.init();
+    console.log("Crypto: initialising Olm device...");
    await this._olmDevice.init();
+    console.log("Crypto: loading device list...");
    await this._deviceList.load();

    // build our device keys: these will later be uploaded
@@ -207,6 +210,7 @@ Crypto.prototype.init = async function() {
    this._deviceKeys["curve25519:" + this._deviceId] =
        this._olmDevice.deviceCurve25519Key;

+    console.log("Crypto: fetching own devices...");
    let myDevices = this._deviceList.getRawStoredDevicesForUser(
        this._userId,
    );
@@ -217,6 +221,7 @@ Crypto.prototype.init = async function() {

    if (!myDevices[this._deviceId]) {
        // add our own deviceinfo to the cryptoStore
+        console.log("Crypto: adding this device to the store...");
        const deviceInfo = {
            keys: this._deviceKeys,
            algorithms: this._supportedAlgorithms,
@@ -231,6 +236,7 @@ Crypto.prototype.init = async function() {
        this._deviceList.saveIfDirty();
    }

+    console.log("Crypto: checking for key backup...");
    this._checkAndStartKeyBackup();
 };

@@ -351,7 +357,12 @@ Crypto.prototype.isKeyBackupTrusted = async function(backupInfo) {
    const mySigs = backupInfo.auth_data.signatures[this._userId] || [];

    for (const keyId of Object.keys(mySigs)) {
-        const sigInfo = { deviceId: keyId.split(':')[1] }; // XXX: is this how we're supposed to get the device ID?
+        const keyIdParts = keyId.split(':');
+        if (keyIdParts[0] !== 'ed25519') {
+            console.log("Ignoring unknown signature type: " + keyIdParts[0]);
+            continue;
+        }
+        const sigInfo = { deviceId: keyIdParts[1] }; // XXX: is this how we're supposed to get the device ID?
        const device = this._deviceList.getStoredDevice(
            this._userId, sigInfo.deviceId,
        );
@@ -367,7 +378,7 @@ Crypto.prototype.isKeyBackupTrusted = async function(backupInfo) {
                );
                sigInfo.valid = true;
            } catch (e) {
-                logger.info("Bad signature from device " + device.deviceId, e);
+                logger.info("Bad signature from key ID " + keyId, e);
                sigInfo.valid = false;
            }
        } else {
@@ -694,9 +694,17 @@ function promiseifyTxn(txn) {
            if (txn._mx_abortexception !== undefined) {
                reject(txn._mx_abortexception);
            } else {
+                console.log("Error performing indexeddb txn", event);
+                reject(event.target.error);
+            }
+        };
+        txn.onabort = (event) => {
+            if (txn._mx_abortexception !== undefined) {
+                reject(txn._mx_abortexception);
+            } else {
+                console.log("Error performing indexeddb txn", event);
                reject(event.target.error);
            }
        };
-        txn.onabort = () => reject(txn._mx_abortexception);
    });
 }
@@ -90,6 +90,7 @@ export default class IndexedDBCryptoStore {
            };

            req.onerror = (ev) => {
+                console.log("Error connecting to indexeddb", ev);
                reject(ev.target.error);
            };

@@ -159,6 +160,7 @@ export default class IndexedDBCryptoStore {
            };

            req.onerror = (ev) => {
+                console.log("Error deleting data from indexeddb", ev);
                reject(ev.target.error);
            };

@@ -57,6 +57,16 @@ export default class LocalStorageCryptoStore extends MemoryCryptoStore {
        this.store = webStore;
    }

+    static exists(webStore) {
+        const length = webStore.length;
+        for (let i = 0; i < length; i++) {
+            if (webStore.key(i).startsWith(E2E_PREFIX)) {
+                return true;
+            }
+        }
+        return false;
+    }
+
    // Olm Sessions

    countEndToEndSessions(txn, func) {
@@ -143,17 +143,44 @@ function generateEmojiSas(sasBytes) {
    return emojis.map((num) => emojiMapping[num]);
 }

+const sasGenerators = {
+    decimal: generateDecimalSas,
+    emoji: generateEmojiSas,
+};
+
 function generateSas(sasBytes, methods) {
    const sas = {};
-    if (methods.includes("decimal")) {
-        sas["decimal"] = generateDecimalSas(sasBytes);
-    }
-    if (methods.includes("emoji")) {
-        sas["emoji"] = generateEmojiSas(sasBytes);
+    for (const method of methods) {
+        if (method in sasGenerators) {
+            sas[method] = sasGenerators[method](sasBytes);
+        }
    }
    return sas;
 }

+const macMethods = {
+    "hkdf-hmac-sha256": "calculate_mac",
+    "hmac-sha256": "calculate_mac_long_kdf",
+};
+
+/* lists of algorithms/methods that are supported.  The key agreement, hashes,
+ * and MAC lists should be sorted in order of preference (most preferred
+ * first).
+ */
+const KEY_AGREEMENT_LIST = ["curve25519"];
+const HASHES_LIST = ["sha256"];
+const MAC_LIST = ["hkdf-hmac-sha256", "hmac-sha256"];
+const SAS_LIST = Object.keys(sasGenerators);
+
+const KEY_AGREEMENT_SET = new Set(KEY_AGREEMENT_LIST);
+const HASHES_SET = new Set(HASHES_LIST);
+const MAC_SET = new Set(MAC_LIST);
+const SAS_SET = new Set(SAS_LIST);
+
+function intersection(anArray, aSet) {
+    return anArray instanceof Array ? anArray.filter(x => aSet.has(x)) : [];
+}
+
 /**
 * @alias module:crypto/verification/SAS
 * @extends {module:crypto/verification/Base}
@@ -181,11 +208,11 @@ export default class SAS extends Base {
        const initialMessage = {
            method: SAS.NAME,
            from_device: this._baseApis.deviceId,
-            key_agreement_protocols: ["curve25519"],
-            hashes: ["sha256"],
-            message_authentication_codes: ["hmac-sha256"],
+            key_agreement_protocols: KEY_AGREEMENT_LIST,
+            hashes: HASHES_LIST,
+            message_authentication_codes: MAC_LIST,
            // FIXME: allow app to specify what SAS methods can be used
-            short_authentication_string: ["decimal", "emoji"],
+            short_authentication_string: SAS_LIST,
            transaction_id: this.transactionId,
        };
        this._sendToDevice("m.key.verification.start", initialMessage);
@@ -193,19 +220,19 @@ export default class SAS extends Base {

        let e = await this._waitForEvent("m.key.verification.accept");
        let content = e.getContent();
-        if (!(content.key_agreement_protocol === "curve25519"
-              && content.hash === "sha256"
-              && content.message_authentication_code === "hmac-sha256"
-              && content.short_authentication_string instanceof Array
-              && (content.short_authentication_string.includes("decimal")
-                  || content.short_authentication_string.includes("emoji")))) {
+        const sasMethods
+              = intersection(content.short_authentication_string, SAS_SET);
+        if (!(KEY_AGREEMENT_SET.has(content.key_agreement_protocol)
+              && HASHES_SET.has(content.hash)
+              && MAC_SET.has(content.message_authentication_code)
+              && sasMethods.length)) {
            throw newUnknownMethodError();
        }
        if (typeof content.commitment !== "string") {
            throw newInvalidMessageError();
        }
+        const macMethod = content.message_authentication_code;
        const hashCommitment = content.commitment;
-        const sasMethods = content.short_authentication_string;
        const olmSAS = new global.Olm.SAS();
        try {
            this._sendToDevice("m.key.verification.key", {
@@ -217,6 +244,7 @@ export default class SAS extends Base {
            // FIXME: make sure event is properly formed
            content = e.getContent();
            const commitmentStr = content.key + anotherjson.stringify(initialMessage);
+            // TODO: use selected hash function (when we support multiple)
            if (olmutil.sha256(commitmentStr) !== hashCommitment) {
                throw newMismatchedCommitmentError();
            }
@@ -231,7 +259,7 @@ export default class SAS extends Base {
                this.emit("show_sas", {
                    sas: generateSas(sasBytes, sasMethods),
                    confirm: () => {
-                        this._sendMAC(olmSAS);
+                        this._sendMAC(olmSAS, macMethod);
                        resolve();
                    },
                    cancel: () => reject(newUserCancelledError()),
@@ -245,7 +273,7 @@ export default class SAS extends Base {
                verifySAS,
            ]);
            content = e.getContent();
-            await this._checkMAC(olmSAS, content);
+            await this._checkMAC(olmSAS, content, macMethod);
        } finally {
            olmSAS.free();
        }
@@ -253,34 +281,37 @@ export default class SAS extends Base {

    async _doRespondVerification() {
        let content = this.startEvent.getContent();
-        if (!(content.key_agreement_protocols instanceof Array
-              && content.key_agreement_protocols.includes("curve25519")
-              && content.hashes instanceof Array
-              && content.hashes.includes("sha256")
-              && content.message_authentication_codes instanceof Array
-              && content.message_authentication_codes.includes("hmac-sha256")
-              && content.short_authentication_string instanceof Array
-              && (content.short_authentication_string.includes("decimal")
-                  || content.short_authentication_string.includes("emoji")))) {
+        // Note: we intersect using our pre-made lists, rather than the sets,
+        // so that the result will be in our order of preference.  Then
+        // fetching the first element from the array will give our preferred
+        // method out of the ones offered by the other party.
+        const keyAgreement
+              = intersection(
+                  KEY_AGREEMENT_LIST, new Set(content.key_agreement_protocols),
+              )[0];
+        const hashMethod
+              = intersection(HASHES_LIST, new Set(content.hashes))[0];
+        const macMethod
+              = intersection(MAC_LIST, new Set(content.message_authentication_codes))[0];
+        // FIXME: allow app to specify what SAS methods can be used
+        const sasMethods
+              = intersection(content.short_authentication_string, SAS_SET);
+        if (!(keyAgreement !== undefined
+              && hashMethod !== undefined
+              && macMethod !== undefined
+              && sasMethods.length)) {
            throw newUnknownMethodError();
        }

        const olmSAS = new global.Olm.SAS();
-        const sasMethods = [];
-        // FIXME: allow app to specify what SAS methods can be used
-        if (content.short_authentication_string.includes("decimal")) {
-            sasMethods.push("decimal");
-        }
-        if (content.short_authentication_string.includes("emoji")) {
-            sasMethods.push("emoji");
-        }
        try {
            const commitmentStr = olmSAS.get_pubkey() + anotherjson.stringify(content);
            this._sendToDevice("m.key.verification.accept", {
-                key_agreement_protocol: "curve25519",
-                hash: "sha256",
-                message_authentication_code: "hmac-sha256",
+                key_agreement_protocol: keyAgreement,
+                hash: hashMethod,
+                message_authentication_code: macMethod,
                short_authentication_string: sasMethods,
+                // TODO: use selected hash function (when we support multiple)
                commitment: olmutil.sha256(commitmentStr),
            });

@@ -302,7 +333,7 @@ export default class SAS extends Base {
                this.emit("show_sas", {
                    sas: generateSas(sasBytes, sasMethods),
                    confirm: () => {
-                        this._sendMAC(olmSAS);
+                        this._sendMAC(olmSAS, macMethod);
                        resolve();
                    },
                    cancel: () => reject(newUserCancelledError()),
@@ -316,13 +347,13 @@ export default class SAS extends Base {
                verifySAS,
            ]);
            content = e.getContent();
-            await this._checkMAC(olmSAS, content);
+            await this._checkMAC(olmSAS, content, macMethod);
        } finally {
            olmSAS.free();
        }
    }

-    _sendMAC(olmSAS) {
+    _sendMAC(olmSAS, method) {
        const keyId = `ed25519:${this._baseApis.deviceId}`;
        const mac = {};
        const baseInfo = "MATRIX_KEY_VERIFICATION_MAC"
@@ -330,24 +361,24 @@ export default class SAS extends Base {
              + this.userId + this.deviceId
              + this.transactionId;

-        mac[keyId] = olmSAS.calculate_mac(
+        mac[keyId] = olmSAS[macMethods[method]](
            this._baseApis.getDeviceEd25519Key(),
            baseInfo + keyId,
        );
-        const keys = olmSAS.calculate_mac(
+        const keys = olmSAS[macMethods[method]](
            keyId,
            baseInfo + "KEY_IDS",
        );
        this._sendToDevice("m.key.verification.mac", { mac, keys });
    }

-    async _checkMAC(olmSAS, content) {
+    async _checkMAC(olmSAS, content, method) {
        const baseInfo = "MATRIX_KEY_VERIFICATION_MAC"
              + this.userId + this.deviceId
              + this._baseApis.getUserId() + this._baseApis.deviceId
              + this.transactionId;

-        if (content.keys !== olmSAS.calculate_mac(
+        if (content.keys !== olmSAS[macMethods[method]](
            Object.keys(content.mac).sort().join(","),
            baseInfo + "KEY_IDS",
        )) {
@@ -355,7 +386,7 @@ export default class SAS extends Base {
        }

        await this._verifyKeys(this.userId, content.mac, (keyId, device, keyInfo) => {
-            if (keyInfo !== olmSAS.calculate_mac(
+            if (keyInfo !== olmSAS[macMethods[method]](
                device.keys[keyId],
                baseInfo + keyId,
            )) {
@@ -408,8 +408,36 @@ EventTimelineSet.prototype.addEventsToTimeline = function(events, toStartOfTimel
        console.info("Already have timeline for " + eventId +
                     " - joining timeline " + timeline + " to " +
                     existingTimeline);
+
+        // Variables to keep the line length limited below.
+        const existingIsLive = existingTimeline === this._liveTimeline;
+        const timelineIsLive = timeline === this._liveTimeline;
+
+        const backwardsIsLive = direction === EventTimeline.BACKWARDS && existingIsLive;
+        const forwardsIsLive = direction === EventTimeline.FORWARDS && timelineIsLive;
+
+        if (backwardsIsLive || forwardsIsLive) {
+            // The live timeline should never be spliced into a non-live position.
+            // We use independent logging to better discover the problem at a glance.
+            console.warn({backwardsIsLive, forwardsIsLive}); // debugging
+            if (backwardsIsLive) {
+                console.warn(
+                    "Refusing to set a preceding existingTimeLine on our " +
+                    "timeline as the existingTimeLine is live (" + existingTimeline + ")",
+                );
+            }
+            if (forwardsIsLive) {
+                console.warn(
+                    "Refusing to set our preceding timeline on a existingTimeLine " +
+                    "as our timeline is live (" + timeline + ")",
+                );
+            }
+            continue; // abort splicing - try next event
+        }
+
        timeline.setNeighbouringTimeline(existingTimeline, direction);
        existingTimeline.setNeighbouringTimeline(timeline, inverseDirection);
+
        timeline = existingTimeline;
        didUpdate = true;
    }
@@ -418,6 +446,14 @@ EventTimelineSet.prototype.addEventsToTimeline = function(events, toStartOfTimel
    // new information, we update the pagination token for whatever
    // timeline we ended up on.
    if (lastEventWasNew || !didUpdate) {
+        if (direction === EventTimeline.FORWARDS && timeline === this._liveTimeline) {
+            console.warn({lastEventWasNew, didUpdate}); // for debugging
+            console.warn(
+                `Refusing to set forwards pagination token of live timeline ` +
+                `${timeline} to ${paginationToken}`,
+            );
+            return;
+        }
        timeline.setPaginationToken(paginationToken, direction);
    }
 };
@@ -276,7 +276,7 @@ EventTimeline.prototype.getNeighbouringTimeline = function(direction) {
 EventTimeline.prototype.setNeighbouringTimeline = function(neighbour, direction) {
    if (this.getNeighbouringTimeline(direction)) {
        throw new Error("timeline already has a neighbouring timeline - " +
-                        "cannot reset neighbour");
+                        "cannot reset neighbour (direction: " + direction + ")");
    }

    if (direction == EventTimeline.BACKWARDS) {
@@ -1,6 +1,6 @@
 /*
 Copyright 2015, 2016 OpenMarket Ltd
-Copyright 2018 New Vector Ltd
+Copyright 2018, 2019 New Vector Ltd

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -38,7 +38,7 @@ import ReEmitter from '../ReEmitter';
 // to upgrade (ie: "stable"). Eventually, we should remove these when all homeservers
 // return an m.room_versions capability.
 const KNOWN_SAFE_ROOM_VERSION = '1';
-const SAFE_ROOM_VERSIONS = ['1', '2'];
+const SAFE_ROOM_VERSIONS = ['1', '2', '3'];

 function synthesizeReceipt(userId, event, receiptType) {
    // console.log("synthesizing receipt for "+event.getId());
@@ -496,7 +496,7 @@ Room.prototype.loadMembersIfNeeded = function() {
    const inMemoryUpdate = this._loadMembers().then((result) => {
        this.currentState.setOutOfBandMembers(result.memberEvents);
        // now the members are loaded, start to track the e2e devices if needed
-        if (this._client.isRoomEncrypted(this.roomId)) {
+        if (this._client.isCryptoEnabled() && this._client.isRoomEncrypted(this.roomId)) {
            this._client._crypto.trackRoomDevices(this.roomId);
        }
        return result.fromServer;
@@ -598,6 +598,11 @@ Room.prototype._fixUpLegacyTimelineFields = function() {

 /**
 * Returns whether there are any devices in the room that are unverified
+ *
+ * Note: Callers should first check if crypto is enabled on this device. If it is
+ * disabled, then we aren't tracking room devices at all, so we can't answer this, and an
+ * error will be thrown.
+ *
 * @return {bool} the result
 */
 Room.prototype.hasUnverifiedDevices = async function() {
@@ -15,9 +15,12 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */

+/* eslint-disable no-invalid-this */
+
 import Promise from 'bluebird';
 import {MemoryStore} from "./memory";
 import utils from "../utils";
+import {EventEmitter} from 'events';
 import LocalIndexedDBStoreBackend from "./indexeddb-local-backend.js";
 import RemoteIndexedDBStoreBackend from "./indexeddb-remote-backend.js";
 import User from "../models/user";
@@ -110,6 +113,7 @@ const IndexedDBStore = function IndexedDBStore(opts) {
    };
 };
 utils.inherits(IndexedDBStore, MemoryStore);
+utils.extend(IndexedDBStore.prototype, EventEmitter.prototype);

 IndexedDBStore.exists = function(indexedDB, dbName) {
    return LocalIndexedDBStoreBackend.exists(indexedDB, dbName);
@@ -146,28 +150,28 @@ IndexedDBStore.prototype.startup = function() {
 * client state to where it was at the last save, or null if there
 * is no saved sync data.
 */
-IndexedDBStore.prototype.getSavedSync = function() {
+IndexedDBStore.prototype.getSavedSync = degradable(function() {
    return this.backend.getSavedSync();
-};
+}, "getSavedSync");

 /** @return {Promise<bool>} whether or not the database was newly created in this session. */
-IndexedDBStore.prototype.isNewlyCreated = function() {
+IndexedDBStore.prototype.isNewlyCreated = degradable(function() {
    return this.backend.isNewlyCreated();
-};
+}, "isNewlyCreated");

 /**
 * @return {Promise} If there is a saved sync, the nextBatch token
 * for this sync, otherwise null.
 */
-IndexedDBStore.prototype.getSavedSyncToken = function() {
+IndexedDBStore.prototype.getSavedSyncToken = degradable(function() {
    return this.backend.getNextBatchToken();
-},
+}, "getSavedSyncToken"),

 /**
 * Delete all data from this store.
 * @return {Promise} Resolves if the data was deleted from the database.
 */
-IndexedDBStore.prototype.deleteAllData = function() {
+IndexedDBStore.prototype.deleteAllData = degradable(function() {
    MemoryStore.prototype.deleteAllData.call(this);
    return this.backend.clearDatabase().then(() => {
        console.log("Deleted indexeddb data.");
@@ -175,7 +179,7 @@ IndexedDBStore.prototype.deleteAllData = function() {
        console.error(`Failed to delete indexeddb data: ${err}`);
        throw err;
    });
-};
+});

 /**
 * Whether this store would like to save its data
@@ -203,7 +207,7 @@ IndexedDBStore.prototype.save = function() {
    return Promise.resolve();
 };

-IndexedDBStore.prototype._reallySave = function() {
+IndexedDBStore.prototype._reallySave = degradable(function() {
    this._syncTs = Date.now(); // set now to guard against multi-writes

    // work out changed users (this doesn't handle deletions but you
@@ -219,14 +223,12 @@ IndexedDBStore.prototype._reallySave = function() {
        this._userModifiedMap[u.userId] = u.getLastModifiedTime();
    }

-    return this.backend.syncToDatabase(userTuples).catch((err) => {
-        console.error("sync fail:", err);
-    });
-};
+    return this.backend.syncToDatabase(userTuples);
+});

-IndexedDBStore.prototype.setSyncData = function(syncData) {
+IndexedDBStore.prototype.setSyncData = degradable(function(syncData) {
    return this.backend.setSyncData(syncData);
-};
+}, "setSyncData");

 /**
 * Returns the out-of-band membership events for this room that
@@ -235,9 +237,9 @@ IndexedDBStore.prototype.setSyncData = function(syncData) {
 * @returns {event[]} the events, potentially an empty array if OOB loading didn't yield any new members
 * @returns {null} in case the members for this room haven't been stored yet
 */
-IndexedDBStore.prototype.getOutOfBandMembers = function(roomId) {
+IndexedDBStore.prototype.getOutOfBandMembers = degradable(function(roomId) {
    return this.backend.getOutOfBandMembers(roomId);
-};
+}, "getOutOfBandMembers");

 /**
 * Stores the out-of-band membership events for this room. Note that
@@ -247,20 +249,71 @@ IndexedDBStore.prototype.getOutOfBandMembers = function(roomId) {
 * @param {event[]} membershipEvents the membership events to store
 * @returns {Promise} when all members have been stored
 */
-IndexedDBStore.prototype.setOutOfBandMembers = function(roomId, membershipEvents) {
+IndexedDBStore.prototype.setOutOfBandMembers = degradable(function(
+    roomId,
+    membershipEvents,
+) {
+    MemoryStore.prototype.setOutOfBandMembers.call(this, roomId, membershipEvents);
    return this.backend.setOutOfBandMembers(roomId, membershipEvents);
-};
+}, "setOutOfBandMembers");

-IndexedDBStore.prototype.clearOutOfBandMembers = function(roomId) {
+IndexedDBStore.prototype.clearOutOfBandMembers = degradable(function(roomId) {
+    MemoryStore.prototype.clearOutOfBandMembers.call(this);
    return this.backend.clearOutOfBandMembers(roomId);
-};
+}, "clearOutOfBandMembers");

-IndexedDBStore.prototype.getClientOptions = function() {
+IndexedDBStore.prototype.getClientOptions = degradable(function() {
    return this.backend.getClientOptions();
-};
+}, "getClientOptions");

-IndexedDBStore.prototype.storeClientOptions = function(options) {
+IndexedDBStore.prototype.storeClientOptions = degradable(function(options) {
+    MemoryStore.prototype.storeClientOptions.call(this, options);
    return this.backend.storeClientOptions(options);
-};
+}, "storeClientOptions");

 module.exports.IndexedDBStore = IndexedDBStore;
+
+/**
+ * All member functions of `IndexedDBStore` that access the backend use this wrapper to
+ * watch for failures after initial store startup, including `QuotaExceededError` as
+ * free disk space changes, etc.
+ *
+ * When IndexedDB fails via any of these paths, we degrade this back to a `MemoryStore`
+ * in place so that the current operation and all future ones are in-memory only.
+ *
+ * @param {Function} func The degradable work to do.
+ * @param {String} fallback The method name for fallback.
+ * @returns {Function} A wrapped member function.
+ */
+function degradable(func, fallback) {
+    return async function(...args) {
+        try {
+            return await func.call(this, ...args);
+        } catch (e) {
+            console.error("IndexedDBStore failure, degrading to MemoryStore", e);
+            this.emit("degraded", e);
+            try {
+                // We try to delete IndexedDB after degrading since this store is only a
+                // cache (the app will still function correctly without the data).
+                // It's possible that deleting repair IndexedDB for the next app load,
+                // potenially by making a little more space available.
+                console.log("IndexedDBStore trying to delete degraded data");
+                await this.backend.clearDatabase();
+                console.log("IndexedDBStore delete after degrading succeeeded");
+            } catch (e) {
+                console.warn("IndexedDBStore delete after degrading failed", e);
+            }
+            // Degrade the store from being an instance of `IndexedDBStore` to instead be
+            // an instance of `MemoryStore` so that future API calls use the memory path
+            // directly and skip IndexedDB entirely. This should be safe as
+            // `IndexedDBStore` already extends from `MemoryStore`, so we are making the
+            // store become its parent type in a way. The mutator methods of
+            // `IndexedDBStore` also maintain the state that `MemoryStore` uses (many are
+            // not overridden at all).
+            Object.setPrototypeOf(this, MemoryStore.prototype);
+            if (fallback) {
+                return await MemoryStore.prototype[fallback].call(this, ...args);
+            }
+        }
+    };
+}
@@ -385,6 +385,7 @@ module.exports.MemoryStore.prototype = {
        };
        return Promise.resolve();
    },
+
    /**
     * Returns the out-of-band membership events for this room that
     * were previously loaded.
@@ -395,6 +396,7 @@ module.exports.MemoryStore.prototype = {
    getOutOfBandMembers: function(roomId) {
        return Promise.resolve(this._oobMembers[roomId] || null);
    },
+
    /**
     * Stores the out-of-band membership events for this room. Note that
     * it still makes sense to store an empty array as the OOB status for the room is
@@ -408,6 +410,11 @@ module.exports.MemoryStore.prototype = {
        return Promise.resolve();
    },

+    clearOutOfBandMembers: function() {
+        this._oobMembers = {};
+        return Promise.resolve();
+    },
+
    getClientOptions: function() {
        return Promise.resolve(this._clientOptions);
    },
@@ -445,6 +445,16 @@ SyncApi.prototype._wasLazyLoadingToggled = async function(lazyLoadMembers) {
    return false;
 };

+SyncApi.prototype._shouldAbortSync = function(error) {
+    if (error.errcode === "M_UNKNOWN_TOKEN") {
+        // The logout already happened, we just need to stop.
+        console.warn("Token no longer valid - assuming logout");
+        this.stop();
+        return true;
+    }
+    return false;
+};
+
 /**
 * Main entry point
 */
@@ -472,13 +482,17 @@ SyncApi.prototype.sync = function() {

    async function getPushRules() {
        try {
+            debuglog("Getting push rules...");
            const result = await client.getPushRules();
            debuglog("Got push rules");

            client.pushRules = result;
        } catch (err) {
+            console.error("Getting push rules failed", err);
+            if (self._shouldAbortSync(err)) return;
            // wait for saved sync to complete before doing anything else,
            // otherwise the sync state will end up being incorrect
+            debuglog("Waiting for saved sync before retrying push rules...");
            await self.recoverFromSyncStartupError(savedSyncPromise, err);
            getPushRules();
            return;
@@ -487,22 +501,35 @@ SyncApi.prototype.sync = function() {
    }

    const checkLazyLoadStatus = async () => {
+        debuglog("Checking lazy load status...");
        if (this.opts.lazyLoadMembers && client.isGuest()) {
            this.opts.lazyLoadMembers = false;
        }
        if (this.opts.lazyLoadMembers) {
+            debuglog("Checking server lazy load support...");
            const supported = await client.doesServerSupportLazyLoading();
            if (supported) {
-                this.opts.filter = await client.createFilter(
-                    Filter.LAZY_LOADING_SYNC_FILTER,
-                );
+                try {
+                    debuglog("Creating and storing lazy load sync filter...");
+                    this.opts.filter = await client.createFilter(
+                        Filter.LAZY_LOADING_SYNC_FILTER,
+                    );
+                    debuglog("Created and stored lazy load sync filter");
+                } catch (err) {
+                    console.error(
+                        "Creating and storing lazy load sync filter failed",
+                        err,
+                    );
+                    throw err;
+                }
            } else {
-                console.log("LL: lazy loading requested but not supported " +
+                debuglog("LL: lazy loading requested but not supported " +
                    "by server, so disabling");
                this.opts.lazyLoadMembers = false;
            }
        }
        // need to vape the store when enabling LL and wasn't enabled before
+        debuglog("Checking whether lazy loading has changed in store...");
        const shouldClear = await this._wasLazyLoadingToggled(this.opts.lazyLoadMembers);
        if (shouldClear) {
            this._storeIsInvalid = true;
@@ -519,12 +546,20 @@ SyncApi.prototype.sync = function() {
        if (this.opts.lazyLoadMembers && this.opts.crypto) {
            this.opts.crypto.enableLazyLoading();
        }
-        await this.client._storeClientOptions();
+        try {
+            debuglog("Storing client options...");
+            await this.client._storeClientOptions();
+            debuglog("Stored client options");
+        } catch (err) {
+            console.error("Storing client options failed", err);
+            throw err;
+        }

        getFilter(); // Now get the filter and start syncing
    };

    async function getFilter() {
+        debuglog("Getting filter...");
        let filter;
        if (self.opts.filter) {
            filter = self.opts.filter;
@@ -539,8 +574,11 @@ SyncApi.prototype.sync = function() {
                getFilterName(client.credentials.userId), filter,
            );
        } catch (err) {
+            console.error("Getting filter failed", err);
+            if (self._shouldAbortSync(err)) return;
            // wait for saved sync to complete before doing anything else,
            // otherwise the sync state will end up being incorrect
+            debuglog("Waiting for saved sync before retrying filter...");
            await self.recoverFromSyncStartupError(savedSyncPromise, err);
            getFilter();
            return;
@@ -554,11 +592,12 @@ SyncApi.prototype.sync = function() {
        if (self._currentSyncRequest === null) {
            // Send this first sync request here so we can then wait for the saved
            // sync data to finish processing before we process the results of this one.
-            console.log("Sending first sync request...");
+            debuglog("Sending first sync request...");
            self._currentSyncRequest = self._doSyncRequest({ filterId }, savedSyncToken);
        }

        // Now wait for the saved sync to finish...
+        debuglog("Waiting for saved sync before starting sync processing...");
        await savedSyncPromise;
        self._sync({ filterId });
    }
@@ -570,13 +609,19 @@ SyncApi.prototype.sync = function() {
        // Pull the saved sync token out first, before the worker starts sending
        // all the sync data which could take a while. This will let us send our
        // first incremental sync request before we've processed our saved data.
+        debuglog("Getting saved sync token...");
        savedSyncPromise = client.store.getSavedSyncToken().then((tok) => {
+            debuglog("Got saved sync token");
            savedSyncToken = tok;
+            debuglog("Getting saved sync...");
            return client.store.getSavedSync();
        }).then((savedSync) => {
+            debuglog(`Got reply from saved sync, exists? ${!!savedSync}`);
            if (savedSync) {
                return self._syncFromCache(savedSync);
            }
+        }).catch(err => {
+            console.error("Getting saved sync failed", err);
        });
        // Now start the first incremental sync request: this can also
        // take a while so if we set it going now, we can wait for it
@@ -666,6 +711,7 @@ SyncApi.prototype._syncFromCache = async function(savedSync) {
 * @param {boolean} syncOptions.hasSyncedBefore
 */
 SyncApi.prototype._sync = async function(syncOptions) {
+    debuglog("Starting sync request processing...");
    const client = this.client;

    if (!this._running) {
@@ -704,7 +750,9 @@ SyncApi.prototype._sync = async function(syncOptions) {
    // Reset after a successful sync
    this._failedSyncCount = 0;

+    debuglog("Storing sync data...");
    await client.store.setSyncData(data);
+    debuglog("Sync data stored");

    const syncEventData = {
        oldSyncToken: syncToken,
@@ -719,6 +767,7 @@ SyncApi.prototype._sync = async function(syncOptions) {
    }

    try {
+        debuglog("Processing sync response...");
        await this._processSyncResponse(syncEventData, data);
    } catch(e) {
        // log the exception with stack if we have it, else fall back
@@ -840,6 +889,10 @@ SyncApi.prototype._onSyncError = function(err, syncOptions) {
    console.error("/sync error %s", err);
    console.error(err);

+    if(this._shouldAbortSync(err)) {
+        return;
+    }
+
    this._failedSyncCount++;
    console.log('Number of consecutive failed sync requests:', this._failedSyncCount);

@@ -1051,7 +1104,6 @@ SyncApi.prototype._processSyncResponse = async function(
        const stateEvents =
            self._mapSyncEventsFormat(inviteObj.invite_state, room);

-        room.updateMyMembership("invite");
        self._processRoomEvents(room, stateEvents);
        if (inviteObj.isBrandNewRoom) {
            room.recalculate();
@@ -1061,6 +1113,7 @@ SyncApi.prototype._processSyncResponse = async function(
        stateEvents.forEach(function(e) {
            client.emit("event", e);
        });
+        room.updateMyMembership("invite");
    });

    // Handle joins
@@ -1076,12 +1129,19 @@ SyncApi.prototype._processSyncResponse = async function(
            room.setUnreadNotificationCount(
                'total', joinObj.unread_notifications.notification_count,
            );
-            room.setUnreadNotificationCount(
-                'highlight', joinObj.unread_notifications.highlight_count,
-            );
-        }

-        room.updateMyMembership("join");
+            // We track unread notifications ourselves in encrypted rooms, so don't
+            // bother setting it here. We trust our calculations better than the
+            // server's for this case, and therefore will assume that our non-zero
+            // count is accurate.
+            const encrypted = client.isRoomEncrypted(room.roomId);
+            if (!encrypted
+                || (encrypted && room.getUnreadNotificationCount('highlight') <= 0)) {
+                room.setUnreadNotificationCount(
+                    'highlight', joinObj.unread_notifications.highlight_count,
+                );
+            }
+        }

        joinObj.timeline = joinObj.timeline || {};

@@ -1195,6 +1255,8 @@ SyncApi.prototype._processSyncResponse = async function(
        accountDataEvents.forEach(function(e) {
            client.emit("event", e);
        });
+
+        room.updateMyMembership("join");
    });

    // Handle leaves (e.g. kicked rooms)
@@ -1207,8 +1269,6 @@ SyncApi.prototype._processSyncResponse = async function(
        const accountDataEvents =
            self._mapSyncEventsFormat(leaveObj.account_data);

-        room.updateMyMembership("leave");
-
        self._processRoomEvents(room, stateEvents, timelineEvents);
        room.addAccountData(accountDataEvents);

@@ -1229,6 +1289,8 @@ SyncApi.prototype._processSyncResponse = async function(
        accountDataEvents.forEach(function(e) {
            client.emit("event", e);
        });
+
+        room.updateMyMembership("leave");
    });

    // update the notification timeline, if appropriate.
@@ -3520,9 +3520,9 @@ object.pick@^1.3.0:
  dependencies:
    isobject "^3.0.1"

-"olm@https://matrix.org/packages/npm/olm/olm-3.1.0-pre1.tgz":
-  version "3.1.0-pre1"
-  resolved "https://matrix.org/packages/npm/olm/olm-3.1.0-pre1.tgz#54f14fa901ff5c81db516b3b2adb294b91726eaa"
+"olm@https://packages.matrix.org/npm/olm/olm-3.1.0.tgz":
+  version "3.1.0"
+  resolved "https://packages.matrix.org/npm/olm/olm-3.1.0.tgz#2c8fc2a42b7ea12febc31baa73ec03d9b601da16"

 once@1.x, once@^1.3.0:
  version "1.4.0"
Author	SHA1	Message	Date
David Baker	c1956d3f05	v1.1.0	2019-05-07 15:23:32 +01:00
David Baker	56316dc5d9	Prepare changelog for v1.1.0	2019-05-07 15:23:32 +01:00
David Baker	0bb9c56e94	Merge branch 'develop' into release-v1.1.0	2019-04-30 11:50:24 +01:00
David Baker	821f1c876b	Undo unintentional commenting	2019-04-30 11:48:38 +01:00
David Baker	e9b95f8567	v1.1.0-rc.1	2019-04-30 11:43:35 +01:00
David Baker	103d811441	Prepare changelog for v1.1.0-rc.1	2019-04-30 11:43:34 +01:00
David Baker	bb4f5a3fa1	Get the name of the pipeline right	2019-04-26 18:07:04 +01:00
David Baker	f5cbdeac8f	Trigger react-sdk build in buildkite pipeline	2019-04-26 18:06:04 +01:00
Hubert Chathi	56062e8e4e	Merge pull request #903 from uhoreg/olm-3.1.0 use the release version of olm 3.1.0	2019-04-24 18:19:15 -04:00
Hubert Chathi	03c85d48e5	update the yarn.lock with the olm release	2019-04-24 12:07:10 -04:00
Hubert Chathi	dd8f0fbdcb	use the release version of olm 3.1.0	2019-04-23 18:06:09 -04:00
Travis Ralston	fac61a76e9	Merge pull request #901 from matrix-org/travis/olm2 Use new Olm repo link in README	2019-04-18 08:26:12 -06:00
Travis Ralston	3b09ab3ca1	Use new Olm repo link in README	2019-04-17 23:50:51 -06:00
Travis Ralston	16bfe79305	Merge pull request #897 from matrix-org/travis/wk-custom Support being fed a .well-known config object for validation	2019-04-17 09:58:22 -06:00
J. Ryan Stinnett	d668f97c98	Clarify comment Co-Authored-By: turt2live <travpc@gmail.com>	2019-04-17 09:44:52 -06:00
Bruno Windels	18bd10b03d	Merge pull request #900 from matrix-org/bwindels/stylepreviewbar emit self-membership event at end of handling sync update	2019-04-17 12:31:16 +00:00
Bruno Windels	6b1d089caf	Merge branch 'develop' into bwindels/stylepreviewbar	2019-04-17 14:24:25 +02:00
Travis Ralston	af93401385	Use more appropriate errors for some situations	2019-04-16 11:13:36 -06:00
Travis Ralston	fa5add3d99	Use the right error codes	2019-04-16 11:03:58 -06:00
Travis Ralston	fb971580e0	Merge branch 'develop' into travis/wk-custom	2019-04-16 11:01:49 -06:00
Bruno Windels	dcaea98e33	emit self-membership event at end of handling sync update otherwise the room state isn't updated yet (and we can't for example distinguish a leave from a kick) this is only used for updating the UI, seems safe to emit this event at a later point	2019-04-16 17:16:24 +02:00
Travis Ralston	b95079d1c5	Merge pull request #898 from matrix-org/travis/packages.matrix.org Use packages.matrix.org for Olm	2019-04-16 09:14:21 -06:00
J. Ryan Stinnett	e59f36cdc0	Merge pull request #899 from matrix-org/travis/fix-tests Fix tests on develop	2019-04-16 09:36:56 +01:00
Travis Ralston	0f2f041d8b	Use constants for autodiscovery errors To ease usage	2019-04-15 22:04:24 -06:00
Travis Ralston	33d2837ec3	Use packages.matrix.org for Olm See https://github.com/vector-im/riot-web/issues/9497	2019-04-15 21:23:18 -06:00
Travis Ralston	7cede221de	Support being fed a .well-known config object for validation Used by Riot to consume the user's provided config. This also includes a change to carry over custom keys on m.homeserver and m.identity_server which aren't intentionally controlled.	2019-04-15 21:22:49 -06:00
Travis Ralston	fe47435fc7	Fix tests for autodiscovery	2019-04-15 21:12:17 -06:00
Travis Ralston	491226a916	Use the right this in _shouldAbortSync	2019-04-15 21:04:29 -06:00
Travis Ralston	deb7433453	Use toMatch for presence events We don't pass the reference through, so the test fails with toEqual	2019-04-15 20:17:42 -06:00
Travis Ralston	14973a35c2	Use the right error object	2019-04-15 19:46:30 -06:00
Travis Ralston	b5779f8654	Merge pull request #895 from matrix-org/travis/stop-client-on-logout Stop syncing when the token is invalid	2019-04-15 11:30:24 -06:00
Travis Ralston	f7d1984257	De-duplicate usage of shouldAbortSync	2019-04-15 11:29:55 -06:00
Bruno Windels	3fba683090	Merge pull request #887 from jkasun/fix_redact_put change event redact, POST request to PUT request	2019-04-15 15:13:59 +00:00
jkasun	430da8ac09	JS Doc Fix	2019-04-15 20:34:39 +05:30
Travis Ralston	dcd9b5c382	Stop syncing when the token is invalid Fixes https://github.com/vector-im/riot-web/issues/9451	2019-04-14 21:38:51 -06:00
jkasun	348c293962	Argument Length Check, Duplicate Fix for Redact Funcation	2019-04-13 22:23:57 +05:30
Travis Ralston	34309da10c	Merge pull request #894 from matrix-org/travis/guests/better-errors Expose better autodiscovery error messages	2019-04-12 10:21:05 -06:00
Travis Ralston	6db973f430	Expose better autodiscovery error messages Fixes https://github.com/vector-im/riot-web/issues/7925	2019-04-11 15:50:43 -06:00
J. Ryan Stinnett	9f27bafa62	Merge pull request #892 from jryans/degraded-storage Explicitly guard store usage during sync startup	2019-04-09 17:12:07 +01:00
Travis Ralston	b05136146a	Merge pull request #893 from matrix-org/travis/v3-safe Flag v3 rooms as safe	2019-04-09 10:00:56 -06:00
Travis Ralston	13d3be637b	Copyright 2019 NV	2019-04-09 09:58:33 -06:00
J. Ryan Stinnett	44de7fad6f	Preserve previous error flow	2019-04-09 16:44:36 +01:00
J. Ryan Stinnett	b99243406b	Move logs inside try block	2019-04-09 16:40:13 +01:00
Travis Ralston	0e9ec811b0	Merge pull request #890 from matrix-org/travis/cached-capabilities Cache failed capabilities lookups for shorter amounts of time	2019-04-09 09:29:42 -06:00
Travis Ralston	6979177fb2	Log errors for capabilities requests	2019-04-09 09:27:38 -06:00
Travis Ralston	3aa8bfa6ca	Flag v3 rooms as safe The spec says they are, so we might as well too.	2019-04-09 09:25:07 -06:00
Travis Ralston	17b356b08e	Merge pull request #891 from matrix-org/travis/fix-notifs Fix highlight notifications for unencrypted rooms	2019-04-09 09:19:58 -06:00
Travis Ralston	f72ae490a8	Appease the linter	2019-04-09 09:05:20 -06:00
J. Ryan Stinnett	b0c3d0d2e3	Explicitly guard store usage during sync startup This adds explicit `try` blocks in the spots where we interact with the store during sync startup. This shouldn't be necessary as the store should already be catching this and degrading as of https://github.com/matrix-org/matrix-js-sdk/pull/884, but that doesn't seem to have been enough for the affected user in https://github.com/vector-im/riot-web/issues/7769, as they are seeing sync just stop when storing without any further detail.	2019-04-09 15:06:54 +01:00
Travis Ralston	9dc344999e	Fix highlight notifications for unencrypted rooms A logic error introduced by https://github.com/matrix-org/matrix-js-sdk/pull/886 meant that all unencrypted rooms were not getting highlight notifications.	2019-04-08 15:57:44 -06:00
Travis Ralston	663c096400	Cache failed capabilities lookups for shorter amounts of time This should fix https://github.com/vector-im/riot-web/issues/9225 for showing up too often/too long.	2019-04-08 12:24:00 -06:00
J. Ryan Stinnett	420b4d119d	Merge pull request #889 from jryans/guard-missing-crypto Document checking crypto state before using `hasUnverifiedDevices`	2019-04-08 16:56:13 +01:00
J. Ryan Stinnett	58b752c63b	Document checking crypto state before using `hasUnverifiedDevices` It's unclear what `hasUnverifiedDevices` should do when crypto is disabled on the current device. Let's at least document that callers should first check crypto status.	2019-04-08 16:24:25 +01:00
J. Ryan Stinnett	4740232fa4	Merge pull request #888 from jryans/degraded-storage Add logging to sync startup path	2019-04-08 16:20:29 +01:00
J. Ryan Stinnett	0cc9994b8b	Add logging to sync startup path In https://github.com/vector-im/riot-web/issues/7769, we're seeing sync startup fail to complete, but the actual error isn't being logged. Hopefully these extra debug logs will provide more insight into the failing step.	2019-04-08 15:55:52 +01:00
Bruno Windels	1e78628b23	Merge branch 'master' into develop	2019-04-08 16:04:45 +02:00
Bruno Windels	00f5ddc93c	v1.0.4	2019-04-08 16:03:26 +02:00
Bruno Windels	f79f2105fd	Prepare changelog for v1.0.4	2019-04-08 16:03:25 +02:00
Travis Ralston	828c51467f	Refuse to set forwards pagination token on live timeline Should fix the error seen in https://github.com/matrix-org/riot-web-rageshakes/issues/1389 (https://github.com/vector-im/riot-web/issues/8593)	2019-04-08 15:48:03 +02:00
Travis Ralston	963e271bce	Refuse to link live timelines into the forwards/backwards position See https://github.com/vector-im/riot-web/issues/8593#issuecomment-478681816 Previously (https://github.com/matrix-org/matrix-js-sdk/pull/873) we allowed half-linking timelines to each other if they satisfy the conditions, however this appears to not be helping. Instead, it seems like the timelines are getting stuck in a position where one direction is spliced but the other is broken. To avoid this case, we'll just avoid splicing in both directions when one of the directions is invalid.	2019-04-08 15:47:40 +02:00
Travis Ralston	0ab41215f0	Add a tiny bit of logging to work out what timelines are doing See https://github.com/vector-im/riot-web/issues/8593	2019-04-08 15:47:21 +02:00
Travis Ralston	d153c4da07	log the timeline that broke	2019-04-08 15:47:03 +02:00
Travis Ralston	91aa783c3d	Use better words for warnings	2019-04-08 15:46:56 +02:00
Travis Ralston	a54845bf76	Appease the linter	2019-04-08 15:46:48 +02:00
Travis Ralston	8a56a5f1ed	Refuse splicing the live timeline into a broken position Credit to Matthew for basically solving this. Theoretically fixes spontaneous timeline corruption: https://github.com/vector-im/riot-web/issues/8593 When the live timeline ends up in a position where it can no longer be live (such as becoming the second timeline in the set, rather than the first) we end up getting neighbouring timeline errors. By refusing to splice the live timeline into such a position, we hopefully keep the live timeline in a position of still being live for when it is next used. The running theory that leads to this fix is multiple limited syncs coming in, causing holes in the timeline. When trying to patch up the holes, the timeline set would end up splicing all over the place, leading to potentially splicing the live timeline into a broken position.	2019-04-08 15:46:41 +02:00
Travis Ralston	f585c80491	Merge pull request #886 from matrix-org/travis/e2e-notifs-2 Track e2e highlights better, particularly in 'Mentions Only' rooms	2019-04-08 07:40:31 -06:00
jkasun	c495b12cef	change event redact, POST request to PUT request	2019-04-07 00:19:02 +05:30
Travis Ralston	1d6f7f862f	Track e2e highlights better, particularly in 'Mentions Only' rooms Fixes https://github.com/vector-im/riot-web/issues/9280 The server is unable to calculate encrypted highlights for us, so we calculate them. This also means the server always sends a zero for highlight_count, and therefore in sync.js we now trust our judgement over the server's. In future, this check will need to be altered to support server-side encrypted notifications if that happens. This fixes the part of 9280 where the badge count ends up disappearing unless the message received also happens to be a mention. The changes in client.js are more to support rooms which are mentions only. Because the server doesn't send an unread_count for these rooms, the total notifications will always be zero. Therefore, we try and calculate that. In order to do that, we need to assume that our highlight count is also wrong and calculate it appropriately.	2019-04-05 14:49:38 -06:00
Hubert Chathi	0a82c84006	Merge pull request #882 from uhoreg/fix_mac support both the incorrect and correct MAC methods	2019-04-05 16:45:45 -04:00
Travis Ralston	a614a02b90	Merge pull request #885 from matrix-org/travis/refuse-forwards-token Refuse to set forwards pagination token on live timeline	2019-04-05 11:54:33 -06:00
Travis Ralston	0945e2c5c6	Refuse to set forwards pagination token on live timeline Should fix the error seen in https://github.com/matrix-org/riot-web-rageshakes/issues/1389 (https://github.com/vector-im/riot-web/issues/8593)	2019-04-05 11:36:01 -06:00
J. Ryan Stinnett	b1b49413d0	Merge pull request #884 from jryans/degraded-storage Degrade `IndexedDBStore` back to memory only on failure	2019-04-05 11:32:42 +01:00
Hubert Chathi	01af303d63	fix the selection of the verification methods, and test more things	2019-04-04 14:08:30 -04:00
Hubert Chathi	751060305c	update the name of the MAC method	2019-04-04 14:07:16 -04:00
J. Ryan Stinnett	389fcfaf3d	Ensure IDB store maintains all memory state A few of the IDB store methods weren't updating memory store state, so let's improve those so we can reliably fall back to it from IDB at any time.	2019-04-04 17:39:50 +01:00
J. Ryan Stinnett	3eb0c534a5	Clarify why it's safe to change store types on demand	2019-04-04 15:54:26 +01:00
J. Ryan Stinnett	8a2f84b678	Emit event when `IndexedDBStore` degrades This allows for optional tracking of when the store degrades to see how often it happens in the field.	2019-04-04 12:06:41 +01:00
J. Ryan Stinnett	dd00735409	Degrade `IndexedDBStore` back to memory only on failure IndexedDB may fail at any moment with `QuoteExceededError` when space is low or other random issues we can't control. Since `IndexedDBStore` is just a cache for improving performance, we can give up on it if it fails. This causes `IndexedDBStore` to degrade in place back to using memory only. This allow (for example) login to complete even if IndexedDB is exploding. Hopefully improves https://github.com/vector-im/riot-web/issues/7769	2019-04-04 12:06:41 +01:00
J. Ryan Stinnett	6ba7e85e24	Ensure we have crypto before accessing it in `Room` model If crypto startup has failed, we shouldn't try to access any of its methods. This fixes a variant of this in the `Room` model.	2019-04-04 12:06:41 +01:00
Travis Ralston	941d93c2f4	Merge pull request #877 from matrix-org/travis/tlexpl-full-abort Refuse to link live timelines into the forwards/backwards position when either is invalid	2019-04-03 09:59:09 -06:00
David Baker	c73e9cbc7c	Merge pull request #883 from matrix-org/dbkr/logging_for_9357 Key backup logging improvements	2019-04-03 11:45:10 +01:00
David Baker	ae89e4bf21	Key backup logging improvements To try & diagnose https://github.com/vector-im/riot-web/issues/9357	2019-04-03 11:00:31 +01:00
Hubert Chathi	d1e64d0cfb	support both the incorrect and correct MAC methods also do some refactoring to make it easier to support choices in the other methods in the future	2019-04-02 23:36:49 -04:00
David Baker	8f9f9590d9	Merge pull request #880 from matrix-org/dbkr/more_logging_for_7769 Don't assume aborts are always from txn.abort()	2019-04-02 18:33:32 +01:00
David Baker	ed68093310	lint	2019-04-02 18:30:49 +01:00
David Baker	23655e748d	Don't assume aborts are always from txn.abort() According to https://developer.mozilla.org/en-US/docs/Web/API/IDBTransaction/abort_event a txn can abort for any number of reasons, not just because of an abort call, so we can't assume our abort error is set. This should give us more info on https://github.com/vector-im/riot-web/issues/7769	2019-04-02 18:25:23 +01:00
David Baker	98624871bd	Merge pull request #878 from matrix-org/dbkr/logging_for_7769 Add a bunch of logging	2019-04-02 13:34:26 +01:00
Will Hunt	7b154c0834	Revert "Remove "Event sent to" line from client.js" This reverts commit `d9f056242f`.	2019-04-02 13:24:58 +01:00
Will Hunt	d9f056242f	Remove "Event sent to" line from client.js Emitting a log line here means that bridges emit this for each event sent in stdout, which bloats forever.log	2019-04-02 13:22:27 +01:00
David Baker	a4268d288e	Add a bunch of logging to try & diagnose https://github.com/vector-im/riot-web/issues/7769	2019-04-02 12:30:24 +01:00
Travis Ralston	f90c91dded	Refuse to link live timelines into the forwards/backwards position See https://github.com/vector-im/riot-web/issues/8593#issuecomment-478681816 Previously (https://github.com/matrix-org/matrix-js-sdk/pull/873) we allowed half-linking timelines to each other if they satisfy the conditions, however this appears to not be helping. Instead, it seems like the timelines are getting stuck in a position where one direction is spliced but the other is broken. To avoid this case, we'll just avoid splicing in both directions when one of the directions is invalid.	2019-04-01 18:43:48 -06:00
Travis Ralston	5c8890c3c1	Add a tiny bit of logging to work out what timelines are doing See https://github.com/vector-im/riot-web/issues/8593	2019-04-01 11:11:35 -06:00
David Baker	cd3c6809a9	Merge branch 'master' into develop	2019-04-01 13:33:51 +01:00
David Baker	e2c17528c2	v1.0.3	2019-04-01 13:31:45 +01:00
David Baker	e00f565f37	Prepare changelog for v1.0.3	2019-04-01 13:31:44 +01:00
J. Ryan Stinnett	085e797c30	Merge pull request #874 from matrix-org/travis/1.0.6-fire/872 Add existence check to local storage based crypto store	2019-04-01 09:26:39 +01:00
J. Ryan Stinnett	d753db590b	Add existence check to local storage based crypto store This supports additional diagnostics of stores in https://github.com/vector-im/riot-web/issues/9309.	2019-03-29 16:19:54 -06:00
Travis Ralston	eab074a27b	Merge pull request #873 from matrix-org/travis/refuse-splicing Refuse splicing the live timeline into a broken position	2019-03-28 19:20:58 -06:00
Travis Ralston	e2a3e3816f	log the timeline that broke	2019-03-28 19:14:17 -06:00
Travis Ralston	01dd57adab	Use better words for warnings	2019-03-28 18:59:40 -06:00
Travis Ralston	08e674b695	Appease the linter	2019-03-28 18:34:57 -06:00
Travis Ralston	9f70970e61	Refuse splicing the live timeline into a broken position Credit to Matthew for basically solving this. Theoretically fixes spontaneous timeline corruption: https://github.com/vector-im/riot-web/issues/8593 When the live timeline ends up in a position where it can no longer be live (such as becoming the second timeline in the set, rather than the first) we end up getting neighbouring timeline errors. By refusing to splice the live timeline into such a position, we hopefully keep the live timeline in a position of still being live for when it is next used. The running theory that leads to this fix is multiple limited syncs coming in, causing holes in the timeline. When trying to patch up the holes, the timeline set would end up splicing all over the place, leading to potentially splicing the live timeline into a broken position.	2019-03-28 18:30:41 -06:00
J. Ryan Stinnett	e9ffd5a125	Merge pull request #872 from jryans/storage-existence-check-ls-crypto Add existence check to local storage based crypto store	2019-03-28 17:49:08 +00:00
J. Ryan Stinnett	20f4469361	Add existence check to local storage based crypto store This supports additional diagnostics of stores in https://github.com/vector-im/riot-web/issues/9309.	2019-03-28 12:20:51 +00:00