v0.11.0

Only emit CATCHUP if recovering from conn error

CHANGELOG.md

@@ -1,3 +1,142 @@
+Changes in [0.11.0](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.11.0) (2018-09-10)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.11.0-rc.1...v0.11.0)
+
+BREAKING CHANGES
+----------------
+ * v0.11.0-rc.1 introduced some breaking changes - see the respective release notes.
+
+No changes since rc.1
+
+Changes in [0.11.0-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.11.0-rc.1) (2018-09-07)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.9...v0.11.0-rc.1)
+
+ * Support for lazy loading members. This should improve performance for
+   users who joined big rooms a lot. Pass to `lazyLoadMembers = true` option when calling `startClient`.
+
+BREAKING CHANGES
+----------------
+
+ * `MatrixClient::startClient` now returns a Promise. No method should be called on the client before that promise resolves. Before this method didn't return anything.
+ * A new `CATCHUP` sync state, emitted by `MatrixClient#"sync"` and returned by `MatrixClient::getSyncState()`, when doing initial sync after the `ERROR` state. See `MatrixClient` documentation for details.
+ * `RoomState::maySendEvent('m.room.message', userId)` & `RoomState::maySendMessage(userId)` do not check the membership of the user anymore, only the power level. To check if the syncing user is allowed to write in a room, use `Room::maySendMessage()` as `RoomState` is not always aware of the syncing user's membership anymore, in case lazy loading of members is enabled.
+
+All Changes
+-----------
+
+ * Only emit CATCHUP if recovering from conn error
+   [\#727](https://github.com/matrix-org/matrix-js-sdk/pull/727)
+ * Fix docstring for sync data.error
+   [\#725](https://github.com/matrix-org/matrix-js-sdk/pull/725)
+ * Re-apply "Don't rely on members to query if syncing user can post to room"
+   [\#723](https://github.com/matrix-org/matrix-js-sdk/pull/723)
+ * Revert "Don't rely on members to query if syncing user can post to room"
+   [\#721](https://github.com/matrix-org/matrix-js-sdk/pull/721)
+ * Don't rely on members to query if syncing user can post to room
+   [\#717](https://github.com/matrix-org/matrix-js-sdk/pull/717)
+ * Fixes for room.guessDMUserId
+   [\#719](https://github.com/matrix-org/matrix-js-sdk/pull/719)
+ * Fix filepanel also filtering main timeline with LL turned on.
+   [\#716](https://github.com/matrix-org/matrix-js-sdk/pull/716)
+ * Remove lazy loaded members when leaving room
+   [\#711](https://github.com/matrix-org/matrix-js-sdk/pull/711)
+ * Fix: show spinner again while recovering from connection error
+   [\#702](https://github.com/matrix-org/matrix-js-sdk/pull/702)
+ * Add method to query LL state in client
+   [\#714](https://github.com/matrix-org/matrix-js-sdk/pull/714)
+ * Fix: also load invited members when lazy loading members
+   [\#707](https://github.com/matrix-org/matrix-js-sdk/pull/707)
+ * Pass through function to discard megolm session
+   [\#704](https://github.com/matrix-org/matrix-js-sdk/pull/704)
+
+Changes in [0.10.9](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.9) (2018-09-03)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.9-rc.2...v0.10.9)
+
+ * No changes since rc.2
+
+Changes in [0.10.9-rc.2](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.9-rc.2) (2018-08-31)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.9-rc.1...v0.10.9-rc.2)
+
+ * Fix for "otherMember.getAvatarUrl is not a function"
+   [\#708](https://github.com/matrix-org/matrix-js-sdk/pull/708)
+
+Changes in [0.10.9-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.9-rc.1) (2018-08-30)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.8...v0.10.9-rc.1)
+
+ * Fix DM avatar
+   [\#706](https://github.com/matrix-org/matrix-js-sdk/pull/706)
+ * Lazy loading: avoid loading members at initial sync for e2e rooms
+   [\#699](https://github.com/matrix-org/matrix-js-sdk/pull/699)
+ * Improve setRoomEncryption guard against multiple m.room.encryption st…
+   [\#700](https://github.com/matrix-org/matrix-js-sdk/pull/700)
+ * Revert "Lazy loading: don't block on setting up room crypto"
+   [\#698](https://github.com/matrix-org/matrix-js-sdk/pull/698)
+ * Lazy loading: don't block on setting up room crypto
+   [\#696](https://github.com/matrix-org/matrix-js-sdk/pull/696)
+ * Add getVisibleRooms()
+   [\#695](https://github.com/matrix-org/matrix-js-sdk/pull/695)
+ * Add wrapper around getJoinedMemberCount()
+   [\#697](https://github.com/matrix-org/matrix-js-sdk/pull/697)
+ * Api to fetch events via /room/.../event/..
+   [\#694](https://github.com/matrix-org/matrix-js-sdk/pull/694)
+ * Support for room upgrades
+   [\#693](https://github.com/matrix-org/matrix-js-sdk/pull/693)
+ * Lazy loading of room members
+   [\#691](https://github.com/matrix-org/matrix-js-sdk/pull/691)
+
+Changes in [0.10.8](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.8) (2018-08-20)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.8-rc.1...v0.10.8)
+
+ * No changes since rc.1
+
+Changes in [0.10.8-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.8-rc.1) (2018-08-16)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.7...v0.10.8-rc.1)
+
+ * Add getVersion to Room
+   [\#689](https://github.com/matrix-org/matrix-js-sdk/pull/689)
+ * Add getSyncStateData()
+   [\#680](https://github.com/matrix-org/matrix-js-sdk/pull/680)
+ * Send sync error to listener
+   [\#679](https://github.com/matrix-org/matrix-js-sdk/pull/679)
+ * make sure room.tags is always a valid object to avoid crashes
+   [\#675](https://github.com/matrix-org/matrix-js-sdk/pull/675)
+ * Fix infinite spinner upon joining a room
+   [\#673](https://github.com/matrix-org/matrix-js-sdk/pull/673)
+
+Changes in [0.10.7](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.7) (2018-07-30)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.7-rc.1...v0.10.7)
+
+ * No changes since rc.1
+
+Changes in [0.10.7-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.7-rc.1) (2018-07-24)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.6...v0.10.7-rc.1)
+
+ * encrypt for invited users if history visibility allows.
+   [\#666](https://github.com/matrix-org/matrix-js-sdk/pull/666)
+
+Changes in [0.10.6](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.6) (2018-07-09)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.6-rc.1...v0.10.6)
+
+ * No changes since rc.1
+
+Changes in [0.10.6-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.6-rc.1) (2018-07-06)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.5...v0.10.6-rc.1)
+
+ * Expose event decryption error via Event.decrypted event
+   [\#665](https://github.com/matrix-org/matrix-js-sdk/pull/665)
+ * Add decryption error codes to base.DecryptionError
+   [\#663](https://github.com/matrix-org/matrix-js-sdk/pull/663)
+
 Changes in [0.10.5](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.5) (2018-06-29)
 ==================================================================================================
 [Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.5-rc.1...v0.10.5)

examples/node/app.js

@@ -202,9 +202,9 @@ function printRoomList() {
             dateStr = new Date(msg.getTs()).toISOString().replace(
                 /T/, ' ').replace(/\..+/, '');
         }
-        var me = roomList[i].getMember(myUserId);
-        if (me) {
-            fmt = fmts[me.membership];
+        var myMembership = roomList[i].getMyMembership();
+        if (myMembership) {
+            fmt = fmts[myMembership];
         }
         var roomName = fixWidth(roomList[i].name, 25);
         print(

package.json

@@ -1,6 +1,6 @@
 {
   "name": "matrix-js-sdk",
-  "version": "0.10.5",
+  "version": "0.11.0",
   "description": "Matrix Client-Server SDK for Javascript",
   "main": "index.js",
   "scripts": {

spec/integ/matrix-client-methods.spec.js

@@ -159,7 +159,7 @@ describe("MatrixClient", function() {
     describe("joinRoom", function() {
         it("should no-op if you've already joined a room", function() {
             const roomId = "!foo:bar";
-            const room = new Room(roomId);
+            const room = new Room(roomId, userId);
             room.addLiveEvents([
                 utils.mkMembership({
                     user: userId, room: roomId, mship: "join", event: true,

spec/unit/matrix-client.spec.js

@@ -380,7 +380,7 @@ describe("MatrixClient", function() {
             client.startClient();
         });
 
-        it("should transition ERROR -> PREPARED after /sync if prev failed",
+        it("should transition ERROR -> CATCHUP after /sync if prev failed",
         function(done) {
             const expectedStates = [];
             acceptKeepalives = false;
@@ -403,7 +403,7 @@ describe("MatrixClient", function() {
 
             expectedStates.push(["RECONNECTING", null]);
             expectedStates.push(["ERROR", "RECONNECTING"]);
-            expectedStates.push(["PREPARED", "ERROR"]);
+            expectedStates.push(["CATCHUP", "ERROR"]);
             client.on("sync", syncChecker(expectedStates, done));
             client.startClient();
         });

spec/unit/room-member.spec.js

@@ -192,6 +192,31 @@ describe("RoomMember", function() {
         });
     });
 
+    describe("isOutOfBand", function() {
+        it("should be set by markOutOfBand", function() {
+            const member = new RoomMember();
+            expect(member.isOutOfBand()).toEqual(false);
+            member.markOutOfBand();
+            expect(member.isOutOfBand()).toEqual(true);
+        });
+    });
+
+    describe("supersedesOutOfBand", function() {
+        it("should be set by markSupersedesOutOfBand", function() {
+            const member = new RoomMember();
+            expect(member.supersedesOutOfBand()).toEqual(false);
+            member.markSupersedesOutOfBand();
+            expect(member.supersedesOutOfBand()).toEqual(true);
+        });
+        it("should be cleared by clearSupersedesOutOfBand", function() {
+            const member = new RoomMember();
+            member.markSupersedesOutOfBand();
+            expect(member.supersedesOutOfBand()).toEqual(true);
+            member.clearSupersedesOutOfBand();
+            expect(member.supersedesOutOfBand()).toEqual(false);
+        });
+    });
+
     describe("setMembershipEvent", function() {
         const joinEvent = utils.mkMembership({
             event: true,

spec/unit/room-state.spec.js

@@ -11,6 +11,9 @@ describe("RoomState", function() {
     const roomId = "!foo:bar";
     const userA = "@alice:bar";
     const userB = "@bob:bar";
+    const userC = "@cleo:bar";
+    const userLazy = "@lazy:bar";
+
     let state;
 
     beforeEach(function() {
@@ -78,8 +81,8 @@ describe("RoomState", function() {
     });
 
     describe("getSentinelMember", function() {
-        it("should return null if there is no member", function() {
-            expect(state.getSentinelMember("@no-one:here")).toEqual(null);
+        it("should return a member with the user id as name", function() {
+            expect(state.getSentinelMember("@no-one:here").name).toEqual("@no-one:here");
         });
 
         it("should return a member which doesn't change when the state is updated",
@@ -162,6 +165,7 @@ describe("RoomState", function() {
             ];
             let emitCount = 0;
             state.on("RoomState.newMember", function(ev, st, mem) {
+                expect(state.getMember(mem.userId)).toEqual(mem);
                 expect(mem.userId).toEqual(memberEvents[emitCount].getSender());
                 expect(mem.membership).toBeFalsy();  // not defined yet
                 emitCount += 1;
@@ -222,7 +226,6 @@ describe("RoomState", function() {
 
         it("should call setPowerLevelEvent on a new RoomMember if power levels exist",
         function() {
-            const userC = "@cleo:bar";
             const memberEvent = utils.mkMembership({
                 mship: "join", user: userC, room: roomId, event: true,
             });
@@ -262,6 +265,166 @@ describe("RoomState", function() {
         });
     });
 
+    describe("setOutOfBandMembers", function() {
+        it("should add a new member", function() {
+            const oobMemberEvent = utils.mkMembership({
+                user: userLazy, mship: "join", room: roomId, event: true,
+            });
+            state.markOutOfBandMembersStarted();
+            state.setOutOfBandMembers([oobMemberEvent]);
+            const member = state.getMember(userLazy);
+            expect(member.userId).toEqual(userLazy);
+            expect(member.isOutOfBand()).toEqual(true);
+        });
+
+        it("should have no effect when not in correct status", function() {
+            state.setOutOfBandMembers([utils.mkMembership({
+                user: userLazy, mship: "join", room: roomId, event: true,
+            })]);
+            expect(state.getMember(userLazy)).toBeFalsy();
+        });
+
+        it("should emit newMember when adding a member", function() {
+            const userLazy = "@oob:hs";
+            const oobMemberEvent = utils.mkMembership({
+                user: userLazy, mship: "join", room: roomId, event: true,
+            });
+            let eventReceived = false;
+            state.once('RoomState.newMember', (_, __, member) => {
+                expect(member.userId).toEqual(userLazy);
+                eventReceived = true;
+            });
+            state.markOutOfBandMembersStarted();
+            state.setOutOfBandMembers([oobMemberEvent]);
+            expect(eventReceived).toEqual(true);
+        });
+
+        it("should overwrite existing members", function() {
+            const oobMemberEvent = utils.mkMembership({
+                user: userA, mship: "join", room: roomId, event: true,
+            });
+            state.markOutOfBandMembersStarted();
+            state.setOutOfBandMembers([oobMemberEvent]);
+            const memberA = state.getMember(userA);
+            expect(memberA.events.member.getId()).toEqual(oobMemberEvent.getId());
+            expect(memberA.isOutOfBand()).toEqual(true);
+        });
+
+        it("should allow later state events to overwrite", function() {
+            const oobMemberEvent = utils.mkMembership({
+                user: userA, mship: "join", room: roomId, event: true,
+            });
+            const memberEvent = utils.mkMembership({
+                user: userA, mship: "join", room: roomId, event: true,
+            });
+
+            state.markOutOfBandMembersStarted();
+            state.setOutOfBandMembers([oobMemberEvent]);
+            state.setStateEvents([memberEvent]);
+
+            const memberA = state.getMember(userA);
+            expect(memberA.events.member.getId()).toEqual(memberEvent.getId());
+            expect(memberA.isOutOfBand()).toEqual(false);
+        });
+
+        it("should emit members when updating a member", function() {
+            const oobMemberEvent = utils.mkMembership({
+                user: userA, mship: "join", room: roomId, event: true,
+            });
+            let eventReceived = false;
+            state.once('RoomState.members', (_, __, member) => {
+                expect(member.userId).toEqual(userA);
+                eventReceived = true;
+            });
+
+            state.markOutOfBandMembersStarted();
+            state.setOutOfBandMembers([oobMemberEvent]);
+            expect(eventReceived).toEqual(true);
+        });
+
+
+        it("should not overwrite members updated since starting loading oob",
+        function() {
+            const oobMemberEvent = utils.mkMembership({
+                user: userA, mship: "join", room: roomId, event: true,
+            });
+
+            const existingMemberEvent = utils.mkMembership({
+                user: userA, mship: "join", room: roomId, event: true,
+            });
+
+            state.markOutOfBandMembersStarted();
+            state.setStateEvents([existingMemberEvent]);
+            expect(state.getMember(userA).supersedesOutOfBand()).toEqual(true);
+            state.setOutOfBandMembers([oobMemberEvent]);
+
+            const memberA = state.getMember(userA);
+            expect(memberA.events.member.getId()).toEqual(existingMemberEvent.getId());
+            expect(memberA.isOutOfBand()).toEqual(false);
+            expect(memberA.supersedesOutOfBand()).toEqual(false);
+        });
+    });
+
+    describe("clone", function() {
+        it("should contain same information as original", function() {
+            // include OOB members in copy
+            state.markOutOfBandMembersStarted();
+            state.setOutOfBandMembers([utils.mkMembership({
+                user: userLazy, mship: "join", room: roomId, event: true,
+            })]);
+            const copy = state.clone();
+            // check individual members
+            [userA, userB, userLazy].forEach((userId) => {
+                const member = state.getMember(userId);
+                const memberCopy = copy.getMember(userId);
+                expect(member.name).toEqual(memberCopy.name);
+                expect(member.isOutOfBand()).toEqual(memberCopy.isOutOfBand());
+            });
+            // check member keys
+            expect(Object.keys(state.members)).toEqual(Object.keys(copy.members));
+            // check join count
+            expect(state.getJoinedMemberCount()).toEqual(copy.getJoinedMemberCount());
+        });
+
+        it("should copy supersedes flag when OOB loading is progress",
+        function() {
+            // include OOB members in copy
+            state.markOutOfBandMembersStarted();
+            state.setStateEvents([utils.mkMembership({
+                user: userA, mship: "join", room: roomId, event: true,
+            })]);
+            const copy = state.clone();
+            const memberA = state.getMember(userA);
+            const memberACopy = copy.getMember(userA);
+            expect(memberA.supersedesOutOfBand()).toEqual(true);
+            expect(memberACopy.supersedesOutOfBand()).toEqual(true);
+        });
+
+        it("should mark old copy as not waiting for out of band anymore", function() {
+            state.markOutOfBandMembersStarted();
+            const copy = state.clone();
+            copy.setOutOfBandMembers([utils.mkMembership({
+                user: userA, mship: "join", room: roomId, event: true,
+            })]);
+            // should have no effect as it should be marked in status finished just like copy
+            state.setOutOfBandMembers([utils.mkMembership({
+                user: userLazy, mship: "join", room: roomId, event: true,
+            })]);
+            expect(state.getMember(userLazy)).toBeFalsy();
+        });
+
+        it("should return copy independent of original", function() {
+            const copy = state.clone();
+            copy.setStateEvents([utils.mkMembership({
+                user: userLazy, mship: "join", room: roomId, event: true,
+            })]);
+
+            expect(state.getMember(userLazy)).toBeFalsy();
+            expect(state.getJoinedMemberCount()).toEqual(2);
+            expect(copy.getJoinedMemberCount()).toEqual(3);
+        });
+    });
+
     describe("setTypingEvent", function() {
         it("should call setTypingEvent on each RoomMember", function() {
             const typingEvent = utils.mkEvent({
@@ -284,13 +447,6 @@ describe("RoomState", function() {
     });
 
     describe("maySendStateEvent", function() {
-        it("should say non-joined members may not send state",
-        function() {
-            expect(state.maySendStateEvent(
-                'm.room.name', "@nobody:nowhere",
-            )).toEqual(false);
-        });
-
         it("should say any member may send state with no power level event",
         function() {
             expect(state.maySendStateEvent('m.room.name', userA)).toEqual(true);
@@ -366,15 +522,117 @@ describe("RoomState", function() {
         });
     });
 
-    describe("maySendEvent", function() {
-        it("should say non-joined members may not send events",
-        function() {
-            expect(state.maySendEvent(
-                'm.room.message', "@nobody:nowhere",
-            )).toEqual(false);
-            expect(state.maySendMessage("@nobody:nowhere")).toEqual(false);
+    describe("getJoinedMemberCount", function() {
+        beforeEach(() => {
+            state = new RoomState(roomId);
         });
 
+        it("should update after adding joined member", function() {
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "join",
+                    user: userA, room: roomId}),
+            ]);
+            expect(state.getJoinedMemberCount()).toEqual(1);
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "join",
+                    user: userC, room: roomId}),
+            ]);
+            expect(state.getJoinedMemberCount()).toEqual(2);
+        });
+    });
+
+    describe("getInvitedMemberCount", function() {
+        beforeEach(() => {
+            state = new RoomState(roomId);
+        });
+
+        it("should update after adding invited member", function() {
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "invite",
+                    user: userA, room: roomId}),
+            ]);
+            expect(state.getInvitedMemberCount()).toEqual(1);
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "invite",
+                    user: userC, room: roomId}),
+            ]);
+            expect(state.getInvitedMemberCount()).toEqual(2);
+        });
+    });
+
+    describe("setJoinedMemberCount", function() {
+        beforeEach(() => {
+            state = new RoomState(roomId);
+        });
+
+        it("should, once used, override counting members from state", function() {
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "join",
+                    user: userA, room: roomId}),
+            ]);
+            expect(state.getJoinedMemberCount()).toEqual(1);
+            state.setJoinedMemberCount(100);
+            expect(state.getJoinedMemberCount()).toEqual(100);
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "join",
+                    user: userC, room: roomId}),
+            ]);
+            expect(state.getJoinedMemberCount()).toEqual(100);
+        });
+
+        it("should, once used, override counting members from state, " +
+        "also after clone", function() {
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "join",
+                    user: userA, room: roomId}),
+            ]);
+            state.setJoinedMemberCount(100);
+            const copy = state.clone();
+            copy.setStateEvents([
+                utils.mkMembership({event: true, mship: "join",
+                    user: userC, room: roomId}),
+            ]);
+            expect(state.getJoinedMemberCount()).toEqual(100);
+        });
+    });
+
+    describe("setInvitedMemberCount", function() {
+        beforeEach(() => {
+            state = new RoomState(roomId);
+        });
+
+        it("should, once used, override counting members from state", function() {
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "invite",
+                    user: userB, room: roomId}),
+            ]);
+            expect(state.getInvitedMemberCount()).toEqual(1);
+            state.setInvitedMemberCount(100);
+            expect(state.getInvitedMemberCount()).toEqual(100);
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "invite",
+                    user: userC, room: roomId}),
+            ]);
+            expect(state.getInvitedMemberCount()).toEqual(100);
+        });
+
+        it("should, once used, override counting members from state, " +
+        "also after clone", function() {
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "invite",
+                    user: userB, room: roomId}),
+            ]);
+            state.setInvitedMemberCount(100);
+            const copy = state.clone();
+            copy.setStateEvents([
+                utils.mkMembership({event: true, mship: "invite",
+                    user: userC, room: roomId}),
+            ]);
+            expect(state.getInvitedMemberCount()).toEqual(100);
+        });
+    });
+
+    describe("maySendEvent", function() {
         it("should say any member may send events with no power level event",
         function() {
             expect(state.maySendEvent('m.room.message', userA)).toEqual(true);

spec/unit/room.spec.js

@@ -67,13 +67,14 @@ describe("Room", function() {
 
     describe("getMember", function() {
         beforeEach(function() {
-            // clobber members property with test data
-            room.currentState.members = {
-                "@alice:bar": {
-                    userId: userA,
-                    roomId: roomId,
-                },
-            };
+            room.currentState.getMember.andCall(function(userId) {
+                return {
+                    "@alice:bar": {
+                        userId: userA,
+                        roomId: roomId,
+                    },
+                }[userId];
+            });
         });
 
         it("should return null if the member isn't in current state", function() {
@@ -386,7 +387,7 @@ describe("Room", function() {
         let events = null;
 
         beforeEach(function() {
-            room = new Room(roomId, {timelineSupport: timelineSupport});
+            room = new Room(roomId, null, null, {timelineSupport: timelineSupport});
             // set events each time to avoid resusing Event objects (which
             // doesn't work because they get frozen)
             events = [
@@ -468,7 +469,7 @@ describe("Room", function() {
 
     describe("compareEventOrdering", function() {
         beforeEach(function() {
-            room = new Room(roomId, {timelineSupport: true});
+            room = new Room(roomId, null, null, {timelineSupport: true});
         });
 
         const events = [
@@ -570,72 +571,75 @@ describe("Room", function() {
     describe("hasMembershipState", function() {
         it("should return true for a matching userId and membership",
         function() {
-            room.currentState.members = {
-                "@alice:bar": { userId: "@alice:bar", membership: "join" },
-                "@bob:bar": { userId: "@bob:bar", membership: "invite" },
-            };
+            room.currentState.getMember.andCall(function(userId) {
+                return {
+                    "@alice:bar": { userId: "@alice:bar", membership: "join" },
+                    "@bob:bar": { userId: "@bob:bar", membership: "invite" },
+                }[userId];
+            });
             expect(room.hasMembershipState("@bob:bar", "invite")).toBe(true);
         });
 
         it("should return false if match membership but no match userId",
         function() {
-            room.currentState.members = {
-                "@alice:bar": { userId: "@alice:bar", membership: "join" },
-            };
+            room.currentState.getMember.andCall(function(userId) {
+                return {
+                    "@alice:bar": { userId: "@alice:bar", membership: "join" },
+                }[userId];
+            });
             expect(room.hasMembershipState("@bob:bar", "join")).toBe(false);
         });
 
         it("should return false if match userId but no match membership",
         function() {
-            room.currentState.members = {
-                "@alice:bar": { userId: "@alice:bar", membership: "join" },
-            };
+            room.currentState.getMember.andCall(function(userId) {
+                return {
+                    "@alice:bar": { userId: "@alice:bar", membership: "join" },
+                }[userId];
+            });
             expect(room.hasMembershipState("@alice:bar", "ban")).toBe(false);
         });
 
         it("should return false if no match membership or userId",
         function() {
-            room.currentState.members = {
-                "@alice:bar": { userId: "@alice:bar", membership: "join" },
-            };
+            room.currentState.getMember.andCall(function(userId) {
+                return {
+                    "@alice:bar": { userId: "@alice:bar", membership: "join" },
+                }[userId];
+            });
             expect(room.hasMembershipState("@bob:bar", "invite")).toBe(false);
         });
 
         it("should return false if no members exist",
         function() {
-            room.currentState.members = {};
             expect(room.hasMembershipState("@foo:bar", "join")).toBe(false);
         });
     });
 
     describe("recalculate", function() {
-        let stateLookup = {
-            // event.type + "$" event.state_key : MatrixEvent
-        };
-
         const setJoinRule = function(rule) {
-            stateLookup["m.room.join_rules$"] = utils.mkEvent({
+            room.addLiveEvents([utils.mkEvent({
                 type: "m.room.join_rules", room: roomId, user: userA, content: {
                     join_rule: rule,
                 }, event: true,
-            });
+            })]);
         };
         const setAliases = function(aliases, stateKey) {
             if (!stateKey) {
                 stateKey = "flibble";
             }
-            stateLookup["m.room.aliases$" + stateKey] = utils.mkEvent({
+            room.addLiveEvents([utils.mkEvent({
                 type: "m.room.aliases", room: roomId, skey: stateKey, content: {
                     aliases: aliases,
                 }, event: true,
-            });
+            })]);
         };
         const setRoomName = function(name) {
-            stateLookup["m.room.name$"] = utils.mkEvent({
+            room.addLiveEvents([utils.mkEvent({
                 type: "m.room.name", room: roomId, user: userA, content: {
                     name: name,
                 }, event: true,
-            });
+            })]);
         };
         const addMember = function(userId, state, opts) {
             if (!state) {
@@ -647,56 +651,14 @@ describe("Room", function() {
             opts.user = opts.user || userId;
             opts.skey = userId;
             opts.event = true;
-            stateLookup["m.room.member$" + userId] = utils.mkMembership(opts);
+            const event = utils.mkMembership(opts);
+            room.addLiveEvents([event]);
+            return event;
         };
 
         beforeEach(function() {
-            stateLookup = {};
-            room.currentState.getStateEvents.andCall(function(type, key) {
-                if (key === undefined) {
-                    const prefix = type + "$";
-                    const list = [];
-                    for (const stateBlob in stateLookup) {
-                        if (!stateLookup.hasOwnProperty(stateBlob)) {
-                            continue;
-                        }
-                        if (stateBlob.indexOf(prefix) === 0) {
-                            list.push(stateLookup[stateBlob]);
-                        }
-                    }
-                    return list;
-                } else {
-                    return stateLookup[type + "$" + key];
-                }
-            });
-            room.currentState.getMembers.andCall(function() {
-                const memberEvents = room.currentState.getStateEvents("m.room.member");
-                const members = [];
-                for (let i = 0; i < memberEvents.length; i++) {
-                    members.push({
-                        name: memberEvents[i].event.content &&
-                                memberEvents[i].event.content.displayname ?
-                                memberEvents[i].event.content.displayname :
-                                memberEvents[i].getStateKey(),
-                        userId: memberEvents[i].getStateKey(),
-                        events: { member: memberEvents[i] },
-                    });
-                }
-                return members;
-            });
-            room.currentState.getMember.andCall(function(userId) {
-                const memberEvent = room.currentState.getStateEvents(
-                    "m.room.member", userId,
-                );
-                return {
-                    name: memberEvent.event.content &&
-                            memberEvent.event.content.displayname ?
-                            memberEvent.event.content.displayname :
-                            memberEvent.getStateKey(),
-                    userId: memberEvent.getStateKey(),
-                    events: { member: memberEvent },
-                };
-            });
+            // no mocking
+            room = new Room(roomId, null, userA);
         });
 
         describe("Room.recalculate => Stripped State Events", function() {
@@ -704,8 +666,8 @@ describe("Room", function() {
             "room is an invite room", function() {
                 const roomName = "flibble";
 
-                addMember(userA, "invite");
-                stateLookup["m.room.member$" + userA].event.invite_room_state = [
+                const event = addMember(userA, "invite");
+                event.event.invite_room_state = [
                     {
                         type: "m.room.name",
                         state_key: "",
@@ -715,30 +677,108 @@ describe("Room", function() {
                     },
                 ];
 
-                room.recalculate(userA);
-                expect(room.currentState.setStateEvents).toHaveBeenCalled();
-                // first call, first arg (which is an array), first element in array
-                const fakeEvent = room.currentState.setStateEvents.calls[0].
-                      arguments[0][0];
-                expect(fakeEvent.getContent()).toEqual({
-                    name: roomName,
-                });
+                room.recalculate();
+                expect(room.name).toEqual(roomName);
             });
 
             it("should not clobber state events if it isn't an invite room", function() {
-                addMember(userA, "join");
-                stateLookup["m.room.member$" + userA].event.invite_room_state = [
+                const event = addMember(userA, "join");
+                const roomName = "flibble";
+                setRoomName(roomName);
+                const roomNameToIgnore = "ignoreme";
+                event.event.invite_room_state = [
                     {
                         type: "m.room.name",
                         state_key: "",
                         content: {
-                            name: "flibble",
+                            name: roomNameToIgnore,
                         },
                     },
                 ];
 
-                room.recalculate(userA);
-                expect(room.currentState.setStateEvents).toNotHaveBeenCalled();
+                room.recalculate();
+                expect(room.name).toEqual(roomName);
+            });
+        });
+
+        describe("Room.recalculate => Room Name using room summary", function() {
+            it("should use room heroes if available", function() {
+                addMember(userA, "invite");
+                addMember(userB);
+                addMember(userC);
+                addMember(userD);
+                room.setSummary({
+                    "m.heroes": [userB, userC, userD],
+                });
+
+                room.recalculate();
+                expect(room.name).toEqual(`${userB} and 2 others`);
+            });
+
+            it("missing hero member state reverts to mxid", function() {
+                room.setSummary({
+                    "m.heroes": [userB],
+                    "m.joined_member_count": 2,
+                });
+
+                room.recalculate();
+                expect(room.name).toEqual(userB);
+            });
+
+            it("uses hero name from state", function() {
+                const name = "Mr B";
+                addMember(userA, "invite");
+                addMember(userB, "join", {name});
+                room.setSummary({
+                    "m.heroes": [userB],
+                });
+
+                room.recalculate();
+                expect(room.name).toEqual(name);
+            });
+
+            it("uses counts from summary", function() {
+                const name = "Mr B";
+                addMember(userB, "join", {name});
+                room.setSummary({
+                    "m.heroes": [userB],
+                    "m.joined_member_count": 50,
+                    "m.invited_member_count": 50,
+                });
+                room.recalculate();
+                expect(room.name).toEqual(`${name} and 98 others`);
+            });
+
+            it("relies on heroes in case of absent counts", function() {
+                const nameB = "Mr Bean";
+                const nameC = "Mel C";
+                addMember(userB, "join", {name: nameB});
+                addMember(userC, "join", {name: nameC});
+                room.setSummary({
+                    "m.heroes": [userB, userC],
+                });
+                room.recalculate();
+                expect(room.name).toEqual(`${nameB} and ${nameC}`);
+            });
+
+            it("uses only heroes", function() {
+                const nameB = "Mr Bean";
+                addMember(userB, "join", {name: nameB});
+                addMember(userC, "join");
+                room.setSummary({
+                    "m.heroes": [userB],
+                });
+                room.recalculate();
+                expect(room.name).toEqual(nameB);
+            });
+
+            it("reverts to empty room in case of self chat", function() {
+                room.setSummary({
+                    "m.heroes": [],
+                    "m.invited_member_count": 1,
+                });
+                room.recalculate();
+                expect(room.name).toEqual("Empty room");
             });
         });
 
@@ -751,7 +791,7 @@ describe("Room", function() {
                 addMember(userB);
                 addMember(userC);
                 addMember(userD);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 // we expect at least 1 member to be mentioned
                 const others = [userB, userC, userD];
@@ -772,7 +812,7 @@ describe("Room", function() {
                 addMember(userA);
                 addMember(userB);
                 addMember(userC);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name.indexOf(userB)).toNotEqual(-1, name);
                 expect(name.indexOf(userC)).toNotEqual(-1, name);
@@ -785,7 +825,7 @@ describe("Room", function() {
                 addMember(userA);
                 addMember(userB);
                 addMember(userC);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name.indexOf(userB)).toNotEqual(-1, name);
                 expect(name.indexOf(userC)).toNotEqual(-1, name);
@@ -797,7 +837,7 @@ describe("Room", function() {
                 setJoinRule("public");
                 addMember(userA);
                 addMember(userB);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name.indexOf(userB)).toNotEqual(-1, name);
             });
@@ -808,7 +848,7 @@ describe("Room", function() {
                 setJoinRule("invite");
                 addMember(userA);
                 addMember(userB);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name.indexOf(userB)).toNotEqual(-1, name);
             });
@@ -818,7 +858,7 @@ describe("Room", function() {
                 setJoinRule("invite");
                 addMember(userA, "invite", {user: userB});
                 addMember(userB);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name.indexOf(userB)).toNotEqual(-1, name);
             });
@@ -828,7 +868,7 @@ describe("Room", function() {
                 const alias = "#room_alias:here";
                 setJoinRule("invite");
                 setAliases([alias, "#another:one"]);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name).toEqual(alias);
             });
@@ -838,7 +878,7 @@ describe("Room", function() {
                 const alias = "#room_alias:here";
                 setJoinRule("public");
                 setAliases([alias, "#another:one"]);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name).toEqual(alias);
             });
@@ -848,7 +888,7 @@ describe("Room", function() {
                 const roomName = "A mighty name indeed";
                 setJoinRule("invite");
                 setRoomName(roomName);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name).toEqual(roomName);
             });
@@ -858,25 +898,23 @@ describe("Room", function() {
                 const roomName = "A mighty name indeed";
                 setJoinRule("public");
                 setRoomName(roomName);
-                room.recalculate(userA);
-                const name = room.name;
-                expect(name).toEqual(roomName);
+                room.recalculate();
+                expect(room.name).toEqual(roomName);
             });
 
             it("should return 'Empty room' for private (invite join_rules) rooms if" +
             " a room name and alias don't exist and it is a self-chat.", function() {
                 setJoinRule("invite");
                 addMember(userA);
-                room.recalculate(userA);
-                const name = room.name;
-                expect(name).toEqual("Empty room");
+                room.recalculate();
+                expect(room.name).toEqual("Empty room");
             });
 
             it("should return 'Empty room' for public (public join_rules) rooms if a" +
             " room name and alias don't exist and it is a self-chat.", function() {
                 setJoinRule("public");
                 addMember(userA);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name).toEqual("Empty room");
             });
@@ -884,7 +922,7 @@ describe("Room", function() {
             it("should return 'Empty room' if there is no name, " +
                "alias or members in the room.",
             function() {
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name).toEqual("Empty room");
             });
@@ -893,9 +931,9 @@ describe("Room", function() {
                "available",
             function() {
                 setJoinRule("invite");
-                addMember(userA, 'join', {name: "Alice"});
-                addMember(userB, "invite", {user: userA});
-                room.recalculate(userB);
+                addMember(userB, 'join', {name: "Alice"});
+                addMember(userA, "invite", {user: userA});
+                room.recalculate();
                 const name = room.name;
                 expect(name).toEqual("Alice");
             });
@@ -903,11 +941,11 @@ describe("Room", function() {
             it("should return inviter mxid if display name not available",
             function() {
                 setJoinRule("invite");
-                addMember(userA);
-                addMember(userB, "invite", {user: userA});
-                room.recalculate(userB);
+                addMember(userB);
+                addMember(userA, "invite", {user: userA});
+                room.recalculate();
                 const name = room.name;
-                expect(name).toEqual(userA);
+                expect(name).toEqual(userB);
             });
         });
     });
@@ -1154,7 +1192,7 @@ describe("Room", function() {
     describe("addPendingEvent", function() {
         it("should add pending events to the pendingEventList if " +
                       "pendingEventOrdering == 'detached'", function() {
-            const room = new Room(roomId, {
+            const room = new Room(roomId, null, userA, {
                 pendingEventOrdering: "detached",
             });
             const eventA = utils.mkMessage({
@@ -1180,7 +1218,7 @@ describe("Room", function() {
 
         it("should add pending events to the timeline if " +
                       "pendingEventOrdering == 'chronological'", function() {
-            room = new Room(roomId, {
+            room = new Room(roomId, null, userA, {
                 pendingEventOrdering: "chronological",
             });
             const eventA = utils.mkMessage({
@@ -1204,7 +1242,7 @@ describe("Room", function() {
 
     describe("updatePendingEvent", function() {
         it("should remove cancelled events from the pending list", function() {
-            const room = new Room(roomId, {
+            const room = new Room(roomId, null, userA, {
                 pendingEventOrdering: "detached",
             });
             const eventA = utils.mkMessage({
@@ -1240,7 +1278,7 @@ describe("Room", function() {
 
 
         it("should remove cancelled events from the timeline", function() {
-            const room = new Room(roomId);
+            const room = new Room(roomId, null, userA);
             const eventA = utils.mkMessage({
                 room: roomId, user: userA, event: true,
             });
@@ -1272,4 +1310,141 @@ describe("Room", function() {
             expect(callCount).toEqual(1);
         });
     });
+
+    describe("loadMembersIfNeeded", function() {
+        function createClientMock(serverResponse, storageResponse = null) {
+            return {
+                getEventMapper: function() {
+                    // events should already be MatrixEvents
+                    return function(event) {return event;};
+                },
+                isRoomEncrypted: function() {
+                    return false;
+                },
+                _http: {
+                    serverResponse,
+                    authedRequest: function() {
+                        if (this.serverResponse instanceof Error) {
+                            return Promise.reject(this.serverResponse);
+                        } else {
+                            return Promise.resolve({chunk: this.serverResponse});
+                        }
+                    },
+                },
+                store: {
+                    storageResponse,
+                    storedMembers: null,
+                    getOutOfBandMembers: function() {
+                        if (this.storageResponse instanceof Error) {
+                            return Promise.reject(this.storageResponse);
+                        } else {
+                            return Promise.resolve(this.storageResponse);
+                        }
+                    },
+                    setOutOfBandMembers: function(roomId, memberEvents) {
+                        this.storedMembers = memberEvents;
+                        return Promise.resolve();
+                    },
+                    getSyncToken: () => "sync_token",
+                },
+            };
+        }
+
+        const memberEvent = utils.mkMembership({
+            user: "@user_a:bar", mship: "join",
+            room: roomId, event: true, name: "User A",
+        });
+
+        it("should load members from server on first call", async function() {
+            const client = createClientMock([memberEvent]);
+            const room = new Room(roomId, client, null, {lazyLoadMembers: true});
+            await room.loadMembersIfNeeded();
+            const memberA = room.getMember("@user_a:bar");
+            expect(memberA.name).toEqual("User A");
+            const storedMembers = client.store.storedMembers;
+            expect(storedMembers.length).toEqual(1);
+            expect(storedMembers[0].event_id).toEqual(memberEvent.getId());
+        });
+
+        it("should take members from storage if available", async function() {
+            const memberEvent2 = utils.mkMembership({
+                user: "@user_a:bar", mship: "join",
+                room: roomId, event: true, name: "Ms A",
+            });
+            const client = createClientMock([memberEvent2], [memberEvent]);
+            const room = new Room(roomId, client, null, {lazyLoadMembers: true});
+
+            await room.loadMembersIfNeeded();
+
+            const memberA = room.getMember("@user_a:bar");
+            expect(memberA.name).toEqual("User A");
+        });
+
+        it("should allow retry on error", async function() {
+            const client = createClientMock(new Error("server says no"));
+            const room = new Room(roomId, client, null, {lazyLoadMembers: true});
+            let hasThrown = false;
+            try {
+                await room.loadMembersIfNeeded();
+            } catch(err) {
+                hasThrown = true;
+            }
+            expect(hasThrown).toEqual(true);
+
+            client._http.serverResponse = [memberEvent];
+            await room.loadMembersIfNeeded();
+            const memberA = room.getMember("@user_a:bar");
+            expect(memberA.name).toEqual("User A");
+        });
+    });
+
+    describe("getMyMembership", function() {
+        it("should return synced membership if membership isn't available yet",
+        function() {
+            const room = new Room(roomId, null, userA);
+            room.setSyncedMembership("invite");
+            expect(room.getMyMembership()).toEqual("invite");
+            room.addLiveEvents([utils.mkMembership({
+                user: userA, mship: "join",
+                room: roomId, event: true,
+            })]);
+            expect(room.getMyMembership()).toEqual("join");
+        });
+    });
+
+    describe("guessDMUserId", function() {
+        it("should return first hero id",
+        function() {
+            const room = new Room(roomId, null, userA);
+            room.setSummary({'m.heroes': [userB]});
+            expect(room.guessDMUserId()).toEqual(userB);
+        });
+        it("should return first member that isn't self",
+        function() {
+            const room = new Room(roomId, null, userA);
+            room.addLiveEvents([utils.mkMembership({
+                user: userB, mship: "join",
+                room: roomId, event: true,
+            })]);
+            expect(room.guessDMUserId()).toEqual(userB);
+        });
+        it("should return self if only member present",
+        function() {
+            const room = new Room(roomId, null, userA);
+            expect(room.guessDMUserId()).toEqual(userA);
+        });
+    });
+
+    describe("maySendMessage", function() {
+        it("should return false if synced membership not join",
+        function() {
+            const room = new Room(roomId, null, userA);
+            room.setSyncedMembership("invite");
+            expect(room.maySendMessage()).toEqual(false);
+            room.setSyncedMembership("leave");
+            expect(room.maySendMessage()).toEqual(false);
+            room.setSyncedMembership("join");
+            expect(room.maySendMessage()).toEqual(true);
+        });
+    });
 });

spec/unit/sync-accumulator.spec.js

@@ -52,6 +52,11 @@ describe("SyncAccumulator", function() {
                                 member("bob", "join"),
                             ],
                         },
+                        summary: {
+                            "m.heroes": undefined,
+                            "m.joined_member_count": undefined,
+                            "m.invited_member_count": undefined,
+                        },
                         timeline: {
                             events: [msg("alice", "hi")],
                             prev_batch: "something",
@@ -318,6 +323,58 @@ describe("SyncAccumulator", function() {
             },
         });
     });
+
+    describe("summary field", function() {
+        function createSyncResponseWithSummary(summary) {
+            return {
+                next_batch: "abc",
+                rooms: {
+                    invite: {},
+                    leave: {},
+                    join: {
+                        "!foo:bar": {
+                            account_data: { events: [] },
+                            ephemeral: { events: [] },
+                            unread_notifications: {},
+                            state: {
+                                events: [],
+                            },
+                            summary: summary,
+                            timeline: {
+                                events: [],
+                                prev_batch: "something",
+                            },
+                        },
+                    },
+                },
+            };
+        }
+
+        it("should copy summary properties", function() {
+            sa.accumulate(createSyncResponseWithSummary({
+                "m.heroes": ["@alice:bar"],
+                "m.invited_member_count": 2,
+            }));
+            const summary = sa.getJSON().roomsData.join["!foo:bar"].summary;
+            expect(summary["m.invited_member_count"]).toEqual(2);
+            expect(summary["m.heroes"]).toEqual(["@alice:bar"]);
+        });
+
+        it("should accumulate summary properties", function() {
+            sa.accumulate(createSyncResponseWithSummary({
+                "m.heroes": ["@alice:bar"],
+                "m.invited_member_count": 2,
+            }));
+            sa.accumulate(createSyncResponseWithSummary({
+                "m.heroes": ["@bob:bar"],
+                "m.joined_member_count": 5,
+            }));
+            const summary = sa.getJSON().roomsData.join["!foo:bar"].summary;
+            expect(summary["m.invited_member_count"]).toEqual(2);
+            expect(summary["m.joined_member_count"]).toEqual(5);
+            expect(summary["m.heroes"]).toEqual(["@bob:bar"]);
+        });
+    });
 });
 
 function syncSkeleton(joinObj) {

src/base-apis.js

@@ -417,6 +417,69 @@ MatrixBaseApis.prototype.roomState = function(roomId, callback) {
     return this._http.authedRequest(callback, "GET", path);
 };
 
+/**
+ * Get an event in a room by its event id.
+ * @param {string} roomId
+ * @param {string} eventId
+ * @param {module:client.callback} callback Optional.
+ *
+ * @return {Promise} Resolves to an object containing the event.
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.fetchRoomEvent = function(roomId, eventId, callback) {
+    const path = utils.encodeUri(
+        "/rooms/$roomId/event/$eventId", {
+            $roomId: roomId,
+            $eventId: eventId,
+        },
+    );
+    return this._http.authedRequest(callback, "GET", path);
+};
+
+/**
+ * @param {string} roomId
+ * @param {string} includeMembership the membership type to include in the response
+ * @param {string} excludeMembership the membership type to exclude from the response
+ * @param {string} atEventId the id of the event for which moment in the timeline the members should be returned for
+ * @param {module:client.callback} callback Optional.
+ * @return {module:client.Promise} Resolves: dictionary of userid to profile information
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.members =
+function(roomId, includeMembership, excludeMembership, atEventId, callback) {
+    const queryParams = {};
+    if (includeMembership) {
+        queryParams.membership = includeMembership;
+    }
+    if (excludeMembership) {
+        queryParams.not_membership = excludeMembership;
+    }
+    if (atEventId) {
+        queryParams.at = atEventId;
+    }
+
+    const queryString = utils.encodeParams(queryParams);
+
+    const path = utils.encodeUri("/rooms/$roomId/members?" + queryString,
+        {$roomId: roomId});
+    return this._http.authedRequest(callback, "GET", path);
+};
+
+/**
+ * Upgrades a room to a new protocol version
+ * @param {string} roomId
+ * @param {string} newVersion The target version to upgrade to
+ * @return {module:client.Promise} Resolves: Object with key 'replacement_room'
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.upgradeRoom = function(roomId, newVersion) {
+    const path = utils.encodeUri("/rooms/$roomId/upgrade", {$roomId: roomId});
+    return this._http.authedRequest(
+        undefined, "POST", path, undefined, {new_version: newVersion},
+    );
+};
+
+
 /**
  * @param {string} groupId
  * @return {module:client.Promise} Resolves: Group summary object

src/client.js

@@ -45,6 +45,18 @@ const ContentHelpers = require("./content-helpers");
 import ReEmitter from './ReEmitter';
 import RoomList from './crypto/RoomList';
 
+
+const LAZY_LOADING_MESSAGES_FILTER = {
+    lazy_load_members: true,
+};
+
+const LAZY_LOADING_SYNC_FILTER = {
+    room: {
+        state: LAZY_LOADING_MESSAGES_FILTER,
+    },
+};
+
+
 const SCROLLBACK_DELAY_MS = 3000;
 let CRYPTO_ENABLED = false;
 
@@ -191,6 +203,8 @@ function MatrixClient(opts) {
 
     // The pushprocessor caches useful things, so keep one and re-use it
     this._pushProcessor = new PushProcessor(this);
+
+    this._serverSupportsLazyLoading = null;
 }
 utils.inherits(MatrixClient, EventEmitter);
 utils.extend(MatrixClient.prototype, MatrixBaseApis.prototype);
@@ -287,6 +301,21 @@ MatrixClient.prototype.getSyncState = function() {
     return this._syncApi.getSyncState();
 };
 
+/**
+ * Returns the additional data object associated with
+ * the current sync state, or null if there is no
+ * such data.
+ * Sync errors, if available, are put in the 'error' key of
+ * this object.
+ * @return {?Object}
+ */
+MatrixClient.prototype.getSyncStateData = function() {
+    if (!this._syncApi) {
+        return null;
+    }
+    return this._syncApi.getSyncStateData();
+};
+
 /**
  * Return whether the client is configured for a guest account.
  * @return {boolean} True if this is a guest access_token (or no token is supplied).
@@ -673,6 +702,21 @@ MatrixClient.prototype.isRoomEncrypted = function(roomId) {
     return this._roomList.isRoomEncrypted(roomId);
 };
 
+/**
+ * Forces the current outbound group session to be discarded such
+ * that another one will be created next time an event is sent.
+ *
+ * @param {string} roomId The ID of the room to discard the session for
+ *
+ * This should not normally be necessary.
+ */
+MatrixClient.prototype.forceDiscardSession = function(roomId) {
+    if (!this._crypto) {
+        throw new Error("End-to-End encryption disabled");
+    }
+    this._crypto.forceDiscardSession(roomId);
+};
+
 /**
  * Get a list containing all of the room keys
  *
@@ -750,6 +794,37 @@ MatrixClient.prototype.getRooms = function() {
     return this.store.getRooms();
 };
 
+/**
+ * Retrieve all rooms that should be displayed to the user
+ * This is essentially getRooms() with some rooms filtered out, eg. old versions
+ * of rooms that have been replaced or (in future) other rooms that have been
+ * marked at the protocol level as not to be displayed to the user.
+ * @return {Room[]} A list of rooms, or an empty list if there is no data store.
+ */
+MatrixClient.prototype.getVisibleRooms = function() {
+    const allRooms = this.store.getRooms();
+
+    const replacedRooms = new Set();
+    for (const r of allRooms) {
+        const createEvent = r.currentState.getStateEvents('m.room.create', '');
+        // invites are included in this list and we don't know their create events yet
+        if (createEvent) {
+            const predecessor = createEvent.getContent()['predecessor'];
+            if (predecessor && predecessor['room_id']) {
+                replacedRooms.add(predecessor['room_id']);
+            }
+        }
+    }
+
+    return allRooms.filter((r) => {
+        const tombstone = r.currentState.getStateEvents('m.room.tombstone', '');
+        if (tombstone && replacedRooms.has(r.roomId)) {
+            return false;
+        }
+        return true;
+    });
+};
+
 /**
  * Retrieve a user.
  * @param {string} userId The user ID to retrieve.
@@ -1919,14 +1994,6 @@ MatrixClient.prototype.scrollback = function(room, limit, callback) {
     // reduce the required number of events appropriately
     limit = limit - numAdded;
 
-    const path = utils.encodeUri(
-        "/rooms/$roomId/messages", {$roomId: room.roomId},
-    );
-    const params = {
-        from: room.oldState.paginationToken,
-        limit: limit,
-        dir: 'b',
-    };
     const defer = Promise.defer();
     info = {
         promise: defer.promise,
@@ -1936,9 +2003,17 @@ MatrixClient.prototype.scrollback = function(room, limit, callback) {
     // wait for a time before doing this request
     // (which may be 0 in order not to special case the code paths)
     Promise.delay(timeToWaitMs).then(function() {
-        return self._http.authedRequest(callback, "GET", path, params);
+        return self._createMessagesRequest(
+            room.roomId,
+            room.oldState.paginationToken,
+            limit,
+            'b');
     }).done(function(res) {
         const matrixEvents = utils.map(res.chunk, _PojoToMatrixEventMapper(self));
+        if (res.state) {
+            const stateEvents = utils.map(res.state, _PojoToMatrixEventMapper(self));
+            room.currentState.setUnknownStateEvents(stateEvents);
+        }
         room.addEventsToTimeline(matrixEvents, true, room.getLiveTimeline());
         room.oldState.paginationToken = res.end;
         if (res.chunk.length === 0) {
@@ -1957,73 +2032,6 @@ MatrixClient.prototype.scrollback = function(room, limit, callback) {
     return defer.promise;
 };
 
-/**
- * Take an EventContext, and back/forward-fill results.
- *
- * @param {module:models/event-context.EventContext} eventContext  context
- *    object to be updated
- * @param {Object}  opts
- * @param {boolean} opts.backwards  true to fill backwards, false to go forwards
- * @param {boolean} opts.limit      number of events to request
- *
- * @return {module:client.Promise} Resolves: updated EventContext object
- * @return {Error} Rejects: with an error response.
- */
-MatrixClient.prototype.paginateEventContext = function(eventContext, opts) {
-    // TODO: we should implement a backoff (as per scrollback()) to deal more
-    // nicely with HTTP errors.
-    opts = opts || {};
-    const backwards = opts.backwards || false;
-
-    const token = eventContext.getPaginateToken(backwards);
-    if (!token) {
-        // no more results.
-        return Promise.reject(new Error("No paginate token"));
-    }
-
-    const dir = backwards ? 'b' : 'f';
-    const pendingRequest = eventContext._paginateRequests[dir];
-
-    if (pendingRequest) {
-        // already a request in progress - return the existing promise
-        return pendingRequest;
-    }
-
-    const path = utils.encodeUri(
-        "/rooms/$roomId/messages", {$roomId: eventContext.getEvent().getRoomId()},
-    );
-    const params = {
-        from: token,
-        limit: ('limit' in opts) ? opts.limit : 30,
-        dir: dir,
-    };
-
-    const self = this;
-    const promise =
-        self._http.authedRequest(undefined, "GET", path, params,
-    ).then(function(res) {
-        let token = res.end;
-        if (res.chunk.length === 0) {
-            token = null;
-        } else {
-            const matrixEvents = utils.map(res.chunk, self.getEventMapper());
-            if (backwards) {
-                // eventContext expects the events in timeline order, but
-                // back-pagination returns them in reverse order.
-                matrixEvents.reverse();
-            }
-            eventContext.addEvents(matrixEvents, backwards);
-        }
-        eventContext.setPaginateToken(token, backwards);
-        return eventContext;
-    }).finally(function() {
-        eventContext._paginateRequests[dir] = null;
-    });
-    eventContext._paginateRequests[dir] = promise;
-
-    return promise;
-};
-
 /**
  * Get an EventTimeline for the given event
  *
@@ -2057,11 +2065,16 @@ MatrixClient.prototype.getEventTimeline = function(timelineSet, eventId) {
         },
     );
 
+    let params = undefined;
+    if (this._clientOpts.lazyLoadMembers) {
+        params = {filter: JSON.stringify(LAZY_LOADING_MESSAGES_FILTER)};
+    }
+
     // TODO: we should implement a backoff (as per scrollback()) to deal more
     // nicely with HTTP errors.
     const self = this;
     const promise =
-        self._http.authedRequest(undefined, "GET", path,
+        self._http.authedRequest(undefined, "GET", path, params,
     ).then(function(res) {
         if (!res.event) {
             throw new Error("'event' not in '/context' result - homeserver too old?");
@@ -2088,6 +2101,8 @@ MatrixClient.prototype.getEventTimeline = function(timelineSet, eventId) {
             timeline.initialiseState(utils.map(res.state,
                                                self.getEventMapper()));
             timeline.getState(EventTimeline.FORWARDS).paginationToken = res.end;
+        } else {
+            timeline.getState(EventTimeline.BACKWARDS).setUnknownStateEvents(res.state);
         }
         timelineSet.addEventsToTimeline(matrixEvents, true, timeline, res.start);
 
@@ -2102,6 +2117,49 @@ MatrixClient.prototype.getEventTimeline = function(timelineSet, eventId) {
     return promise;
 };
 
+/**
+ * Makes a request to /messages with the appropriate lazy loading filter set.
+ * XXX: if we do get rid of scrollback (as it's not used at the moment),
+ * we could inline this method again in paginateEventTimeline as that would
+ * then be the only call-site
+ * @param {string} roomId
+ * @param {string} fromToken
+ * @param {number} limit the maximum amount of events the retrieve
+ * @param {string} dir 'f' or 'b'
+ * @param {Filter} timelineFilter the timeline filter to pass
+ * @return {Promise}
+ */
+MatrixClient.prototype._createMessagesRequest =
+function(roomId, fromToken, limit, dir, timelineFilter = undefined) {
+    const path = utils.encodeUri(
+        "/rooms/$roomId/messages", {$roomId: roomId},
+    );
+    if (limit === undefined) {
+        limit = 30;
+    }
+    const params = {
+        from: fromToken,
+        limit: limit,
+        dir: dir,
+    };
+
+    let filter = null;
+    if (this._clientOpts.lazyLoadMembers) {
+        // create a shallow copy of LAZY_LOADING_MESSAGES_FILTER,
+        // so the timelineFilter doesn't get written into it below
+        filter = Object.assign({}, LAZY_LOADING_MESSAGES_FILTER);
+    }
+    if (timelineFilter) {
+        // XXX: it's horrific that /messages' filter parameter doesn't match
+        // /sync's one - see https://matrix.org/jira/browse/SPEC-451
+        filter = filter || {};
+        Object.assign(filter, timelineFilter.getRoomTimelineFilterComponent());
+    }
+    if (filter) {
+        params.filter = JSON.stringify(filter);
+    }
+    return this._http.authedRequest(undefined, "GET", path, params);
+};
 
 /**
  * Take an EventTimeline, and back/forward-fill results.
@@ -2196,25 +2254,18 @@ MatrixClient.prototype.paginateEventTimeline = function(eventTimeline, opts) {
             throw new Error("Unknown room " + eventTimeline.getRoomId());
         }
 
-        path = utils.encodeUri(
-            "/rooms/$roomId/messages", {$roomId: eventTimeline.getRoomId()},
-        );
-        params = {
-            from: token,
-            limit: ('limit' in opts) ? opts.limit : 30,
-            dir: dir,
-        };
-
-        const filter = eventTimeline.getFilter();
-        if (filter) {
-            // XXX: it's horrific that /messages' filter parameter doesn't match
-            // /sync's one - see https://matrix.org/jira/browse/SPEC-451
-            params.filter = JSON.stringify(filter.getRoomTimelineFilterComponent());
-        }
-
-        promise =
-            this._http.authedRequest(undefined, "GET", path, params,
-        ).then(function(res) {
+        promise = this._createMessagesRequest(
+            eventTimeline.getRoomId(),
+            token,
+            opts.limit,
+            dir,
+            eventTimeline.getFilter());
+        promise.then(function(res) {
+            if (res.state) {
+                const roomState = eventTimeline.getState(dir);
+                const stateEvents = utils.map(res.state, self.getEventMapper());
+                roomState.setUnknownStateEvents(stateEvents);
+            }
             const token = res.end;
             const matrixEvents = utils.map(res.chunk, self.getEventMapper());
             eventTimeline.getTimelineSet()
@@ -3019,8 +3070,11 @@ MatrixClient.prototype.getTurnServers = function() {
  *
  * @param {Boolean=} opts.disablePresence True to perform syncing without automatically
  * updating presence.
+ * @param {Boolean=} opts.lazyLoadMembers True to not load all membership events during
+ * initial sync but fetch them when needed by calling `loadOutOfBandMembers`
+ * This will override the filter option at this moment.
  */
-MatrixClient.prototype.startClient = function(opts) {
+MatrixClient.prototype.startClient = async function(opts) {
     if (this.clientRunning) {
         // client is already running.
         return;
@@ -3050,6 +3104,21 @@ MatrixClient.prototype.startClient = function(opts) {
     // shallow-copy the opts dict before modifying and storing it
     opts = Object.assign({}, opts);
 
+    if (opts.lazyLoadMembers) {
+        const supported = await this.doesServerSupportLazyLoading();
+        if (supported) {
+            opts.filter = await this.createFilter(LAZY_LOADING_SYNC_FILTER);
+        } else {
+            console.log("LL: lazy loading requested but not supported " +
+                "by server, so disabling");
+            opts.lazyLoadMembers = false;
+        }
+    }
+
+    if (opts.lazyLoadMembers && this._crypto) {
+        this._crypto.enableLazyLoading();
+    }
+
     opts.crypto = this._crypto;
     opts.canResetEntireTimeline = (roomId) => {
         if (!this._canResetTimelineCallback) {
@@ -3085,6 +3154,36 @@ MatrixClient.prototype.stopClient = function() {
     global.clearTimeout(this._checkTurnServersTimeoutID);
 };
 
+/*
+ * Query the server to see if it support members lazy loading
+ * @return {Promise<boolean>} true if server supports lazy loading
+ */
+MatrixClient.prototype.doesServerSupportLazyLoading = async function() {
+    if (this._serverSupportsLazyLoading === null) {
+        const response = await this._http.request(
+            undefined, // callback
+            "GET", "/_matrix/client/versions",
+            undefined, // queryParams
+            undefined, // data
+            {
+                prefix: '',
+            },
+        );
+        const unstableFeatures = response["unstable_features"];
+        this._serverSupportsLazyLoading =
+            unstableFeatures && unstableFeatures["m.lazy_load_members"];
+    }
+    return this._serverSupportsLazyLoading;
+};
+
+/*
+ * Get if lazy loading members is being used.
+ * @return {boolean} Whether or not members are lazy loaded by this client
+ */
+MatrixClient.prototype.hasLazyLoadMembersEnabled = function() {
+    return !!this._clientOpts.lazyLoadMembers;
+};
+
 /*
  * Set a function which is called when /sync returns a 'limited' response.
  * It is called with a room ID and returns a boolean. It should return 'true' if the SDK
@@ -3430,6 +3529,12 @@ module.exports.CRYPTO_ENABLED = CRYPTO_ENABLED;
  * a state of SYNCING. <i>This is the equivalent of "syncComplete" in the
  * previous API.</i></li>
  *
+ * <li>CATCHUP: The client has detected the connection to the server might be
+ * available again and will now try to do a sync again. As this sync might take
+ * a long time (depending how long ago was last synced, and general server
+ * performance) the client is put in this mode so the UI can reflect trying
+ * to catch up with the server after losing connection.</li>
+ *
  * <li>SYNCING : The client is currently polling for new events from the server.
  * This will be called <i>after</i> processing latest events from a sync.</li>
  *
@@ -3453,11 +3558,11 @@ module.exports.CRYPTO_ENABLED = CRYPTO_ENABLED;
  *                                          +---->STOPPED
  *                                          |
  *              +----->PREPARED -------> SYNCING <--+
- *              |        ^                |  ^      |
- *              |        |                |  |      |
- *              |        |                V  |      |
- *   null ------+        |  +--------RECONNECTING   |
- *              |        |  V                       |
+ *              |                        ^  |  ^    |
+ *              |      CATCHUP ----------+  |  |    |
+ *              |        ^                  V  |    |
+ *   null ------+        |  +------- RECONNECTING   |
+ *              |        V  V                       |
  *              +------->ERROR ---------------------+
  *
  * NB: 'null' will never be emitted by this event.
@@ -3507,7 +3612,7 @@ module.exports.CRYPTO_ENABLED = CRYPTO_ENABLED;
  *
  * @param {?Object} data Data about this transition.
  *
- * @param {MatrixError} data.err The matrix error if <code>state=ERROR</code>.
+ * @param {MatrixError} data.error The matrix error if <code>state=ERROR</code>.
  *
  * @param {String} data.oldSyncToken The 'since' token passed to /sync.
  *    <code>null</code> for the first successful sync since this client was

src/crypto/RoomList.js

@@ -71,6 +71,9 @@ export default class RoomList {
     }
 
     async setRoomEncryption(roomId, roomInfo) {
+        // important that this happens before calling into the store
+        // as it prevents the Crypto::setRoomEncryption from calling
+        // this twice for consecutive m.room.encryption events
         this._roomEncryption[roomId] = roomInfo;
         await this._cryptoStore.doTxn(
             'readwrite', [IndexedDBCryptoStore.STORE_ROOMS], (txn) => {

src/crypto/algorithms/base.js

@@ -176,8 +176,9 @@ export {DecryptionAlgorithm}; // https://github.com/jsdoc3/jsdoc/issues/1272
  * @extends Error
  */
 class DecryptionError extends Error {
-    constructor(msg, details) {
+    constructor(code, msg, details) {
         super(msg);
+        this.code = code;
         this.name = 'DecryptionError';
         this.detailedString = _detailedStringForDecryptionError(this, details);
     }

src/crypto/algorithms/megolm.js

@@ -1,5 +1,6 @@
 /*
 Copyright 2015, 2016 OpenMarket Ltd
+Copyright 2018 New Vector Ltd
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -488,6 +489,8 @@ MegolmEncryption.prototype.encryptMessage = function(room, eventType, content) {
             session_id: session.sessionId,
              // Include our device ID so that recipients can send us a
              // m.new_device message if they don't have our session key.
+             // XXX: Do we still need this now that m.new_device messages
+             // no longer exist since #483?
             device_id: self._deviceId,
         };
 
@@ -496,6 +499,16 @@ MegolmEncryption.prototype.encryptMessage = function(room, eventType, content) {
     });
 };
 
+/**
+ * Forces the current outbound group session to be discarded such
+ * that another one will be created next time an event is sent.
+ *
+ * This should not normally be necessary.
+ */
+MegolmEncryption.prototype.forceDiscardSession = function() {
+    this._setupPromise = this._setupPromise.then(() => null);
+};
+
 /**
  * Checks the devices we're about to send to and see if any are entirely
  * unknown to the user.  If so, warn the user, and mark them as known to
@@ -535,9 +548,9 @@ MegolmEncryption.prototype._checkForUnknownDevices = function(devicesInRoom) {
  * @return {module:client.Promise} Promise which resolves to a map
  *     from userId to deviceId to deviceInfo
  */
-MegolmEncryption.prototype._getDevicesInRoom = function(room) {
-    // XXX what about rooms where invitees can see the content?
-    const roomMembers = utils.map(room.getJoinedMembers(), function(u) {
+MegolmEncryption.prototype._getDevicesInRoom = async function(room) {
+    const members = await room.getEncryptionTargetMembers();
+    const roomMembers = utils.map(members, function(u) {
         return u.userId;
     });
 
@@ -550,35 +563,31 @@ MegolmEncryption.prototype._getDevicesInRoom = function(room) {
     // We are happy to use a cached version here: we assume that if we already
     // have a list of the user's devices, then we already share an e2e room
     // with them, which means that they will have announced any new devices via
-    // an m.new_device.
-    //
-    // XXX: what if the cache is stale, and the user left the room we had in
-    // common and then added new devices before joining this one? --Matthew
-    //
-    // yup, see https://github.com/vector-im/riot-web/issues/2305 --richvdh
-    return this._crypto.downloadKeys(roomMembers, false).then((devices) => {
-        // remove any blocked devices
-        for (const userId in devices) {
-            if (!devices.hasOwnProperty(userId)) {
+    // device_lists in their /sync response.  This cache should then be maintained
+    // using all the device_lists changes and left fields.
+    // See https://github.com/vector-im/riot-web/issues/2305 for details.
+    const devices = await this._crypto.downloadKeys(roomMembers, false);
+    // remove any blocked devices
+    for (const userId in devices) {
+        if (!devices.hasOwnProperty(userId)) {
+            continue;
+        }
+
+        const userDevices = devices[userId];
+        for (const deviceId in userDevices) {
+            if (!userDevices.hasOwnProperty(deviceId)) {
                 continue;
             }
 
-            const userDevices = devices[userId];
-            for (const deviceId in userDevices) {
-                if (!userDevices.hasOwnProperty(deviceId)) {
-                    continue;
-                }
-
-                if (userDevices[deviceId].isBlocked() ||
-                    (userDevices[deviceId].isUnverified() && isBlacklisting)
-                   ) {
-                    delete userDevices[deviceId];
-                }
+            if (userDevices[deviceId].isBlocked() ||
+                (userDevices[deviceId].isUnverified() && isBlacklisting)
+               ) {
+                delete userDevices[deviceId];
             }
         }
+    }
 
-        return devices;
-    });
+    return devices;
 };
 
 /**
@@ -618,7 +627,10 @@ MegolmDecryption.prototype.decryptEvent = async function(event) {
     if (!content.sender_key || !content.session_id ||
         !content.ciphertext
        ) {
-        throw new base.DecryptionError("Missing fields in input");
+        throw new base.DecryptionError(
+            "MEGOLM_MISSING_FIELDS",
+            "Missing fields in input",
+        );
     }
 
     // we add the event to the pending list *before* we start decryption.
@@ -635,10 +647,16 @@ MegolmDecryption.prototype.decryptEvent = async function(event) {
             event.getId(), event.getTs(),
         );
     } catch (e) {
+        let errorCode = "OLM_DECRYPT_GROUP_MESSAGE_ERROR";
+
         if (e.message === 'OLM.UNKNOWN_MESSAGE_INDEX') {
             this._requestKeysForEvent(event);
+
+            errorCode = 'OLM_UNKNOWN_MESSAGE_INDEX';
         }
+
         throw new base.DecryptionError(
+            errorCode,
             e.toString(), {
                 session: content.sender_key + '|' + content.session_id,
             },
@@ -655,6 +673,7 @@ MegolmDecryption.prototype.decryptEvent = async function(event) {
         // scheduled, so we needn't send out the request here.)
         this._requestKeysForEvent(event);
         throw new base.DecryptionError(
+            "MEGOLM_UNKNOWN_INBOUND_SESSION_ID",
             "The sender's device has not sent us the keys for this message.",
             {
                 session: content.sender_key + '|' + content.session_id,
@@ -673,6 +692,7 @@ MegolmDecryption.prototype.decryptEvent = async function(event) {
     // room, so neither the sender nor a MITM can lie about the room_id).
     if (payload.room_id !== event.getRoomId()) {
         throw new base.DecryptionError(
+            "MEGOLM_BAD_ROOM",
             "Message intended for room " + payload.room_id,
         );
     }

src/crypto/algorithms/olm.js

@@ -83,60 +83,62 @@ OlmEncryption.prototype._ensureSession = function(roomMembers) {
  *
  * @return {module:client.Promise} Promise which resolves to the new event body
  */
-OlmEncryption.prototype.encryptMessage = function(room, eventType, content) {
+OlmEncryption.prototype.encryptMessage = async function(room, eventType, content) {
     // pick the list of recipients based on the membership list.
     //
     // TODO: there is a race condition here! What if a new user turns up
     // just as you are sending a secret message?
 
-    const users = utils.map(room.getJoinedMembers(), function(u) {
+    const members = await room.getEncryptionTargetMembers();
+
+    const users = utils.map(members, function(u) {
         return u.userId;
     });
 
     const self = this;
-    return this._ensureSession(users).then(function() {
-        const payloadFields = {
-            room_id: room.roomId,
-            type: eventType,
-            content: content,
-        };
+    await this._ensureSession(users);
 
-        const encryptedContent = {
-            algorithm: olmlib.OLM_ALGORITHM,
-            sender_key: self._olmDevice.deviceCurve25519Key,
-            ciphertext: {},
-        };
+    const payloadFields = {
+        room_id: room.roomId,
+        type: eventType,
+        content: content,
+    };
 
-        const promises = [];
+    const encryptedContent = {
+        algorithm: olmlib.OLM_ALGORITHM,
+        sender_key: self._olmDevice.deviceCurve25519Key,
+        ciphertext: {},
+    };
 
-        for (let i = 0; i < users.length; ++i) {
-            const userId = users[i];
-            const devices = self._crypto.getStoredDevicesForUser(userId);
+    const promises = [];
 
-            for (let j = 0; j < devices.length; ++j) {
-                const deviceInfo = devices[j];
-                const key = deviceInfo.getIdentityKey();
-                if (key == self._olmDevice.deviceCurve25519Key) {
-                    // don't bother sending to ourself
-                    continue;
-                }
-                if (deviceInfo.verified == DeviceVerification.BLOCKED) {
-                    // don't bother setting up sessions with blocked users
-                    continue;
-                }
+    for (let i = 0; i < users.length; ++i) {
+        const userId = users[i];
+        const devices = self._crypto.getStoredDevicesForUser(userId);
 
-                promises.push(
-                    olmlib.encryptMessageForDevice(
-                        encryptedContent.ciphertext,
-                        self._userId, self._deviceId, self._olmDevice,
-                        userId, deviceInfo, payloadFields,
-                    ),
-                );
+        for (let j = 0; j < devices.length; ++j) {
+            const deviceInfo = devices[j];
+            const key = deviceInfo.getIdentityKey();
+            if (key == self._olmDevice.deviceCurve25519Key) {
+                // don't bother sending to ourself
+                continue;
+            }
+            if (deviceInfo.verified == DeviceVerification.BLOCKED) {
+                // don't bother setting up sessions with blocked users
+                continue;
             }
-        }
 
-        return Promise.all(promises).return(encryptedContent);
-    });
+            promises.push(
+                olmlib.encryptMessageForDevice(
+                    encryptedContent.ciphertext,
+                    self._userId, self._deviceId, self._olmDevice,
+                    userId, deviceInfo, payloadFields,
+                ),
+            );
+        }
+    }
+
+    return await Promise.all(promises).return(encryptedContent);
 };
 
 /**
@@ -168,11 +170,17 @@ OlmDecryption.prototype.decryptEvent = async function(event) {
     const ciphertext = content.ciphertext;
 
     if (!ciphertext) {
-        throw new base.DecryptionError("Missing ciphertext");
+        throw new base.DecryptionError(
+            "OLM_MISSING_CIPHERTEXT",
+            "Missing ciphertext",
+        );
     }
 
     if (!(this._olmDevice.deviceCurve25519Key in ciphertext)) {
-        throw new base.DecryptionError("Not included in recipients");
+        throw new base.DecryptionError(
+            "OLM_NOT_INCLUDED_IN_RECIPIENTS",
+            "Not included in recipients",
+        );
     }
     const message = ciphertext[this._olmDevice.deviceCurve25519Key];
     let payloadString;
@@ -181,6 +189,7 @@ OlmDecryption.prototype.decryptEvent = async function(event) {
         payloadString = await this._decryptMessage(deviceKey, message);
     } catch (e) {
         throw new base.DecryptionError(
+            "OLM_BAD_ENCRYPTED_MESSAGE",
             "Bad Encrypted Message", {
                 sender: deviceKey,
                 err: e,
@@ -194,12 +203,14 @@ OlmDecryption.prototype.decryptEvent = async function(event) {
     // https://github.com/vector-im/vector-web/issues/2483
     if (payload.recipient != this._userId) {
         throw new base.DecryptionError(
+            "OLM_BAD_RECIPIENT",
             "Message was intented for " + payload.recipient,
         );
     }
 
     if (payload.recipient_keys.ed25519 != this._olmDevice.deviceEd25519Key) {
         throw new base.DecryptionError(
+            "OLM_BAD_RECIPIENT_KEY",
             "Message not intended for this device", {
                 intended: payload.recipient_keys.ed25519,
                 our_key: this._olmDevice.deviceEd25519Key,
@@ -213,6 +224,7 @@ OlmDecryption.prototype.decryptEvent = async function(event) {
     // which is checked elsewhere).
     if (payload.sender != event.getSender()) {
         throw new base.DecryptionError(
+            "OLM_FORWARDED_MESSAGE",
             "Message forwarded from " + payload.sender, {
                 reported_sender: event.getSender(),
             },
@@ -222,6 +234,7 @@ OlmDecryption.prototype.decryptEvent = async function(event) {
     // Olm events intended for a room have a room_id.
     if (payload.room_id !== event.getRoomId()) {
         throw new base.DecryptionError(
+            "OLM_BAD_ROOM",
             "Message intended for room " + payload.room_id, {
                 reported_room: event.room_id,
             },

src/crypto/index.js

@@ -106,6 +106,15 @@ function Crypto(baseApis, sessionStore, userId, deviceId,
     this._receivedRoomKeyRequestCancellations = [];
     // true if we are currently processing received room key requests
     this._processingRoomKeyRequests = false;
+    // controls whether device tracking is delayed
+    // until calling encryptEvent or trackRoomDevices,
+    // or done immediately upon enabling room encryption.
+    this._lazyLoadMembers = false;
+    // in case _lazyLoadMembers is true,
+    // track if an initial tracking of all the room members
+    // has happened for a given room. This is delayed
+    // to avoid loading room members as long as possible.
+    this._roomDeviceTrackingState = {};
 }
 utils.inherits(Crypto, EventEmitter);
 
@@ -167,6 +176,12 @@ Crypto.prototype.init = async function() {
     }
 };
 
+/**
+ */
+Crypto.prototype.enableLazyLoading = function() {
+    this._lazyLoadMembers = true;
+};
+
 /**
  * Tell the crypto module to register for MatrixClient events which it needs to
  * listen for
@@ -606,6 +621,23 @@ Crypto.prototype.getEventSenderDeviceInfo = function(event) {
     return device;
 };
 
+/**
+ * Forces the current outbound group session to be discarded such
+ * that another one will be created next time an event is sent.
+ *
+ * @param {string} roomId The ID of the room to discard the session for
+ *
+ * This should not normally be necessary.
+ */
+Crypto.prototype.forceDiscardSession = function(roomId) {
+    const alg = this._roomEncryptors[roomId];
+    if (alg === undefined) throw new Error("Room not encrypted");
+    if (alg.forceDiscardSession === undefined) {
+        throw new Error("Room encryption algorithm doesn't support session discarding");
+    }
+    alg.forceDiscardSession();
+};
+
 /**
  * Configure a room to use encryption (ie, save a flag in the sessionstore).
  *
@@ -614,25 +646,49 @@ Crypto.prototype.getEventSenderDeviceInfo = function(event) {
  * @param {object} config The encryption config for the room.
  *
  * @param {boolean=} inhibitDeviceQuery true to suppress device list query for
- *   users in the room (for now)
+ *   users in the room (for now). In case lazy loading is enabled,
+ *   the device query is always inhibited as the members are not tracked.
  */
 Crypto.prototype.setRoomEncryption = async function(roomId, config, inhibitDeviceQuery) {
-    // if we already have encryption in this room, we should ignore this event
-    // (for now at least. maybe we should alert the user somehow?)
+    // if state is being replayed from storage, we might already have a configuration
+    // for this room as they are persisted as well.
+    // We just need to make sure the algorithm is initialized in this case.
+    // However, if the new config is different,
+    // we should bail out as room encryption can't be changed once set.
     const existingConfig = this._roomList.getRoomEncryption(roomId);
-    if (existingConfig && JSON.stringify(existingConfig) != JSON.stringify(config)) {
-        console.error("Ignoring m.room.encryption event which requests " +
-                      "a change of config in " + roomId);
+    if (existingConfig) {
+        if (JSON.stringify(existingConfig) != JSON.stringify(config)) {
+            console.error("Ignoring m.room.encryption event which requests " +
+                          "a change of config in " + roomId);
+            return;
+        }
+    }
+    // if we already have encryption in this room, we should ignore this event,
+    // as it would reset the encryption algorithm.
+    // This is at least expected to be called twice, as sync calls onCryptoEvent
+    // for both the timeline and state sections in the /sync response,
+    // the encryption event would appear in both.
+    // If it's called more than twice though,
+    // it signals a bug on client or server.
+    const existingAlg = this._roomEncryptors[roomId];
+    if (existingAlg) {
         return;
     }
 
+    // _roomList.getRoomEncryption will not race with _roomList.setRoomEncryption
+    // because it first stores in memory. We should await the promise only
+    // after all the in-memory state (_roomEncryptors and _roomList) has been updated
+    // to avoid races when calling this method multiple times. Hence keep a hold of the promise.
+    let storeConfigPromise = null;
+    if(!existingConfig) {
+        storeConfigPromise = this._roomList.setRoomEncryption(roomId, config);
+    }
+
     const AlgClass = algorithms.ENCRYPTION_CLASSES[config.algorithm];
     if (!AlgClass) {
         throw new Error("Unable to encrypt with " + config.algorithm);
     }
 
-    await this._roomList.setRoomEncryption(roomId, config);
-
     const alg = new AlgClass({
         userId: this._userId,
         deviceId: this._deviceId,
@@ -644,24 +700,59 @@ Crypto.prototype.setRoomEncryption = async function(roomId, config, inhibitDevic
     });
     this._roomEncryptors[roomId] = alg;
 
-    // make sure we are tracking the device lists for all users in this room.
-    console.log("Enabling encryption in " + roomId + "; " +
-                "starting to track device lists for all users therein");
-    const room = this._clientStore.getRoom(roomId);
-    if (!room) {
-        throw new Error(`Unable to enable encryption in unknown room ${roomId}`);
+    if (storeConfigPromise) {
+        await storeConfigPromise;
     }
 
-    const members = room.getJoinedMembers();
-    members.forEach((m) => {
-        this._deviceList.startTrackingDeviceList(m.userId);
-    });
-    if (!inhibitDeviceQuery) {
-        this._deviceList.refreshOutdatedDeviceLists();
+    if (!this._lazyLoadMembers) {
+        console.log("Enabling encryption in " + roomId + "; " +
+            "starting to track device lists for all users therein");
+
+        await this.trackRoomDevices(roomId);
+        // TODO: this flag is only not used from MatrixClient::setRoomEncryption
+        // which is never used (inside riot at least)
+        // but didn't want to remove it as it technically would
+        // be a breaking change.
+        if(!this.inhibitDeviceQuery) {
+            this._deviceList.refreshOutdatedDeviceLists();
+        }
+    } else {
+        console.log("Enabling encryption in " + roomId);
     }
 };
 
 
+/**
+ * Make sure we are tracking the device lists for all users in this room.
+ *
+ * @param {string} roomId The room ID to start tracking devices in.
+ * @returns {Promise} when all devices for the room have been fetched and marked to track
+ */
+Crypto.prototype.trackRoomDevices = function(roomId) {
+    const trackMembers = async () => {
+        // not an encrypted room
+        if (!this._roomEncryptors[roomId]) {
+            return;
+        }
+        const room = this._clientStore.getRoom(roomId);
+        if (!room) {
+            throw new Error(`Unable to start tracking devices in unknown room ${roomId}`);
+        }
+        console.log(`Starting to track devices for room ${roomId} ...`);
+        const members = await room.getEncryptionTargetMembers();
+        members.forEach((m) => {
+            this._deviceList.startTrackingDeviceList(m.userId);
+        });
+    };
+
+    let promise = this._roomDeviceTrackingState[roomId];
+    if (!promise) {
+        promise = trackMembers();
+        this._roomDeviceTrackingState[roomId] = promise;
+    }
+    return promise;
+};
+
 /**
  * @typedef {Object} module:crypto~OlmSessionResult
  * @property {module:crypto/deviceinfo} device  device info
@@ -752,7 +843,7 @@ Crypto.prototype.importRoomKeys = function(keys) {
         },
     );
 };
-
+/* eslint-disable valid-jsdoc */    //https://github.com/eslint/eslint/issues/7307
 /**
  * Encrypt an event according to the configuration of the room.
  *
@@ -763,7 +854,8 @@ Crypto.prototype.importRoomKeys = function(keys) {
  * @return {module:client.Promise?} Promise which resolves when the event has been
  *     encrypted, or null if nothing was needed
  */
-Crypto.prototype.encryptEvent = function(event, room) {
+/* eslint-enable valid-jsdoc */
+Crypto.prototype.encryptEvent = async function(event, room) {
     if (!room) {
         throw new Error("Cannot send encrypted messages in unknown rooms");
     }
@@ -781,6 +873,12 @@ Crypto.prototype.encryptEvent = function(event, room) {
         );
     }
 
+    if (!this._roomDeviceTrackingState[roomId]) {
+        this.trackRoomDevices(roomId);
+    }
+    // wait for all the room devices to be loaded
+    await this._roomDeviceTrackingState[roomId];
+
     let content = event.getContent();
     // If event has an m.relates_to then we need
     // to put this on the wrapping event instead
@@ -791,20 +889,19 @@ Crypto.prototype.encryptEvent = function(event, room) {
         delete content['m.relates_to'];
     }
 
-    return alg.encryptMessage(
-        room, event.getType(), content,
-    ).then((encryptedContent) => {
-        if (mRelatesTo) {
-            encryptedContent['m.relates_to'] = mRelatesTo;
-        }
+    const encryptedContent = await alg.encryptMessage(
+        room, event.getType(), content);
 
-        event.makeEncrypted(
-            "m.room.encrypted",
-            encryptedContent,
-            this._olmDevice.deviceCurve25519Key,
-            this._olmDevice.deviceEd25519Key,
-        );
-    });
+    if (mRelatesTo) {
+        encryptedContent['m.relates_to'] = mRelatesTo;
+    }
+
+    event.makeEncrypted(
+        "m.room.encrypted",
+        encryptedContent,
+        this._olmDevice.deviceCurve25519Key,
+        this._olmDevice.deviceEd25519Key,
+    );
 };
 
 /**
@@ -852,7 +949,7 @@ Crypto.prototype.handleDeviceListChanges = async function(syncData, syncDeviceLi
     // If we didn't make this assumption, we'd have to use the /keys/changes API
     // to get key changes between the sync token in the device list and the 'old'
     // sync token used here to make sure we didn't miss any.
-    this._evalDeviceListChanges(syncDeviceLists);
+    await this._evalDeviceListChanges(syncDeviceLists);
 };
 
 /**
@@ -919,6 +1016,7 @@ Crypto.prototype.onSyncWillProcess = async function(syncData) {
         // at which point we'll start tracking all the users of that room.
         console.log("Initial sync performed - resetting device tracking state");
         this._deviceList.stopTrackingAllDeviceLists();
+        this._roomDeviceTrackingState = {};
     }
 };
 
@@ -964,11 +1062,12 @@ Crypto.prototype._evalDeviceListChanges = async function(deviceLists) {
         });
     }
 
-    if (deviceLists.left && Array.isArray(deviceLists.left)) {
+    if (deviceLists.left && Array.isArray(deviceLists.left) &&
+        deviceLists.left.length) {
         // Check we really don't share any rooms with these users
         // any more: the server isn't required to give us the
         // exact correct set.
-        const e2eUserIds = new Set(this._getE2eUsers());
+        const e2eUserIds = new Set(await this._getTrackedE2eUsers());
 
         deviceLists.left.forEach((u) => {
             if (!e2eUserIds.has(u)) {
@@ -980,13 +1079,14 @@ Crypto.prototype._evalDeviceListChanges = async function(deviceLists) {
 
 /**
  * Get a list of all the IDs of users we share an e2e room with
+ * for which we are tracking devices already
  *
  * @returns {string[]} List of user IDs
  */
-Crypto.prototype._getE2eUsers = function() {
+Crypto.prototype._getTrackedE2eUsers = async function() {
     const e2eUserIds = [];
-    for (const room of this._getE2eRooms()) {
-        const members = room.getJoinedMembers();
+    for (const room of this._getTrackedE2eRooms()) {
+        const members = await room.getEncryptionTargetMembers();
         for (const member of members) {
             e2eUserIds.push(member.userId);
         }
@@ -995,27 +1095,25 @@ Crypto.prototype._getE2eUsers = function() {
 };
 
 /**
- * Get a list of the e2e-enabled rooms we are members of
+ * Get a list of the e2e-enabled rooms we are members of,
+ * and for which we are already tracking the devices
  *
  * @returns {module:models.Room[]}
  */
-Crypto.prototype._getE2eRooms = function() {
+Crypto.prototype._getTrackedE2eRooms = function() {
     return this._clientStore.getRooms().filter((room) => {
         // check for rooms with encryption enabled
         const alg = this._roomEncryptors[room.roomId];
         if (!alg) {
             return false;
         }
-
-        // ignore any rooms which we have left
-        const me = room.getMember(this._userId);
-        if (!me || (
-            me.membership !== "join" && me.membership !== "invite"
-        )) {
+        if (!this._roomDeviceTrackingState[room.roomId]) {
             return false;
         }
 
-        return true;
+        // ignore any rooms which we have left
+        const myMembership = room.getMyMembership();
+        return myMembership === "join" || myMembership === "invite";
     });
 };
 
@@ -1080,11 +1178,20 @@ Crypto.prototype._onRoomMembership = function(event, member, oldMembership) {
         // not encrypting in this room
         return;
     }
-
-    if (member.membership == 'join') {
-        console.log('Join event for ' + member.userId + ' in ' + roomId);
-        // make sure we are tracking the deviceList for this user
-        this._deviceList.startTrackingDeviceList(member.userId);
+    // only mark users in this room as tracked if we already started tracking in this room
+    // this way we don't start device queries after sync on behalf of this room which we won't use
+    // the result of anyway, as we'll need to do a query again once all the members are fetched
+    // by calling _trackRoomDevices
+    if (this._roomDeviceTrackingState[roomId]) {
+        if (member.membership == 'join') {
+            console.log('Join event for ' + member.userId + ' in ' + roomId);
+            // make sure we are tracking the deviceList for this user
+            this._deviceList.startTrackingDeviceList(member.userId);
+        } else if (member.membership == 'invite' &&
+                 this._clientStore.getRoom(roomId).shouldEncryptForInvitedMembers()) {
+            console.log('Invite event for ' + member.userId + ' in ' + roomId);
+            this._deviceList.startTrackingDeviceList(member.userId);
+        }
     }
 
     alg.onRoomMembership(event, member, oldMembership);
@@ -1275,6 +1382,7 @@ Crypto.prototype._getRoomDecryptor = function(roomId, algorithm) {
     const AlgClass = algorithms.DECRYPTION_CLASSES[algorithm];
     if (!AlgClass) {
         throw new algorithms.DecryptionError(
+            'UNKNOWN_ENCRYPTION_ALGORITHM',
             'Unknown encryption algorithm "' + algorithm + '".',
         );
     }

src/models/event-timeline-set.js

@@ -168,49 +168,19 @@ EventTimelineSet.prototype.resetLiveTimeline = function(
     // if timeline support is disabled, forget about the old timelines
     const resetAllTimelines = !this._timelineSupport || !forwardPaginationToken;
 
-    let newTimeline;
+    const oldTimeline = this._liveTimeline;
+    const newTimeline = resetAllTimelines ?
+        oldTimeline.forkLive(EventTimeline.FORWARDS) :
+        oldTimeline.fork(EventTimeline.FORWARDS);
+
     if (resetAllTimelines) {
-        newTimeline = new EventTimeline(this);
         this._timelines = [newTimeline];
         this._eventIdToTimeline = {};
     } else {
-        newTimeline = this.addTimeline();
+        this._timelines.push(newTimeline);
     }
 
-    const oldTimeline = this._liveTimeline;
-
-    // Collect the state events from the old timeline
-    const evMap = oldTimeline.getState(EventTimeline.FORWARDS).events;
-    const events = [];
-    for (const evtype in evMap) {
-        if (!evMap.hasOwnProperty(evtype)) {
-            continue;
-        }
-        for (const stateKey in evMap[evtype]) {
-            if (!evMap[evtype].hasOwnProperty(stateKey)) {
-                continue;
-            }
-            events.push(evMap[evtype][stateKey]);
-        }
-    }
-
-    // Use those events to initialise the state of the new live timeline
-    newTimeline.initialiseState(events);
-
-    const freshEndState = newTimeline._endState;
-    // Now clobber the end state of the new live timeline with that from the
-    // previous live timeline. It will be identical except that we'll keep
-    // using the same RoomMember objects for the 'live' set of members with any
-    // listeners still attached
-    newTimeline._endState = oldTimeline._endState;
-
-    // If we're not resetting all timelines, we need to fix up the old live timeline
-    if (!resetAllTimelines) {
-        // Firstly, we just stole the old timeline's end state, so it needs a new one.
-        // Just swap them around and give it the one we just generated for the
-        // new live timeline.
-        oldTimeline._endState = freshEndState;
-
+    if (forwardPaginationToken) {
         // Now set the forward pagination token on the old live timeline
         // so it can be forward-paginated.
         oldTimeline.setPaginationToken(

src/models/event-timeline.js

@@ -106,6 +106,50 @@ EventTimeline.prototype.initialiseState = function(stateEvents) {
     this._endState.setStateEvents(stateEvents);
 };
 
+/**
+ * Forks the (live) timeline, taking ownership of the existing directional state of this timeline.
+ * All attached listeners will keep receiving state updates from the new live timeline state.
+ * The end state of this timeline gets replaced with an independent copy of the current RoomState,
+ * and will need a new pagination token if it ever needs to paginate forwards.
+
+ * @param {string} direction   EventTimeline.BACKWARDS to get the state at the
+ *   start of the timeline; EventTimeline.FORWARDS to get the state at the end
+ *   of the timeline.
+ *
+ * @return {EventTimeline} the new timeline
+ */
+EventTimeline.prototype.forkLive = function(direction) {
+    const forkState = this.getState(direction);
+    const timeline = new EventTimeline(this._eventTimelineSet);
+    timeline._startState = forkState.clone();
+    // Now clobber the end state of the new live timeline with that from the
+    // previous live timeline. It will be identical except that we'll keep
+    // using the same RoomMember objects for the 'live' set of members with any
+    // listeners still attached
+    timeline._endState = forkState;
+    // Firstly, we just stole the current timeline's end state, so it needs a new one.
+    // Make an immutable copy of the state so back pagination will get the correct sentinels.
+    this._endState = forkState.clone();
+    return timeline;
+};
+
+/**
+ * Creates an independent timeline, inheriting the directional state from this timeline.
+ *
+ * @param {string} direction   EventTimeline.BACKWARDS to get the state at the
+ *   start of the timeline; EventTimeline.FORWARDS to get the state at the end
+ *   of the timeline.
+ *
+ * @return {EventTimeline} the new timeline
+ */
+EventTimeline.prototype.fork = function(direction) {
+    const forkState = this.getState(direction);
+    const timeline = new EventTimeline(this._eventTimelineSet);
+    timeline._startState = forkState.clone();
+    timeline._endState = forkState.clone();
+    return timeline;
+};
+
 /**
  * Get the ID of the room for this timeline
  * @return {string} room ID

src/models/event.js

@@ -404,6 +404,7 @@ utils.extend(module.exports.MatrixEvent.prototype, {
             this._retryDecryption = false;
 
             let res;
+            let err;
             try {
                 if (!crypto) {
                     res = this._badEncryptedMessage("Encryption not enabled");
@@ -422,6 +423,8 @@ utils.extend(module.exports.MatrixEvent.prototype, {
                     return;
                 }
 
+                err = e;
+
                 // see if we have a retry queued.
                 //
                 // NB: make sure to keep this check in the same tick of the
@@ -467,6 +470,9 @@ utils.extend(module.exports.MatrixEvent.prototype, {
             this._decryptionPromise = null;
             this._retryDecryption = false;
             this._setClearData(res);
+
+            this.emit("Event.decrypted", this, err);
+
             return;
         }
     },
@@ -503,7 +509,6 @@ utils.extend(module.exports.MatrixEvent.prototype, {
             decryptionResult.claimedEd25519Key || null;
         this._forwardingCurve25519KeyChain =
             decryptionResult.forwardingCurve25519KeyChain || [];
-        this.emit("Event.decrypted", this);
     },
 
     /**
@@ -708,4 +713,7 @@ const _REDACT_KEEP_CONTENT_MAP = {
  *
  * @param {module:models/event.MatrixEvent} event
  *    The matrix event which has been decrypted
+ * @param {module:crypto/algorithms/base.DecryptionError?} err
+ *    The error that occured during decryption, or `undefined` if no
+ *    error occured.
  */

src/models/room-member.js

@@ -58,10 +58,53 @@ function RoomMember(roomId, userId) {
     this.events = {
         member: null,
     };
+    this._isOutOfBand = false;
+    this._supersedesOutOfBand = false;
     this._updateModifiedTime();
 }
 utils.inherits(RoomMember, EventEmitter);
 
+/**
+ * Mark the member as coming from a channel that is not sync
+ */
+RoomMember.prototype.markOutOfBand = function() {
+    this._isOutOfBand = true;
+};
+
+/**
+ * @return {bool} does the member come from a channel that is not sync?
+ * This is used to store the member seperately
+ * from the sync state so it available across browser sessions.
+ */
+RoomMember.prototype.isOutOfBand = function() {
+    return this._isOutOfBand;
+};
+
+/**
+ * Does the member supersede an incoming out-of-band
+ * member? If so the out-of-band member should be ignored.
+ * @return {bool}
+ */
+RoomMember.prototype.supersedesOutOfBand = function() {
+    return this._supersedesOutOfBand;
+};
+
+/**
+ * Mark the member as superseding the future incoming
+ * out-of-band members.
+ */
+RoomMember.prototype.markSupersedesOutOfBand = function() {
+    this._supersedesOutOfBand = true;
+};
+
+/**
+ * Clear the member superseding the future incoming
+ * out-of-band members, as loading finished or failed.
+ */
+RoomMember.prototype.clearSupersedesOutOfBand = function() {
+    this._supersedesOutOfBand = false;
+};
+
 /**
  * Update this room member's membership event. May fire "RoomMember.name" if
  * this event updates this member's name.
@@ -75,13 +118,20 @@ RoomMember.prototype.setMembershipEvent = function(event, roomState) {
     if (event.getType() !== "m.room.member") {
         return;
     }
+
+    this._isOutOfBand = false;
+
     this.events.member = event;
 
     const oldMembership = this.membership;
     this.membership = event.getDirectionalContent().membership;
 
     const oldName = this.name;
-    this.name = calculateDisplayName(this, event, roomState);
+    this.name = calculateDisplayName(
+        this.userId,
+        event.getDirectionalContent().displayname,
+        roomState);
+
     this.rawDisplayName = event.getDirectionalContent().displayname || this.userId;
     if (oldMembership !== this.membership) {
         this._updateModifiedTime();
@@ -177,6 +227,44 @@ RoomMember.prototype.getLastModifiedTime = function() {
     return this._modified;
 };
 
+
+RoomMember.prototype.isKicked = function() {
+    return this.membership === "leave" &&
+        this.events.member.getSender() !== this.events.member.getStateKey();
+};
+
+/**
+ * If this member was invited with the is_direct flag set, return
+ * the user that invited this member
+ * @return {string} user id of the inviter
+ */
+RoomMember.prototype.getDMInviter = function() {
+    // when not available because that room state hasn't been loaded in,
+    // we don't really know, but more likely to not be a direct chat
+    if (this.events.member) {
+        // TODO: persist the is_direct flag on the member as more member events
+        //       come in caused by displayName changes.
+
+        // the is_direct flag is set on the invite member event.
+        // This is copied on the prev_content section of the join member event
+        // when the invite is accepted.
+
+        const memberEvent = this.events.member;
+        let memberContent = memberEvent.getContent();
+        let inviteSender = memberEvent.getSender();
+
+        if (memberContent.membership === "join") {
+            memberContent = memberEvent.getPrevContent();
+            inviteSender = memberEvent.getUnsigned().prev_sender;
+        }
+
+        if (memberContent.membership === "invite" && memberContent.is_direct) {
+            return inviteSender;
+        }
+    }
+};
+
+
 /**
  * Get the avatar URL for a room member.
  * @param {string} baseUrl The base homeserver URL See
@@ -200,10 +288,12 @@ RoomMember.prototype.getAvatarUrl =
     if (allowDefault === undefined) {
         allowDefault = true;
     }
-    if (!this.events.member && !allowDefault) {
+
+    const rawUrl = this.getMxcAvatarUrl();
+
+    if (!rawUrl && !allowDefault) {
         return null;
     }
-    const rawUrl = this.events.member ? this.events.member.getContent().avatar_url : null;
     const httpUrl = ContentRepo.getHttpUriForMxc(
         baseUrl, rawUrl, width, height, resizeMethod, allowDirectLinks,
     );
@@ -216,12 +306,21 @@ RoomMember.prototype.getAvatarUrl =
     }
     return null;
 };
+/**
+ * get the mxc avatar url, either from a state event, or from a lazily loaded member
+ * @return {string} the mxc avatar url
+ */
+RoomMember.prototype.getMxcAvatarUrl = function() {
+    if(this.events.member) {
+        return this.events.member.getDirectionalContent().avatar_url;
+    } else if(this.user) {
+        return this.user.avatarUrl;
+    }
+    return null;
+};
 
-function calculateDisplayName(member, event, roomState) {
-    const displayName = event.getDirectionalContent().displayname;
-    const selfUserId = member.userId;
-
-    if (!displayName) {
+function calculateDisplayName(selfUserId, displayName, roomState) {
+    if (!displayName || displayName === selfUserId) {
         return selfUserId;
     }
 
@@ -229,18 +328,23 @@ function calculateDisplayName(member, event, roomState) {
         return displayName;
     }
 
+    // First check if the displayname is something we consider truthy
+    // after stripping it of zero width characters and padding spaces
+    const strippedDisplayName = utils.removeHiddenChars(displayName);
+    if (!strippedDisplayName) {
+        return selfUserId;
+    }
 
-    // Check if the name contains something that look like a mxid
+    // Next check if the name contains something that look like a mxid
     // If it does, it may be someone trying to impersonate someone else
     // Show full mxid in this case
     // Also show mxid if there are other people with the same displayname
+    // ignoring any zero width chars (unicode 200B-200D)
+    // if their displayname is made up of just zero width chars, show full mxid
     let disambiguate = /@.+:.+/.test(displayName);
     if (!disambiguate) {
-        const userIds = roomState.getUserIdsWithDisplayName(displayName);
-        const otherUsers = userIds.filter(function(u) {
-            return u !== selfUserId;
-        });
-        disambiguate = otherUsers.length > 0;
+        const userIds = roomState.getUserIdsWithDisplayName(strippedDisplayName);
+        disambiguate = userIds.some((u) => u !== selfUserId);
     }
 
     if (disambiguate) {

src/models/room-state.js

@@ -22,6 +22,11 @@ const EventEmitter = require("events").EventEmitter;
 const utils = require("../utils");
 const RoomMember = require("./room-member");
 
+// possible statuses for out-of-band member loading
+const OOB_STATUS_NOTSTARTED = 1;
+const OOB_STATUS_INPROGRESS = 2;
+const OOB_STATUS_FINISHED = 3;
+
 /**
  * Construct room state.
  *
@@ -46,13 +51,17 @@ const RoomMember = require("./room-member");
  * @constructor
  * @param {?string} roomId Optional. The ID of the room which has this state.
  * If none is specified it just tracks paginationTokens, useful for notifTimelineSet
+ * @param {?object} oobMemberFlags Optional. The state of loading out of bound members.
+ * As the timeline might get reset while they are loading, this state needs to be inherited
+ * and shared when the room state is cloned for the new timeline.
+ * This should only be passed from clone.
  * @prop {Object.<string, RoomMember>} members The room member dictionary, keyed
  * on the user's ID.
  * @prop {Object.<string, Object.<string, MatrixEvent>>} events The state
  * events dictionary, keyed on the event type and then the state_key value.
  * @prop {string} paginationToken The pagination token for this state.
  */
-function RoomState(roomId) {
+function RoomState(roomId, oobMemberFlags = undefined) {
     this.roomId = roomId;
     this.members = {
         // userId: RoomMember
@@ -70,6 +79,22 @@ function RoomState(roomId) {
     this._userIdsToDisplayNames = {};
     this._tokenToInvite = {}; // 3pid invite state_key to m.room.member invite
     this._joinedMemberCount = null; // cache of the number of joined members
+    // joined members count from summary api
+    // once set, we know the server supports the summary api
+    // and we should only trust that
+    // we could also only trust that before OOB members
+    // are loaded but doesn't seem worth the hassle atm
+    this._summaryJoinedMemberCount = null;
+    // same for invited member count
+    this._invitedMemberCount = null;
+    this._summaryInvitedMemberCount = null;
+
+    if (!oobMemberFlags) {
+        oobMemberFlags = {
+            status: OOB_STATUS_NOTSTARTED,
+        };
+    }
+    this._oobMemberFlags = oobMemberFlags;
 }
 utils.inherits(RoomState, EventEmitter);
 
@@ -79,14 +104,48 @@ utils.inherits(RoomState, EventEmitter);
  * @return {integer} The number of members in this room whose membership is 'join'
  */
 RoomState.prototype.getJoinedMemberCount = function() {
+    if (this._summaryJoinedMemberCount !== null) {
+        return this._summaryJoinedMemberCount;
+    }
     if (this._joinedMemberCount === null) {
-        this._joinedMemberCount = this.getMembers().filter((m) => {
-            return m.membership === 'join';
-        }).length;
+        this._joinedMemberCount = this.getMembers().reduce((count, m) => {
+            return m.membership === 'join' ? count + 1 : count;
+        }, 0);
     }
     return this._joinedMemberCount;
 };
 
+/**
+ * Set the joined member count explicitly (like from summary part of the sync response)
+ * @param {number} count the amount of joined members
+ */
+RoomState.prototype.setJoinedMemberCount = function(count) {
+    this._summaryJoinedMemberCount = count;
+};
+/**
+ * Returns the number of invited members in this room
+ * @return {integer} The number of members in this room whose membership is 'invite'
+ */
+RoomState.prototype.getInvitedMemberCount = function() {
+    if (this._summaryInvitedMemberCount !== null) {
+        return this._summaryInvitedMemberCount;
+    }
+    if (this._invitedMemberCount === null) {
+        this._invitedMemberCount = this.getMembers().reduce((count, m) => {
+            return m.membership === 'invite' ? count + 1 : count;
+        }, 0);
+    }
+    return this._invitedMemberCount;
+};
+
+/**
+ * Set the amount of invited members in this room
+ * @param {number} count the amount of invited members
+ */
+RoomState.prototype.setInvitedMemberCount = function(count) {
+    this._summaryInvitedMemberCount = count;
+};
+
 /**
  * Get all RoomMembers in this room.
  * @return {Array<RoomMember>} A list of RoomMembers.
@@ -119,12 +178,9 @@ RoomState.prototype.getSentinelMember = function(userId) {
 
     if (sentinel === undefined) {
         sentinel = new RoomMember(this.roomId, userId);
-        const membershipEvent = this.getStateEvents("m.room.member", userId);
-        if (!membershipEvent) return null;
-        sentinel.setMembershipEvent(membershipEvent, this);
-        const pwrLvlEvent = this.getStateEvents("m.room.power_levels", "");
-        if (pwrLvlEvent) {
-            sentinel.setPowerLevelEvent(pwrLvlEvent);
+        const member = this.members[userId];
+        if (member) {
+            sentinel.setMembershipEvent(member.events.member);
         }
         this._sentinels[userId] = sentinel;
     }
@@ -152,6 +208,76 @@ RoomState.prototype.getStateEvents = function(eventType, stateKey) {
     return event ? event : null;
 };
 
+/**
+ * Creates a copy of this room state so that mutations to either won't affect the other.
+ * @return {RoomState} the copy of the room state
+ */
+RoomState.prototype.clone = function() {
+    const copy = new RoomState(this.roomId, this._oobMemberFlags);
+
+    // Ugly hack: because setStateEvents will mark
+    // members as susperseding future out of bound members
+    // if loading is in progress (through _oobMemberFlags)
+    // since these are not new members, we're merely copying them
+    // set the status to not started
+    // after copying, we set back the status and
+    // copy the superseding flag from the current state
+    const status = this._oobMemberFlags.status;
+    this._oobMemberFlags.status = OOB_STATUS_NOTSTARTED;
+
+    Object.values(this.events).forEach((eventsByStateKey) => {
+        const eventsForType = Object.values(eventsByStateKey);
+        copy.setStateEvents(eventsForType);
+    });
+
+    // Ugly hack: see above
+    this._oobMemberFlags.status = status;
+
+    if (this._summaryInvitedMemberCount !== null) {
+        copy.setInvitedMemberCount(this.getInvitedMemberCount());
+    }
+    if (this._summaryJoinedMemberCount !== null) {
+        copy.setJoinedMemberCount(this.getJoinedMemberCount());
+    }
+
+    // copy out of band flags if needed
+    if (this._oobMemberFlags.status == OOB_STATUS_FINISHED) {
+        // copy markOutOfBand flags
+        this.getMembers().forEach((member) => {
+            if (member.isOutOfBand()) {
+                const copyMember = copy.getMember(member.userId);
+                copyMember.markOutOfBand();
+            }
+        });
+    } else if (this._oobMemberFlags.status == OOB_STATUS_INPROGRESS) {
+        // copy markSupersedesOutOfBand flags
+        this.getMembers().forEach((member) => {
+            if (member.supersedesOutOfBand()) {
+                const copyMember = copy.getMember(member.userId);
+                copyMember.markSupersedesOutOfBand();
+            }
+        });
+    }
+
+    return copy;
+};
+
+/**
+ * Add previously unknown state events.
+ * When lazy loading members while back-paginating,
+ * the relevant room state for the timeline chunk at the end
+ * of the chunk can be set with this method.
+ * @param {MatrixEvent[]} events state events to prepend
+ */
+RoomState.prototype.setUnknownStateEvents = function(events) {
+    const unknownStateEvents = events.filter((event) => {
+        return this.events[event.getType()] === undefined ||
+            this.events[event.getType()][event.getStateKey()] === undefined;
+    });
+
+    this.setStateEvents(unknownStateEvents);
+};
+
 /**
  * Add an array of one or more state MatrixEvents, overwriting
  * any existing state with the same {type, stateKey} tuple. Will fire
@@ -175,10 +301,7 @@ RoomState.prototype.setStateEvents = function(stateEvents) {
             return;
         }
 
-        if (self.events[event.getType()] === undefined) {
-            self.events[event.getType()] = {};
-        }
-        self.events[event.getType()][event.getStateKey()] = event;
+        self._setStateEvent(event);
         if (event.getType() === "m.room.member") {
             _updateDisplayNameCache(
                 self, event.getStateKey(), event.getContent().displayname,
@@ -216,24 +339,15 @@ RoomState.prototype.setStateEvents = function(stateEvents) {
                     event.getPrevContent().displayname;
             }
 
-            let member = self.members[userId];
-            if (!member) {
-                member = new RoomMember(event.getRoomId(), userId);
-                self.emit("RoomState.newMember", event, self, member);
-            }
-
+            const member = self._getOrCreateMember(userId, event);
             member.setMembershipEvent(event, self);
-            // this member may have a power level already, so set it.
-            const pwrLvlEvent = self.getStateEvents("m.room.power_levels", "");
-            if (pwrLvlEvent) {
-                member.setPowerLevelEvent(pwrLvlEvent);
+            // if out of band members are loading,
+            // mark the member as more recent
+            if (self._oobMemberFlags.status == OOB_STATUS_INPROGRESS) {
+                member.markSupersedesOutOfBand();
             }
 
-            // blow away the sentinel which is now outdated
-            delete self._sentinels[userId];
-
-            self.members[userId] = member;
-            self._joinedMemberCount = null;
+            self._updateMember(member);
             self.emit("RoomState.members", event, self, member);
         } else if (event.getType() === "m.room.power_levels") {
             const members = utils.values(self.members);
@@ -248,6 +362,158 @@ RoomState.prototype.setStateEvents = function(stateEvents) {
     });
 };
 
+/**
+ * Looks up a member by the given userId, and if it doesn't exist,
+ * create it and emit the `RoomState.newMember` event.
+ * This method makes sure the member is added to the members dictionary
+ * before emitting, as this is done from setStateEvents and _setOutOfBandMember.
+ * @param {string} userId the id of the user to look up
+ * @param {MatrixEvent} event the membership event for the (new) member. Used to emit.
+ * @fires module:client~MatrixClient#event:"RoomState.newMember"
+ * @returns {RoomMember} the member, existing or newly created.
+ */
+RoomState.prototype._getOrCreateMember = function(userId, event) {
+    let member = this.members[userId];
+    if (!member) {
+        member = new RoomMember(this.roomId, userId);
+        // add member to members before emitting any events,
+        // as event handlers often lookup the member
+        this.members[userId] = member;
+        this.emit("RoomState.newMember", event, this, member);
+    }
+    return member;
+};
+
+RoomState.prototype._setStateEvent = function(event) {
+    if (this.events[event.getType()] === undefined) {
+        this.events[event.getType()] = {};
+    }
+    this.events[event.getType()][event.getStateKey()] = event;
+};
+
+RoomState.prototype._updateMember = function(member) {
+    // this member may have a power level already, so set it.
+    const pwrLvlEvent = this.getStateEvents("m.room.power_levels", "");
+    if (pwrLvlEvent) {
+        member.setPowerLevelEvent(pwrLvlEvent);
+    }
+
+    // blow away the sentinel which is now outdated
+    delete this._sentinels[member.userId];
+
+    this.members[member.userId] = member;
+    this._joinedMemberCount = null;
+    this._invitedMemberCount = null;
+};
+
+/**
+ * Get the out-of-band members loading state, whether loading is needed or not.
+ * Note that loading might be in progress and hence isn't needed.
+ * @return {bool} whether or not the members of this room need to be loaded
+ */
+RoomState.prototype.needsOutOfBandMembers = function() {
+    return this._oobMemberFlags.status === OOB_STATUS_NOTSTARTED;
+};
+
+/**
+ * Mark this room state as waiting for out-of-band members,
+ * ensuring it doesn't ask for them to be requested again
+ * through needsOutOfBandMembers
+ */
+RoomState.prototype.markOutOfBandMembersStarted = function() {
+    if (this._oobMemberFlags.status !== OOB_STATUS_NOTSTARTED) {
+        return;
+    }
+    this._oobMemberFlags.status = OOB_STATUS_INPROGRESS;
+};
+
+/**
+ * Mark this room state as having failed to fetch out-of-band members
+ */
+RoomState.prototype.markOutOfBandMembersFailed = function() {
+    if (this._oobMemberFlags.status !== OOB_STATUS_INPROGRESS) {
+        return;
+    }
+    // the request failed, there is nothing to supersede
+    // in case of a retry, these event would not supersede the
+    // retry anymore.
+    this.getMembers().forEach((m) => {
+        m.clearSupersedesOutOfBand();
+    });
+    this._oobMemberFlags.status = OOB_STATUS_NOTSTARTED;
+};
+
+/**
+ * Clears the loaded out-of-band members
+ */
+RoomState.prototype.clearOutOfBandMembers = function() {
+    let count = 0;
+    Object.keys(this.members).forEach((userId) => {
+        const member = this.members[userId];
+        if (member.isOutOfBand()) {
+            ++count;
+            delete this.members[userId];
+        }
+    });
+    console.log(`LL: RoomState removed ${count} members...`);
+    this._oobMemberFlags.status = OOB_STATUS_NOTSTARTED;
+};
+
+/**
+ * Sets the loaded out-of-band members.
+ * @param {MatrixEvent[]} stateEvents array of membership state events
+ */
+RoomState.prototype.setOutOfBandMembers = function(stateEvents) {
+    console.log(`LL: RoomState about to set ${stateEvents.length} OOB members ...`);
+    if (this._oobMemberFlags.status !== OOB_STATUS_INPROGRESS) {
+        return;
+    }
+    console.log(`LL: RoomState put in OOB_STATUS_FINISHED state ...`);
+    this._oobMemberFlags.status = OOB_STATUS_FINISHED;
+    stateEvents.forEach((e) => this._setOutOfBandMember(e));
+};
+
+/**
+ * Sets a single out of band member, used by both setOutOfBandMembers and clone
+ * @param {MatrixEvent} stateEvent membership state event
+ */
+RoomState.prototype._setOutOfBandMember = function(stateEvent) {
+    if (stateEvent.getType() !== 'm.room.member') {
+        return;
+    }
+    const userId = stateEvent.getStateKey();
+    const existingMember = this.getMember(userId);
+    if (existingMember) {
+        const existingMemberEvent = existingMember.events.member;
+        // ignore out of band members with events we are
+        // already aware of.
+        if (existingMemberEvent.getId() === stateEvent.getId()) {
+            return;
+        }
+        // this member was updated since we started
+        // loading the out of band members.
+        // Ignore the out of band member and clear
+        // the "supersedes" flag as the out of members are now loaded
+        if (existingMember.supersedesOutOfBand()) {
+            existingMember.clearSupersedesOutOfBand();
+            return;
+        }
+    }
+
+    const member = this._getOrCreateMember(userId, stateEvent);
+    member.setMembershipEvent(stateEvent);
+    // needed to know which members need to be stored seperately
+    // as the are not part of the sync accumulator
+    // this is cleared by setMembershipEvent so when it's updated through /sync
+    member.markOutOfBand();
+
+    _updateDisplayNameCache(this, member.userId, member.name);
+
+    this._setStateEvent(stateEvent);
+    this._updateMember(member);
+    this.emit("RoomState.members", {}, stateEvent, member);
+};
+
 /**
  * Set the current typing event for this room.
  * @param {MatrixEvent} event The typing event
@@ -401,11 +667,6 @@ RoomState.prototype.maySendStateEvent = function(stateEventType, userId) {
  *                        according to the room's state.
  */
 RoomState.prototype._maySendEventOfType = function(eventType, userId, state) {
-    const member = this.getMember(userId);
-    if (!member || member.membership == 'leave') {
-        return false;
-    }
-
     const power_levels_event = this.getStateEvents('m.room.power_levels', '');
 
     let power_levels;
@@ -413,25 +674,34 @@ RoomState.prototype._maySendEventOfType = function(eventType, userId, state) {
 
     let state_default = 0;
     let events_default = 0;
+    let powerLevel = 0;
     if (power_levels_event) {
         power_levels = power_levels_event.getContent();
         events_levels = power_levels.events || {};
 
-        if (utils.isNumber(power_levels.state_default)) {
+        if (Number.isFinite(power_levels.state_default)) {
             state_default = power_levels.state_default;
         } else {
             state_default = 50;
         }
-        if (utils.isNumber(power_levels.events_default)) {
+
+        const userPowerLevel = power_levels.users && power_levels.users[userId];
+        if (Number.isFinite(userPowerLevel)) {
+            powerLevel = userPowerLevel;
+        } else if(Number.isFinite(power_levels.users_default)) {
+            powerLevel = power_levels.users_default;
+        }
+
+        if (Number.isFinite(power_levels.events_default)) {
             events_default = power_levels.events_default;
         }
     }
 
     let required_level = state ? state_default : events_default;
-    if (utils.isNumber(events_levels[eventType])) {
+    if (Number.isFinite(events_levels[eventType])) {
         required_level = events_levels[eventType];
     }
-    return member.powerLevel >= required_level;
+    return powerLevel >= required_level;
 };
 
 /**
@@ -494,22 +764,26 @@ function _updateDisplayNameCache(roomState, userId, displayName) {
         // We clobber the user_id > name lookup but the name -> [user_id] lookup
         // means we need to remove that user ID from that array rather than nuking
         // the lot.
-        const existingUserIds = roomState._displayNameToUserIds[oldName] || [];
-        for (let i = 0; i < existingUserIds.length; i++) {
-            if (existingUserIds[i] === userId) {
-                // remove this user ID from this array
-                existingUserIds.splice(i, 1);
-                i--;
-            }
+        const strippedOldName = utils.removeHiddenChars(oldName);
+
+        const existingUserIds = roomState._displayNameToUserIds[strippedOldName];
+        if (existingUserIds) {
+            // remove this user ID from this array
+            const filteredUserIDs = existingUserIds.filter((id) => id !== userId);
+            roomState._displayNameToUserIds[strippedOldName] = filteredUserIDs;
         }
-        roomState._displayNameToUserIds[oldName] = existingUserIds;
     }
 
     roomState._userIdsToDisplayNames[userId] = displayName;
-    if (!roomState._displayNameToUserIds[displayName]) {
-        roomState._displayNameToUserIds[displayName] = [];
+
+    const strippedDisplayname = displayName && utils.removeHiddenChars(displayName);
+    // an empty stripped displayname (undefined/'') will be set to MXID in room-member.js
+    if (strippedDisplayname) {
+        if (!roomState._displayNameToUserIds[strippedDisplayname]) {
+            roomState._displayNameToUserIds[strippedDisplayname] = [];
+        }
+        roomState._displayNameToUserIds[strippedDisplayname].push(userId);
     }
-    roomState._displayNameToUserIds[displayName].push(userId);
 }
 
 /**
@@ -539,7 +813,8 @@ function _updateDisplayNameCache(roomState, userId, displayName) {
 
  /**
  * Fires whenever a member is added to the members dictionary. The RoomMember
- * will not be fully populated yet (e.g. no membership state).
+ * will not be fully populated yet (e.g. no membership state) but will already
+ * be available in the members dictionary.
  * @event module:client~MatrixClient#"RoomState.newMember"
  * @param {MatrixEvent} event The matrix event which caused this event to fire.
  * @param {RoomState} state The room state whose RoomState.members dictionary

src/models/room.js

@@ -1,5 +1,6 @@
 /*
 Copyright 2015, 2016 OpenMarket Ltd
+Copyright 2018 New Vector Ltd
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -21,6 +22,7 @@ const EventEmitter = require("events").EventEmitter;
 
 const EventStatus = require("./event").EventStatus;
 const RoomSummary = require("./room-summary");
+const RoomMember = require("./room-member");
 const MatrixEvent = require("./event").MatrixEvent;
 const utils = require("../utils");
 const ContentRepo = require("../content-repo");
@@ -29,6 +31,8 @@ const EventTimelineSet = require("./event-timeline-set");
 
 import ReEmitter from '../ReEmitter';
 
+const LATEST_ROOM_VERSION = '1';
+
 function synthesizeReceipt(userId, event, receiptType) {
     // console.log("synthesizing receipt for "+event.getId());
     // This is really ugly because JS has no way to express an object literal
@@ -68,6 +72,8 @@ function synthesizeReceipt(userId, event, receiptType) {
  * @constructor
  * @alias module:models/room
  * @param {string} roomId Required. The ID of this room.
+ * @param {MatrixClient} client Required. The client, used to lazy load members.
+ * @param {string} myUserId Required. The ID of the syncing user.
  * @param {Object=} opts Configuration options
  * @param {*} opts.storageToken Optional. The token which a data store can use
  * to remember the state of the room. What this means is dependent on the store
@@ -102,7 +108,7 @@ function synthesizeReceipt(userId, event, receiptType) {
  * @prop {*} storageToken A token which a data store can use to remember
  * the state of the room.
  */
-function Room(roomId, opts) {
+function Room(roomId, client, myUserId, opts) {
     opts = opts || {};
     opts.pendingEventOrdering = opts.pendingEventOrdering || "chronological";
 
@@ -115,6 +121,7 @@ function Room(roomId, opts) {
         );
     }
 
+    this.myUserId = myUserId;
     this.roomId = roomId;
     this.name = roomId;
     this.tags = {
@@ -171,9 +178,53 @@ function Room(roomId, opts) {
 
     // read by megolm; boolean value - null indicates "use global value"
     this._blacklistUnverifiedDevices = null;
+    this._syncedMembership = null;
+    this._summaryHeroes = null;
+    // awaited by getEncryptionTargetMembers while room members are loading
+
+    this._client = client;
+    if (!this._opts.lazyLoadMembers) {
+        this._membersPromise = Promise.resolve();
+    } else {
+        this._membersPromise = null;
+    }
 }
+
 utils.inherits(Room, EventEmitter);
 
+/**
+ * Gets the version of the room
+ * @returns {string} The version of the room, or null if it could not be determined
+ */
+Room.prototype.getVersion = function() {
+    const createEvent = this.currentState.getStateEvents("m.room.create", "");
+    if (!createEvent) return null;
+    const ver = createEvent.getContent()['room_version'];
+    if (ver === undefined) return '1';
+    return ver;
+};
+
+/**
+ * Determines whether this room needs to be upgraded to a new version
+ * @returns {string?} What version the room should be upgraded to, or null if
+ *     the room does not require upgrading at this time.
+ */
+Room.prototype.shouldUpgradeToVersion = function() {
+    // This almost certainly won't be the way this actually works - this
+    // is essentially a stub method.
+    if (this.getVersion() === LATEST_ROOM_VERSION) return null;
+    return LATEST_ROOM_VERSION;
+};
+
+/**
+ * Determines whether the given user is permitted to perform a room upgrade
+ * @param {String} userId The ID of the user to test against
+ * @returns {bool} True if the given user is permitted to upgrade the room
+ */
+Room.prototype.userMayUpgradeRoom = function(userId) {
+    return this.currentState.maySendStateEvent("m.room.tombstone", userId);
+};
+
 /**
  * Get the list of pending sent events for this room
  *
@@ -201,6 +252,223 @@ Room.prototype.getLiveTimeline = function() {
     return this.getUnfilteredTimelineSet().getLiveTimeline();
 };
 
+/**
+ * @param {string} myUserId the user id for the logged in member
+ * @return {string} the membership type (join | leave | invite) for the logged in user
+ */
+Room.prototype.getMyMembership = function() {
+    if (this.myUserId) {
+        const me = this.getMember(this.myUserId);
+        if (me) {
+            return me.membership;
+        }
+    }
+    return this._syncedMembership;
+};
+
+/**
+ * If this room is a DM we're invited to,
+ * try to find out who invited us
+ * @return {string} user id of the inviter
+ */
+Room.prototype.getDMInviter = function() {
+    if (this.myUserId) {
+        const me = this.getMember(this.myUserId);
+        if (me) {
+            return me.getDMInviter();
+        }
+    }
+    if (this._syncedMembership === "invite") {
+        // fall back to summary information
+        const memberCount = this.getInvitedAndJoinedMemberCount();
+        if (memberCount == 2 && this._summaryHeroes.length) {
+            return this._summaryHeroes[0];
+        }
+    }
+};
+
+/**
+ * Assuming this room is a DM room, tries to guess with which user.
+ * @return {string} user id of the other member (could be syncing user)
+ */
+Room.prototype.guessDMUserId = function() {
+    const me = this.getMember(this.myUserId);
+    if (me) {
+        const inviterId = me.getDMInviter();
+        if (inviterId) {
+            return inviterId;
+        }
+    }
+    // remember, we're assuming this room is a DM,
+    // so returning the first member we find should be fine
+    const hasHeroes = Array.isArray(this._summaryHeroes) &&
+        this._summaryHeroes.length;
+    if (hasHeroes) {
+        return this._summaryHeroes[0];
+    }
+    const members = this.currentState.getMembers();
+    const anyMember = members.find((m) => m.userId !== this.myUserId);
+    if (anyMember) {
+        return anyMember.userId;
+    }
+    // it really seems like I'm the only user in the room
+    // so I probably created a room with just me in it
+    // and marked it as a DM. Ok then
+    return this.myUserId;
+};
+
+Room.prototype.getAvatarFallbackMember = function() {
+    const memberCount = this.getInvitedAndJoinedMemberCount();
+    if (memberCount > 2) {
+        return;
+    }
+    const hasHeroes = Array.isArray(this._summaryHeroes) &&
+        this._summaryHeroes.length;
+    if (hasHeroes) {
+        const availableMember = this._summaryHeroes.map((userId) => {
+            return this.getMember(userId);
+        }).find((member) => !!member);
+        if (availableMember) {
+            return availableMember;
+        }
+    }
+    const members = this.currentState.getMembers();
+    // could be different than memberCount
+    // as this includes left members
+    if (members.length <= 2) {
+        const availableMember = members.find((m) => {
+            return m.userId !== this.myUserId;
+        });
+        if (availableMember) {
+            return availableMember;
+        }
+    }
+    // if all else fails, try falling back to a user,
+    // and create a one-off member for it
+    if (hasHeroes) {
+        const availableUser = this._summaryHeroes.map((userId) => {
+            return this._client.getUser(userId);
+        }).find((user) => !!user);
+        if (availableUser) {
+            const member = new RoomMember(
+                this.roomId, availableUser.userId);
+            member.user = availableUser;
+            return member;
+        }
+    }
+};
+
+/**
+ * Sets the membership this room was received as during sync
+ * @param {string} membership join | leave | invite
+ */
+Room.prototype.setSyncedMembership = function(membership) {
+    this._syncedMembership = membership;
+};
+
+Room.prototype._loadMembersFromServer = async function() {
+    const lastSyncToken = this._client.store.getSyncToken();
+    const queryString = utils.encodeParams({
+        not_membership: "leave",
+        at: lastSyncToken,
+    });
+    const path = utils.encodeUri("/rooms/$roomId/members?" + queryString,
+        {$roomId: this.roomId});
+    const http = this._client._http;
+    const response = await http.authedRequest(undefined, "GET", path);
+    return response.chunk;
+};
+
+
+Room.prototype._loadMembers = async function() {
+    // were the members loaded from the server?
+    let fromServer = false;
+    let rawMembersEvents =
+        await this._client.store.getOutOfBandMembers(this.roomId);
+    if (rawMembersEvents === null) {
+        fromServer = true;
+        rawMembersEvents = await this._loadMembersFromServer();
+        console.log(`LL: got ${rawMembersEvents.length} ` +
+            `members from server for room ${this.roomId}`);
+    }
+    const memberEvents = rawMembersEvents.map(this._client.getEventMapper());
+    return {memberEvents, fromServer};
+};
+
+/**
+ * Preloads the member list in case lazy loading
+ * of memberships is in use. Can be called multiple times,
+ * it will only preload once.
+ * @return {Promise} when preloading is done and
+ * accessing the members on the room will take
+ * all members in the room into account
+ */
+Room.prototype.loadMembersIfNeeded = function() {
+    if (this._membersPromise) {
+        return this._membersPromise;
+    }
+
+    // mark the state so that incoming messages while
+    // the request is in flight get marked as superseding
+    // the OOB members
+    this.currentState.markOutOfBandMembersStarted();
+
+    const promise = this._loadMembers().then(({memberEvents, fromServer}) => {
+        this.currentState.setOutOfBandMembers(memberEvents);
+        if (fromServer) {
+            const oobMembers = this.currentState.getMembers()
+                .filter((m) => m.isOutOfBand())
+                .map((m) => m.events.member.event);
+            console.log(`LL: telling store to write ${oobMembers.length}`
+                + ` members for room ${this.roomId}`);
+            const store = this._client.store;
+            return store.setOutOfBandMembers(this.roomId, oobMembers)
+                // swallow any IDB error as we don't want to fail
+                // because of this
+                .catch((err) => {
+                    console.log("LL: storing OOB room members failed, oh well",
+                        err);
+                });
+        }
+    }).catch((err) => {
+        // allow retries on fail
+        this._membersPromise = null;
+        this.currentState.markOutOfBandMembersFailed();
+        throw err;
+    });
+
+    this._membersPromise = promise;
+    // now the members are loaded, start to track the e2e devices if needed
+    if (this._client.isRoomEncrypted(this.roomId)) {
+        this._client._crypto.trackRoomDevices(this.roomId);
+    }
+
+    return this._membersPromise;
+};
+
+/**
+ * Removes the lazily loaded members from storage if needed
+ */
+Room.prototype.clearLoadedMembersIfNeeded = async function() {
+    if (this._opts.lazyLoadMembers && this._membersPromise) {
+        await this.loadMembersIfNeeded();
+        await this._client.store.clearOutOfBandMembers(this.roomId);
+        this.currentState.clearOutOfBandMembers();
+        this._membersPromise = null;
+    }
+};
+
+/**
+ * called when sync receives this room in the leave section
+ * to do cleanup after leaving a room. Possibly called multiple times.
+ */
+Room.prototype.onLeft = function() {
+    this.clearLoadedMembersIfNeeded().catch((err) => {
+        console.error(`error after clearing loaded members from ` +
+            `room ${this.roomId} after leaving`);
+        console.dir(err);
+    });
+};
 
 /**
  * Reset the live timeline of all timelineSets, and start new ones.
@@ -306,6 +574,26 @@ Room.prototype.setUnreadNotificationCount = function(type, count) {
     this._notificationCounts[type] = count;
 };
 
+Room.prototype.setSummary = function(summary) {
+    const heroes = summary["m.heroes"];
+    const joinedCount = summary["m.joined_member_count"];
+    const invitedCount = summary["m.invited_member_count"];
+    if (Number.isInteger(joinedCount)) {
+        this.currentState.setJoinedMemberCount(joinedCount);
+    }
+    if (Number.isInteger(invitedCount)) {
+        this.currentState.setInvitedMemberCount(invitedCount);
+    }
+    if (Array.isArray(heroes)) {
+        // be cautious about trusting server values,
+        // and make sure heroes doesn't contain our own id
+        // just to be sure
+        this._summaryHeroes = heroes.filter((userId) => {
+            return userId !== this.myUserId;
+        });
+    }
+};
+
 /**
  * Whether to send encrypted messages to devices within this room.
  * @param {Boolean} value true to blacklist unverified devices, null
@@ -430,11 +718,7 @@ Room.prototype.addEventsToTimeline = function(events, toStartOfTimeline,
  * @return {RoomMember} The member or <code>null</code>.
  */
  Room.prototype.getMember = function(userId) {
-    const member = this.currentState.members[userId];
-    if (!member) {
-        return null;
-    }
-    return member;
+    return this.currentState.getMember(userId);
  };
 
 /**
@@ -445,6 +729,33 @@ Room.prototype.addEventsToTimeline = function(events, toStartOfTimeline,
     return this.getMembersWithMembership("join");
  };
 
+/**
+ * Returns the number of joined members in this room
+ * This method caches the result.
+ * This is a wrapper around the method of the same name in roomState, returning
+ * its result for the room's current state.
+ * @return {integer} The number of members in this room whose membership is 'join'
+ */
+Room.prototype.getJoinedMemberCount = function() {
+    return this.currentState.getJoinedMemberCount();
+};
+
+/**
+ * Returns the number of invited members in this room
+ * @return {integer} The number of members in this room whose membership is 'invite'
+ */
+Room.prototype.getInvitedMemberCount = function() {
+    return this.currentState.getInvitedMemberCount();
+};
+
+/**
+ * Returns the number of invited + joined members in this room
+ * @return {integer} The number of members in this room whose membership is 'invite' or 'join'
+ */
+Room.prototype.getInvitedAndJoinedMemberCount = function() {
+    return this.getInvitedMemberCount() + this.getJoinedMemberCount();
+};
+
 /**
  * Get a list of members with given membership state.
  * @param {string} membership The membership state.
@@ -456,6 +767,29 @@ Room.prototype.addEventsToTimeline = function(events, toStartOfTimeline,
     });
  };
 
+ /**
+  * Get a list of members we should be encrypting for in this room
+  * @return {Promise<RoomMember[]>} A list of members who
+  * we should encrypt messages for in this room.
+  */
+ Room.prototype.getEncryptionTargetMembers = async function() {
+    await this.loadMembersIfNeeded();
+    let members = this.getMembersWithMembership("join");
+    if (this.shouldEncryptForInvitedMembers()) {
+        members = members.concat(this.getMembersWithMembership("invite"));
+    }
+    return members;
+ };
+
+ /**
+  * Determine whether we should encrypt messages for invited users in this room
+  * @return {boolean} if we should encrypt messages for invited users
+  */
+ Room.prototype.shouldEncryptForInvitedMembers = function() {
+    const ev = this.currentState.getStateEvents("m.room.history_visibility", "");
+    return (ev && ev.getContent() && ev.getContent().history_visibility !== "joined");
+ };
+
  /**
   * Get the default room name (i.e. what a given user would see if the
   * room had no m.room.name)
@@ -909,15 +1243,14 @@ Room.prototype.removeEvent = function(eventId) {
  * Recalculate various aspects of the room, including the room name and
  * room summary. Call this any time the room's current state is modified.
  * May fire "Room.name" if the room name is updated.
- * @param {string} userId The client's user ID.
  * @fires module:client~MatrixClient#event:"Room.name"
  */
-Room.prototype.recalculate = function(userId) {
+Room.prototype.recalculate = function() {
     // set fake stripped state events if this is an invite room so logic remains
     // consistent elsewhere.
     const self = this;
     const membershipEvent = this.currentState.getStateEvents(
-        "m.room.member", userId,
+        "m.room.member", this.myUserId,
     );
     if (membershipEvent && membershipEvent.getContent().membership === "invite") {
         const strippedStateEvents = membershipEvent.event.invite_room_state || [];
@@ -933,14 +1266,14 @@ Room.prototype.recalculate = function(userId) {
                     content: strippedEvent.content,
                     event_id: "$fake" + Date.now(),
                     room_id: self.roomId,
-                    user_id: userId, // technically a lie
+                    user_id: self.myUserId, // technically a lie
                 })]);
             }
         });
     }
 
     const oldName = this.name;
-    this.name = calculateRoomName(this, userId);
+    this.name = calculateRoomName(this, this.myUserId);
     this.summary = new RoomSummary(this.roomId, {
         title: this.name,
     });
@@ -950,7 +1283,6 @@ Room.prototype.recalculate = function(userId) {
     }
 };
 
-
 /**
  * Get a list of user IDs who have <b>read up to</b> the given event.
  * @param {MatrixEvent} event the event to get read receipts for.
@@ -1122,7 +1454,7 @@ Room.prototype.addTags = function(event) {
     // }
 
     // XXX: do we need to deep copy here?
-    this.tags = event.getContent().tags;
+    this.tags = event.getContent().tags || {};
 
     // XXX: we could do a deep-comparison to see if the tags have really
     // changed - but do we want to bother?
@@ -1153,6 +1485,17 @@ Room.prototype.getAccountData = function(type) {
     return this.accountData[type];
 };
 
+
+/**
+ * Returns wheter the syncing user has permission to send a message in the room
+ * @return {boolean} true if the user should be permitted to send
+ *                   message events into the room.
+ */
+Room.prototype.maySendMessage = function() {
+    return this.getMyMembership() === 'join' &&
+        this.currentState.maySendEvent('m.room.message', this.myUserId);
+};
+
 /**
  * This is an internal method. Calculates the name of the room from the current
  * room state.
@@ -1186,87 +1529,83 @@ function calculateRoomName(room, userId, ignoreRoomNameEvent) {
         return alias;
     }
 
-    // get members that are NOT ourselves and are actually in the room.
-    const otherMembers = utils.filter(room.currentState.getMembers(), function(m) {
-        return (
-            m.userId !== userId && m.membership !== "leave" && m.membership !== "ban"
-        );
-    });
-    const allMembers = utils.filter(room.currentState.getMembers(), function(m) {
-        return (m.membership !== "leave");
-    });
-    const myMemberEventArray = utils.filter(room.currentState.getMembers(), function(m) {
-        return (m.userId == userId);
-    });
-    const myMemberEvent = (
-        (myMemberEventArray.length && myMemberEventArray[0].events) ?
-            myMemberEventArray[0].events.member.event : undefined
-    );
+    const joinedMemberCount = room.currentState.getJoinedMemberCount();
+    const invitedMemberCount = room.currentState.getInvitedMemberCount();
+    // -1 because these numbers include the syncing user
+    const inviteJoinCount = joinedMemberCount + invitedMemberCount - 1;
 
-    // TODO: Localisation
-    if (myMemberEvent && myMemberEvent.content.membership == "invite") {
-        if (room.currentState.getMember(myMemberEvent.sender)) {
-            // extract who invited us to the room
-            return room.currentState.getMember(
-                myMemberEvent.sender,
-            ).name;
-        } else if (allMembers[0].events.member) {
-            // use the sender field from the invite event, although this only
-            // gets us the mxid
-            return myMemberEvent.sender;
-        } else {
-            return "Room Invite";
-        }
+    // get members that are NOT ourselves and are actually in the room.
+    let otherNames = null;
+    if (room._summaryHeroes) {
+        // if we have a summary, the member state events
+        // should be in the room state
+        otherNames = room._summaryHeroes.map((userId) => {
+            const member = room.getMember(userId);
+            return member ? member.name : userId;
+        });
+    } else {
+        let otherMembers = room.currentState.getMembers().filter((m) => {
+            return m.userId !== userId &&
+                (m.membership === "invite" || m.membership === "join");
+        });
+        // make sure members have stable order
+        otherMembers.sort((a, b) => a.userId.localeCompare(b.userId));
+        // only 5 first members, immitate _summaryHeroes
+        otherMembers = otherMembers.slice(0, 5);
+        otherNames = otherMembers.map((m) => m.name);
     }
 
+    if (inviteJoinCount) {
+        return memberNamesToRoomName(otherNames, inviteJoinCount);
+    }
 
-    if (otherMembers.length === 0) {
-        const leftMembers = utils.filter(room.currentState.getMembers(), function(m) {
-            return m.userId !== userId && m.membership === "leave";
-        });
-        if (allMembers.length === 1) {
-            // self-chat, peeked room with 1 participant,
-            // or inbound invite, or outbound 3PID invite.
-            if (allMembers[0].userId === userId) {
-                const thirdPartyInvites =
-                    room.currentState.getStateEvents("m.room.third_party_invite");
-                if (thirdPartyInvites && thirdPartyInvites.length > 0) {
-                    let name = "Inviting " +
-                               thirdPartyInvites[0].getContent().display_name;
-                    if (thirdPartyInvites.length > 1) {
-                        if (thirdPartyInvites.length == 2) {
-                            name += " and " +
-                                    thirdPartyInvites[1].getContent().display_name;
-                        } else {
-                            name += " and " +
-                                    thirdPartyInvites.length + " others";
-                        }
-                    }
-                    return name;
-                } else if (leftMembers.length === 1) {
-                    // if it was a chat with one person who's now left, it's still
-                    // notionally a chat with them
-                    return leftMembers[0].name;
-                } else {
-                    return "Empty room";
-                }
-            } else {
-                return allMembers[0].name;
-            }
-        } else {
-            // there really isn't anyone in this room...
-            return "Empty room";
+    const myMembership = room.getMyMembership();
+    // if I have created a room and invited people throuh
+    // 3rd party invites
+    if (myMembership == 'join') {
+        const thirdPartyInvites =
+            room.currentState.getStateEvents("m.room.third_party_invite");
+
+        if (thirdPartyInvites && thirdPartyInvites.length) {
+            const thirdPartyNames = thirdPartyInvites.map((i) => {
+                return i.getContent().display_name;
+            });
+
+            return `Inviting ${memberNamesToRoomName(thirdPartyNames)}`;
         }
-    } else if (otherMembers.length === 1) {
-        return otherMembers[0].name;
-    } else if (otherMembers.length === 2) {
-        return (
-            otherMembers[0].name + " and " + otherMembers[1].name
-        );
+    }
+    // let's try to figure out who was here before
+    let leftNames = otherNames;
+    // if we didn't have heroes, try finding them in the room state
+    if(!leftNames.length) {
+        leftNames = room.currentState.getMembers().filter((m) => {
+            return m.userId !== userId &&
+                m.membership !== "invite" &&
+                m.membership !== "join";
+        }).map((m) => m.name);
+    }
+    if(leftNames.length) {
+        return `Empty room (was ${memberNamesToRoomName(leftNames)})`;
     } else {
-        return (
-            otherMembers[0].name + " and " + (otherMembers.length - 1) + " others"
-        );
+        return "Empty room";
+    }
+}
+
+function memberNamesToRoomName(names, count = (names.length + 1)) {
+    const countWithoutMe = count - 1;
+    if (!names.length) {
+       return count <= 1 ? "Empty room" : null;
+    } else if (names.length === 1 && countWithoutMe <= 1) {
+        return names[0];
+    } else if (names.length === 2 && countWithoutMe <= 2) {
+        return `${names[0]} and ${names[1]}`;
+    } else {
+        const plural = countWithoutMe > 1;
+        if (plural) {
+            return `${names[0]} and ${countWithoutMe} others`;
+        } else {
+            return `${names[0]} and 1 other`;
+        }
     }
 }
 

src/store/indexeddb-local-backend.js

@@ -19,7 +19,7 @@ import Promise from 'bluebird';
 import SyncAccumulator from "../sync-accumulator";
 import utils from "../utils";
 
-const VERSION = 1;
+const VERSION = 2;
 
 function createDatabase(db) {
     // Make user store, clobber based on user ID. (userId property of User objects)
@@ -33,6 +33,14 @@ function createDatabase(db) {
     db.createObjectStore("sync", { keyPath: ["clobber"] });
 }
 
+function upgradeSchemaV2(db) {
+    const oobMembersStore = db.createObjectStore(
+        "oob_membership_events", {
+            keyPath: ["room_id", "state_key"],
+        });
+    oobMembersStore.createIndex("room", "room_id");
+}
+
 /**
  * Helper method to collect results from a Cursor and promiseify it.
  * @param {ObjectStore|Index} store The store to perform openCursor on.
@@ -63,7 +71,7 @@ function selectQuery(store, keyRange, resultMapper) {
     });
 }
 
-function promiseifyTxn(txn) {
+function txnAsPromise(txn) {
     return new Promise((resolve, reject) => {
         txn.oncomplete = function(event) {
             resolve(event);
@@ -74,7 +82,7 @@ function promiseifyTxn(txn) {
     });
 }
 
-function promiseifyRequest(req) {
+function reqAsEventPromise(req) {
     return new Promise((resolve, reject) => {
         req.onsuccess = function(event) {
             resolve(event);
@@ -85,6 +93,17 @@ function promiseifyRequest(req) {
     });
 }
 
+function reqAsPromise(req) {
+    return new Promise((resolve, reject) => {
+        req.onsuccess = () => resolve(req);
+        req.onerror = (err) => reject(err);
+    });
+}
+
+function reqAsCursorPromise(req) {
+    return reqAsEventPromise(req).then((event) => event.target.result);
+}
+
 /**
  * Does the actual reading from and writing to the indexeddb
  *
@@ -136,6 +155,9 @@ LocalIndexedDBStoreBackend.prototype = {
             if (oldVersion < 1) { // The database did not previously exist.
                 createDatabase(db);
             }
+            if (oldVersion < 2) {
+                upgradeSchemaV2(db);
+            }
             // Expand as needed.
         };
 
@@ -148,7 +170,7 @@ LocalIndexedDBStoreBackend.prototype = {
         console.log(
             `LocalIndexedDBStoreBackend.connect: awaiting connection...`,
         );
-        return promiseifyRequest(req).then((ev) => {
+        return reqAsEventPromise(req).then((ev) => {
             console.log(
                 `LocalIndexedDBStoreBackend.connect: connected`,
             );
@@ -187,6 +209,138 @@ LocalIndexedDBStoreBackend.prototype = {
         });
     },
 
+    /**
+     * Returns the out-of-band membership events for this room that
+     * were previously loaded.
+     * @param {string} roomId
+     * @returns {Promise<event[]>} the events, potentially an empty array if OOB loading didn't yield any new members
+     * @returns {null} in case the members for this room haven't been stored yet
+     */
+    getOutOfBandMembers: function(roomId) {
+        return new Promise((resolve, reject) =>{
+            const tx = this.db.transaction(["oob_membership_events"], "readonly");
+            const store = tx.objectStore("oob_membership_events");
+            const roomIndex = store.index("room");
+            const range = IDBKeyRange.only(roomId);
+            const request = roomIndex.openCursor(range);
+
+            const membershipEvents = [];
+            // did we encounter the oob_written marker object
+            // amongst the results? That means OOB member
+            // loading already happened for this room
+            // but there were no members to persist as they
+            // were all known already
+            let oobWritten = false;
+
+            request.onsuccess = (event) => {
+                const cursor = event.target.result;
+                if (!cursor) {
+                    // Unknown room
+                    if (!membershipEvents.length && !oobWritten) {
+                        return resolve(null);
+                    }
+                    return resolve(membershipEvents);
+                }
+                const record = cursor.value;
+                if (record.oob_written) {
+                    oobWritten = true;
+                } else {
+                    membershipEvents.push(record);
+                }
+                cursor.continue();
+            };
+            request.onerror = (err) => {
+                reject(err);
+            };
+        }).then((events) => {
+            console.log(`LL: got ${events && events.length}` +
+                ` membershipEvents from storage for room ${roomId} ...`);
+            return events;
+        });
+    },
+
+    /**
+     * Stores the out-of-band membership events for this room. Note that
+     * it still makes sense to store an empty array as the OOB status for the room is
+     * marked as fetched, and getOutOfBandMembers will return an empty array instead of null
+     * @param {string} roomId
+     * @param {event[]} membershipEvents the membership events to store
+     * @returns {Promise} when all members have been stored
+     */
+    setOutOfBandMembers: function(roomId, membershipEvents) {
+        console.log(`LL: backend about to store ${membershipEvents.length}` +
+            ` members for ${roomId}`);
+        function ignoreResult() {}
+        // run everything in a promise so anything that throws will reject
+        return new Promise((resolve) =>{
+            const tx = this.db.transaction(["oob_membership_events"], "readwrite");
+            const store = tx.objectStore("oob_membership_events");
+            const eventPuts = membershipEvents.map((e) => {
+                const putPromise = reqAsEventPromise(store.put(e));
+                // ignoring the result makes sure we discard the IDB success event
+                // ASAP, and not create a potentially big array containing them
+                // unneccesarily later on by calling Promise.all.
+                return putPromise.then(ignoreResult);
+            });
+            // aside from all the events, we also write a marker object to the store
+            // to mark the fact that OOB members have been written for this room.
+            // It's possible that 0 members need to be written as all where previously know
+            // but we still need to know whether to return null or [] from getOutOfBandMembers
+            // where null means out of band members haven't been stored yet for this room
+            const markerObject = {
+                room_id: roomId,
+                oob_written: true,
+                state_key: 0,
+            };
+            const markerPut = reqAsEventPromise(store.put(markerObject));
+            const allPuts = eventPuts.concat(markerPut);
+            // ignore the empty array Promise.all creates
+            // as this method should just resolve
+            // to undefined on success
+            resolve(Promise.all(allPuts).then(ignoreResult));
+        }).then(() => {
+            console.log(`LL: backend done storing for ${roomId}!`);
+        });
+    },
+
+    clearOutOfBandMembers: async function(roomId) {
+        // the approach to delete all members for a room
+        // is to get the min and max state key from the index
+        // for that room, and then delete between those
+        // keys in the store.
+        // this should be way faster than deleting every member
+        // individually for a large room.
+        const readTx = this.db.transaction(
+            ["oob_membership_events"],
+            "readonly");
+        const store = readTx.objectStore("oob_membership_events");
+        const roomIndex = store.index("room");
+        const roomRange = IDBKeyRange.only(roomId);
+
+        const minStateKeyProm = reqAsCursorPromise(
+                roomIndex.openKeyCursor(roomRange, "next"),
+            ).then((cursor) => cursor && cursor.primaryKey[1]);
+        const maxStateKeyProm = reqAsCursorPromise(
+                roomIndex.openKeyCursor(roomRange, "prev"),
+            ).then((cursor) => cursor && cursor.primaryKey[1]);
+        const [minStateKey, maxStateKey] = await Promise.all(
+            [minStateKeyProm, maxStateKeyProm]);
+
+        const writeTx = this.db.transaction(
+            ["oob_membership_events"],
+            "readwrite");
+        const writeStore = writeTx.objectStore("oob_membership_events");
+        const membersKeyRange = IDBKeyRange.bound(
+            [roomId, minStateKey],
+            [roomId, maxStateKey],
+        );
+
+        console.log(`LL: Deleting all users + marker in storage for ` +
+            `room ${roomId}, with key range:`,
+            [roomId, minStateKey], [roomId, maxStateKey]);
+        await reqAsPromise(writeStore.delete(membersKeyRange));
+    },
+
     /**
      * Clear the entire database. This should be used when logging out of a client
      * to prevent mixing data between accounts.
@@ -284,7 +438,7 @@ LocalIndexedDBStoreBackend.prototype = {
                 roomsData: roomsData,
                 groupsData: groupsData,
             }); // put == UPSERT
-            return promiseifyTxn(txn);
+            return txnAsPromise(txn);
         });
     },
 
@@ -301,7 +455,7 @@ LocalIndexedDBStoreBackend.prototype = {
             for (let i = 0; i < accountData.length; i++) {
                 store.put(accountData[i]); // put == UPSERT
             }
-            return promiseifyTxn(txn);
+            return txnAsPromise(txn);
         });
     },
 
@@ -323,7 +477,7 @@ LocalIndexedDBStoreBackend.prototype = {
                     event: tuple[1],
                 }); // put == UPSERT
             }
-            return promiseifyTxn(txn);
+            return txnAsPromise(txn);
         });
     },
 

src/store/indexeddb-remote-backend.js

@@ -87,6 +87,32 @@ RemoteIndexedDBStoreBackend.prototype = {
         return this._doCmd('syncToDatabase', [users]);
     },
 
+    /**
+     * Returns the out-of-band membership events for this room that
+     * were previously loaded.
+     * @param {string} roomId
+     * @returns {event[]} the events, potentially an empty array if OOB loading didn't yield any new members
+     * @returns {null} in case the members for this room haven't been stored yet
+     */
+    getOutOfBandMembers: function(roomId) {
+        return this._doCmd('getOutOfBandMembers', [roomId]);
+    },
+
+    /**
+     * Stores the out-of-band membership events for this room. Note that
+     * it still makes sense to store an empty array as the OOB status for the room is
+     * marked as fetched, and getOutOfBandMembers will return an empty array instead of null
+     * @param {string} roomId
+     * @param {event[]} membershipEvents the membership events to store
+     * @returns {Promise} when all members have been stored
+     */
+    setOutOfBandMembers: function(roomId, membershipEvents) {
+        return this._doCmd('setOutOfBandMembers', [roomId, membershipEvents]);
+    },
+
+    clearOutOfBandMembers: function(roomId) {
+        return this._doCmd('clearOutOfBandMembers', [roomId]);
+    },
 
     /**
      * Load all user presence events from the database. This is not cached.

src/store/indexeddb-store-worker.js

@@ -92,6 +92,15 @@ class IndexedDBStoreWorker {
             case 'getNextBatchToken':
                 prom = this.backend.getNextBatchToken();
                 break;
+            case 'getOutOfBandMembers':
+                prom = this.backend.getOutOfBandMembers(msg.args[0]);
+                break;
+            case 'clearOutOfBandMembers':
+                prom = this.backend.clearOutOfBandMembers(msg.args[0]);
+                break;
+            case 'setOutOfBandMembers':
+                prom = this.backend.setOutOfBandMembers(msg.args[0], msg.args[1]);
+                break;
         }
 
         if (prom === undefined) {

src/store/indexeddb.js

@@ -219,4 +219,31 @@ IndexedDBStore.prototype.setSyncData = function(syncData) {
     return this.backend.setSyncData(syncData);
 };
 
+/**
+ * Returns the out-of-band membership events for this room that
+ * were previously loaded.
+ * @param {string} roomId
+ * @returns {event[]} the events, potentially an empty array if OOB loading didn't yield any new members
+ * @returns {null} in case the members for this room haven't been stored yet
+ */
+IndexedDBStore.prototype.getOutOfBandMembers = function(roomId) {
+    return this.backend.getOutOfBandMembers(roomId);
+};
+
+/**
+ * Stores the out-of-band membership events for this room. Note that
+ * it still makes sense to store an empty array as the OOB status for the room is
+ * marked as fetched, and getOutOfBandMembers will return an empty array instead of null
+ * @param {string} roomId
+ * @param {event[]} membershipEvents the membership events to store
+ * @returns {Promise} when all members have been stored
+ */
+IndexedDBStore.prototype.setOutOfBandMembers = function(roomId, membershipEvents) {
+    return this.backend.setOutOfBandMembers(roomId, membershipEvents);
+};
+
+IndexedDBStore.prototype.clearOutOfBandMembers = function(roomId) {
+    return this.backend.clearOutOfBandMembers(roomId);
+};
+
 module.exports.IndexedDBStore = IndexedDBStore;

src/store/memory.js

@@ -52,6 +52,9 @@ module.exports.MatrixInMemoryStore = function MatrixInMemoryStore(opts) {
         // type : content
     };
     this.localStorage = opts.localStorage;
+    this._oobMembers = {
+        // roomId: [member events]
+    };
 };
 
 module.exports.MatrixInMemoryStore.prototype = {
@@ -377,4 +380,26 @@ module.exports.MatrixInMemoryStore.prototype = {
         };
         return Promise.resolve();
     },
+    /**
+     * Returns the out-of-band membership events for this room that
+     * were previously loaded.
+     * @param {string} roomId
+     * @returns {event[]} the events, potentially an empty array if OOB loading didn't yield any new members
+     * @returns {null} in case the members for this room haven't been stored yet
+     */
+    getOutOfBandMembers: function(roomId) {
+        return Promise.resolve(this._oobMembers[roomId] || null);
+    },
+    /**
+     * Stores the out-of-band membership events for this room. Note that
+     * it still makes sense to store an empty array as the OOB status for the room is
+     * marked as fetched, and getOutOfBandMembers will return an empty array instead of null
+     * @param {string} roomId
+     * @param {event[]} membershipEvents the membership events to store
+     * @returns {Promise} when all members have been stored
+     */
+    setOutOfBandMembers: function(roomId, membershipEvents) {
+        this._oobMembers[roomId] = membershipEvents;
+        return Promise.resolve();
+    },
 };

src/store/stub.js

@@ -264,6 +264,18 @@ StubStore.prototype = {
     deleteAllData: function() {
         return Promise.resolve();
     },
+
+    getOutOfBandMembers: function() {
+        return Promise.resolve(null);
+    },
+
+    setOutOfBandMembers: function() {
+        return Promise.resolve();
+    },
+
+    clearOutOfBandMembers: function() {
+        return Promise.resolve();
+    },
 };
 
 /** Stub Store class. */

src/sync-accumulator.js

@@ -63,6 +63,11 @@ class SyncAccumulator {
             //       { event: $event, token: null|token },
             //       ...
             //    ],
+            //    _summary: {
+            //       m.heroes: [ $user_id ],
+            //       m.joined_member_count: $count,
+            //       m.invited_member_count: $count
+            //    },
             //    _accountData: { $event_type: json },
             //    _unreadNotifications: { ... unread_notifications JSON ... },
             //    _readReceipts: { $user_id: { data: $json, eventId: $event_id }}
@@ -242,6 +247,7 @@ class SyncAccumulator {
                 _timeline: [],
                 _accountData: Object.create(null),
                 _unreadNotifications: {},
+                _summary: {},
                 _readReceipts: {},
             };
         }
@@ -258,6 +264,17 @@ class SyncAccumulator {
         if (data.unread_notifications) {
             currentData._unreadNotifications = data.unread_notifications;
         }
+        if (data.summary) {
+            const HEROES_KEY = "m.heroes";
+            const INVITED_COUNT_KEY = "m.invited_member_count";
+            const JOINED_COUNT_KEY = "m.joined_member_count";
+
+            const acc = currentData._summary;
+            const sum = data.summary;
+            acc[HEROES_KEY] = sum[HEROES_KEY] || acc[HEROES_KEY];
+            acc[JOINED_COUNT_KEY] = sum[JOINED_COUNT_KEY] || acc[JOINED_COUNT_KEY];
+            acc[INVITED_COUNT_KEY] = sum[INVITED_COUNT_KEY] || acc[INVITED_COUNT_KEY];
+        }
 
         if (data.ephemeral && data.ephemeral.events) {
             data.ephemeral.events.forEach((e) => {
@@ -428,6 +445,7 @@ class SyncAccumulator {
                     prev_batch: null,
                 },
                 unread_notifications: roomData._unreadNotifications,
+                summary: roomData._summary,
             };
             // Add account data
             Object.keys(roomData._accountData).forEach((evType) => {

src/sync.js

@@ -93,6 +93,7 @@ function SyncApi(client, opts) {
     this._peekRoomId = null;
     this._currentSyncRequest = null;
     this._syncState = null;
+    this._syncStateData = null; // additional data (eg. error object for failed sync)
     this._catchingUp = false;
     this._running = false;
     this._keepAliveTimer = null;
@@ -112,7 +113,8 @@ function SyncApi(client, opts) {
  */
 SyncApi.prototype.createRoom = function(roomId) {
     const client = this.client;
-    const room = new Room(roomId, {
+    const room = new Room(roomId, client, client.getUserId(), {
+        lazyLoadMembers: this.opts.lazyLoadMembers,
         pendingEventOrdering: this.opts.pendingEventOrdering,
         timelineSupport: client.timelineSupport,
     });
@@ -231,7 +233,7 @@ SyncApi.prototype.syncLeftRooms = function() {
 
             self._processRoomEvents(room, stateEvents, timelineEvents);
 
-            room.recalculate(client.credentials.userId);
+            room.recalculate();
             client.store.storeRoom(room);
             client.emit("Room", room);
 
@@ -302,7 +304,7 @@ SyncApi.prototype.peek = function(roomId) {
         peekRoom.currentState.setStateEvents(stateEvents);
 
         self._resolveInvites(peekRoom);
-        peekRoom.recalculate(self.client.credentials.userId);
+        peekRoom.recalculate();
 
         // roll backwards to diverge old state. addEventsToTimeline
         // will overwrite the pagination token, so make sure it overwrites
@@ -396,6 +398,18 @@ SyncApi.prototype.getSyncState = function() {
     return this._syncState;
 };
 
+/**
+ * Returns the additional data object associated with
+ * the current sync state, or null if there is no
+ * such data.
+ * Sync errors, if available, are put in the 'error' key of
+ * this object.
+ * @return {?Object}
+ */
+SyncApi.prototype.getSyncStateData = function() {
+    return this._syncStateData;
+};
+
 SyncApi.prototype.recoverFromSyncStartupError = async function(savedSyncPromise, err) {
     // Wait for the saved sync to complete - we send the pushrules and filter requests
     // before the saved sync has finished so they can run in parallel, but only process
@@ -763,9 +777,20 @@ SyncApi.prototype._onSyncError = function(err, syncOptions) {
     // fails, since long lived HTTP connections will
     // go away sometimes and we shouldn't treat this as
     // erroneous. We set the state to 'reconnecting'
-    // instead, so that clients can onserve this state
+    // instead, so that clients can observe this state
     // if they wish.
-    this._startKeepAlives().then(() => {
+    this._startKeepAlives().then((connDidFail) => {
+        // Only emit CATCHUP if we detected a connectivity error: if we didn't,
+        // it's quite likely the sync will fail again for the same reason and we
+        // want to stay in ERROR rather than keep flip-flopping between ERROR
+        // and CATCHUP.
+        if (connDidFail && this.getSyncState() === 'ERROR') {
+            this._updateSyncState("CATCHUP", {
+                oldSyncToken: null,
+                nextSyncToken: null,
+                catchingUp: true,
+            });
+        }
         this._sync(syncOptions);
     });
 
@@ -774,6 +799,7 @@ SyncApi.prototype._onSyncError = function(err, syncOptions) {
     this._updateSyncState(
         this._failedSyncCount >= FAILED_SYNC_ERROR_THRESHOLD ?
             "ERROR" : "RECONNECTING",
+        { error: err },
     );
 };
 
@@ -809,6 +835,11 @@ SyncApi.prototype._processSyncResponse = async function(
     //          state: { events: [] },
     //          timeline: { events: [], prev_batch: $token, limited: true },
     //          ephemeral: { events: [] },
+    //          summary: {
+    //             m.heroes: [ $user_id ],
+    //             m.joined_member_count: $count,
+    //             m.invited_member_count: $count
+    //          },
     //          account_data: { events: [] },
     //          unread_notifications: {
     //              highlight_count: 0,
@@ -945,11 +976,12 @@ SyncApi.prototype._processSyncResponse = async function(
     // Handle invites
     inviteRooms.forEach(function(inviteObj) {
         const room = inviteObj.room;
+        room.setSyncedMembership("invite");
         const stateEvents =
             self._mapSyncEventsFormat(inviteObj.invite_state, room);
         self._processRoomEvents(room, stateEvents);
         if (inviteObj.isBrandNewRoom) {
-            room.recalculate(client.credentials.userId);
+            room.recalculate();
             client.store.storeRoom(room);
             client.emit("Room", room);
         }
@@ -961,6 +993,7 @@ SyncApi.prototype._processSyncResponse = async function(
     // Handle joins
     await Promise.mapSeries(joinRooms, async function(joinObj) {
         const room = joinObj.room;
+        room.setSyncedMembership("join");
         const stateEvents = self._mapSyncEventsFormat(joinObj.state, room);
         const timelineEvents = self._mapSyncEventsFormat(joinObj.timeline, room);
         const ephemeralEvents = self._mapSyncEventsFormat(joinObj.ephemeral);
@@ -1040,6 +1073,13 @@ SyncApi.prototype._processSyncResponse = async function(
 
         self._processRoomEvents(room, stateEvents, timelineEvents);
 
+        // set summary after processing events,
+        // because it will trigger a name calculation
+        // which needs the room state to be up to date
+        if (joinObj.summary) {
+            room.setSummary(joinObj.summary);
+        }
+
         // XXX: should we be adding ephemeralEvents to the timeline?
         // It feels like that for symmetry with room.addAccountData()
         // there should be a room.addEphemeralEvents() or similar.
@@ -1048,7 +1088,7 @@ SyncApi.prototype._processSyncResponse = async function(
         // we deliberately don't add accountData to the timeline
         room.addAccountData(accountDataEvents);
 
-        room.recalculate(client.credentials.userId);
+        room.recalculate();
         if (joinObj.isBrandNewRoom) {
             client.store.storeRoom(room);
             client.emit("Room", room);
@@ -1076,6 +1116,8 @@ SyncApi.prototype._processSyncResponse = async function(
     // Handle leaves (e.g. kicked rooms)
     leaveRooms.forEach(function(leaveObj) {
         const room = leaveObj.room;
+        room.setSyncedMembership("leave");
+
         const stateEvents =
             self._mapSyncEventsFormat(leaveObj.state, room);
         const timelineEvents =
@@ -1086,7 +1128,7 @@ SyncApi.prototype._processSyncResponse = async function(
         self._processRoomEvents(room, stateEvents, timelineEvents);
         room.addAccountData(accountDataEvents);
 
-        room.recalculate(client.credentials.userId);
+        room.recalculate();
         if (leaveObj.isBrandNewRoom) {
             client.store.storeRoom(room);
             client.emit("Room", room);
@@ -1103,6 +1145,8 @@ SyncApi.prototype._processSyncResponse = async function(
         accountDataEvents.forEach(function(e) {
             client.emit("event", e);
         });
+
+        room.onLeft();
     });
 
     // update the notification timeline, if appropriate.
@@ -1175,13 +1219,16 @@ SyncApi.prototype._startKeepAlives = function(delay) {
  *
  * On failure, schedules a call back to itself. On success, resolves
  * this._connectionReturnedDefer.
+ *
+ * @param {bool} connDidFail True if a connectivity failure has been detected. Optional.
  */
-SyncApi.prototype._pokeKeepAlive = function() {
+SyncApi.prototype._pokeKeepAlive = function(connDidFail) {
+    if (connDidFail === undefined) connDidFail = false;
     const self = this;
     function success() {
         clearTimeout(self._keepAliveTimer);
         if (self._connectionReturnedDefer) {
-            self._connectionReturnedDefer.resolve();
+            self._connectionReturnedDefer.resolve(connDidFail);
             self._connectionReturnedDefer = null;
         }
     }
@@ -1198,7 +1245,7 @@ SyncApi.prototype._pokeKeepAlive = function() {
     ).done(function() {
         success();
     }, function(err) {
-        if (err.httpStatus == 400) {
+        if (err.httpStatus == 400 || err.httpStatus == 404) {
             // treat this as a success because the server probably just doesn't
             // support /versions: point is, we're getting a response.
             // We wait a short time though, just in case somehow the server
@@ -1206,8 +1253,9 @@ SyncApi.prototype._pokeKeepAlive = function() {
             // responses fail, this will mean we don't hammer in a loop.
             self._keepAliveTimer = setTimeout(success, 2000);
         } else {
+            connDidFail = true;
             self._keepAliveTimer = setTimeout(
-                self._pokeKeepAlive.bind(self),
+                self._pokeKeepAlive.bind(self, connDidFail),
                 5000 + Math.floor(Math.random() * 5000),
             );
             // A keepalive has failed, so we emit the
@@ -1215,7 +1263,7 @@ SyncApi.prototype._pokeKeepAlive = function() {
             // first failure).
             // Note we do this after setting the timer:
             // this lets the unit tests advance the mock
-            // clock when the get the error.
+            // clock when they get the error.
             self._updateSyncState("ERROR", { error: err });
         }
     });
@@ -1376,7 +1424,7 @@ SyncApi.prototype._processRoomEvents = function(room, stateEventList,
     // a recalculation (like m.room.name) we won't recalculate until we've
     // finished adding all the events, which will cause the notification to have
     // the old room name rather than the new one.
-    room.recalculate(this.client.credentials.userId);
+    room.recalculate();
 
     // If the timeline wasn't empty, we process the state events here: they're
     // defined as updates to the state before the start of the timeline, so this
@@ -1451,6 +1499,7 @@ SyncApi.prototype._getGuestFilter = function() {
 SyncApi.prototype._updateSyncState = function(newState, data) {
     const old = this._syncState;
     this._syncState = newState;
+    this._syncStateData = data;
     this.client.emit("sync", this._syncState, old, data);
 };
 

src/utils.js

@@ -662,3 +662,13 @@ module.exports.inherits = function(ctor, superCtor) {
 module.exports.isNumber = function(value) {
     return typeof value === 'number' && isFinite(value);
 };
+
+/**
+ * Removes zero width chars, diacritics and whitespace from the string
+ * @param {string} str the string to remove hidden characters from
+ * @return {string} a string with the hidden characters removed
+ */
+module.exports.removeHiddenChars = function(str) {
+    return str.normalize('NFD').replace(removeHiddenCharsRegex, '');
+};
+const removeHiddenCharsRegex = /[\u200B-\u200D\u0300-\u036f\uFEFF\s]/g;