v0.14.2

fix some assertions in e2e backup unit test

.travis.yml

@@ -1,5 +1,5 @@
 language: node_js
 node_js:
-    - node # Latest stable version of nodejs.
+    - "10.11.0"
 script:
     - ./travis.sh

CHANGELOG.md

@@ -1,3 +1,295 @@
+Changes in [0.14.2](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.14.2) (2018-12-10)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.14.2-rc.1...v0.14.2)
+
+ * No changes since rc.1
+
+Changes in [0.14.2-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.14.2-rc.1) (2018-12-06)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.14.1...v0.14.2-rc.1)
+
+ * fix some assertions in e2e backup unit test
+   [\#794](https://github.com/matrix-org/matrix-js-sdk/pull/794)
+ * Config should be called with auth
+   [\#798](https://github.com/matrix-org/matrix-js-sdk/pull/798)
+ * Don't re-establish sessions with unknown devices
+   [\#792](https://github.com/matrix-org/matrix-js-sdk/pull/792)
+ * e2e key backups
+   [\#684](https://github.com/matrix-org/matrix-js-sdk/pull/684)
+ * WIP: online incremental megolm backups
+   [\#595](https://github.com/matrix-org/matrix-js-sdk/pull/595)
+ * Support for e2e key backups
+   [\#736](https://github.com/matrix-org/matrix-js-sdk/pull/736)
+ * Passphrase Support for e2e backups
+   [\#786](https://github.com/matrix-org/matrix-js-sdk/pull/786)
+ * Add 'getSsoLoginUrl' function
+   [\#783](https://github.com/matrix-org/matrix-js-sdk/pull/783)
+ * Fix: don't set the room name to null when heroes are missing.
+   [\#784](https://github.com/matrix-org/matrix-js-sdk/pull/784)
+ * Handle crypto db version upgrades
+   [\#785](https://github.com/matrix-org/matrix-js-sdk/pull/785)
+ * Restart broken Olm sessions
+   [\#780](https://github.com/matrix-org/matrix-js-sdk/pull/780)
+ * Use the last olm session that got a message
+   [\#776](https://github.com/matrix-org/matrix-js-sdk/pull/776)
+
+Changes in [0.14.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.14.1) (2018-11-22)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.14.0...v0.14.1)
+
+ * Warning when crypto DB is too new to use.
+
+Changes in [0.14.0](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.14.0) (2018-11-19)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.14.0-rc.1...v0.14.0)
+
+ * No changes since rc.1
+
+Changes in [0.14.0-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.14.0-rc.1) (2018-11-15)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.13.1...v0.14.0-rc.1)
+
+BREAKING CHANGE
+----------------
+ 
+ * js-sdk now uses Olm 3.0. Apps using Olm must update to 3.0 to
+   continue using Olm with the js-sdk. The js-sdk will call Olm's
+   init() method when the client is started.
+
+All Changes
+-----------
+
+ * Prevent messages from being sent if other messages have failed to send
+   [\#781](https://github.com/matrix-org/matrix-js-sdk/pull/781)
+ * A unit test for olm
+   [\#777](https://github.com/matrix-org/matrix-js-sdk/pull/777)
+ * Set access_token and user_id after login in with username and password.
+   [\#778](https://github.com/matrix-org/matrix-js-sdk/pull/778)
+ * Add function to get currently joined rooms.
+   [\#779](https://github.com/matrix-org/matrix-js-sdk/pull/779)
+ * Remove the request-only stuff we don't need anymore
+   [\#775](https://github.com/matrix-org/matrix-js-sdk/pull/775)
+ * Manually construct query strings for browser-request instances
+   [\#770](https://github.com/matrix-org/matrix-js-sdk/pull/770)
+ * Fix: correctly check for crypto being present
+   [\#769](https://github.com/matrix-org/matrix-js-sdk/pull/769)
+ * Update babel-eslint to 8.1.1
+   [\#768](https://github.com/matrix-org/matrix-js-sdk/pull/768)
+ * Support `request` in the browser and support supplying servers to try in
+   joinRoom()
+   [\#764](https://github.com/matrix-org/matrix-js-sdk/pull/764)
+ * loglevel should be a normal dependency
+   [\#767](https://github.com/matrix-org/matrix-js-sdk/pull/767)
+ * Stop devicelist when client is stopped
+   [\#766](https://github.com/matrix-org/matrix-js-sdk/pull/766)
+ * Update to WebAssembly-powered Olm
+   [\#743](https://github.com/matrix-org/matrix-js-sdk/pull/743)
+ * Logging lib. Fixes #332
+   [\#763](https://github.com/matrix-org/matrix-js-sdk/pull/763)
+ * Use new stop() method on matrix-mock-request
+   [\#765](https://github.com/matrix-org/matrix-js-sdk/pull/765)
+
+Changes in [0.13.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.13.1) (2018-11-14)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.13.0...v0.13.1)
+
+ * Add function to get currently joined rooms.
+   [\#779](https://github.com/matrix-org/matrix-js-sdk/pull/779)
+
+Changes in [0.13.0](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.13.0) (2018-11-15)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.12.1...v0.13.0)
+
+BREAKING CHANGE
+----------------
+ * `MatrixClient::login` now sets client `access_token` and `user_id` following successful login with username and password.
+
+Changes in [0.12.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.12.1) (2018-10-29)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.12.1-rc.1...v0.12.1)
+
+ * No changes since rc.1
+
+Changes in [0.12.1-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.12.1-rc.1) (2018-10-24)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.12.0...v0.12.1-rc.1)
+
+ * Add repository type to package.json to make it valid
+   [\#762](https://github.com/matrix-org/matrix-js-sdk/pull/762)
+ * Add getMediaConfig()
+   [\#761](https://github.com/matrix-org/matrix-js-sdk/pull/761)
+ * add new examples, to be expanded into a post
+   [\#739](https://github.com/matrix-org/matrix-js-sdk/pull/739)
+
+Changes in [0.12.0](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.12.0) (2018-10-16)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.12.0-rc.1...v0.12.0)
+
+ * No changes since rc.1
+
+Changes in [0.12.0-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.12.0-rc.1) (2018-10-11)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.11.1...v0.12.0-rc.1)
+
+BREAKING CHANGES
+----------------
+ * If js-sdk finds data in the store that is incompatible with the options currently being used,
+   it will emit sync state ERROR with an error of type InvalidStoreError. It will also stop trying
+   to sync in this situation: the app must stop the client and then either clear the store or
+   change the options (in this case, enable or disable lazy loading of members) and then start
+   the client again.
+
+All Changes
+-----------
+
+ * never replace /sync'ed memberships with OOB ones
+   [\#760](https://github.com/matrix-org/matrix-js-sdk/pull/760)
+ * Don't fail to start up if lazy load check fails
+   [\#759](https://github.com/matrix-org/matrix-js-sdk/pull/759)
+ * Make e2e work on Edge
+   [\#754](https://github.com/matrix-org/matrix-js-sdk/pull/754)
+ * throw error with same name and message over idb worker boundary
+   [\#758](https://github.com/matrix-org/matrix-js-sdk/pull/758)
+ * Default to a room version of 1 when there is no room create event
+   [\#755](https://github.com/matrix-org/matrix-js-sdk/pull/755)
+ * Silence bluebird warnings
+   [\#757](https://github.com/matrix-org/matrix-js-sdk/pull/757)
+ * allow non-ff merge from release branch into master
+   [\#750](https://github.com/matrix-org/matrix-js-sdk/pull/750)
+ * Reject with the actual error on indexeddb error
+   [\#751](https://github.com/matrix-org/matrix-js-sdk/pull/751)
+ * Update mocha to v5
+   [\#744](https://github.com/matrix-org/matrix-js-sdk/pull/744)
+ * disable lazy loading for guests as they cant create filters
+   [\#748](https://github.com/matrix-org/matrix-js-sdk/pull/748)
+ * Revert "Add getMediaLimits to client"
+   [\#745](https://github.com/matrix-org/matrix-js-sdk/pull/745)
+
+Changes in [0.11.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.11.1) (2018-10-01)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.11.1-rc.1...v0.11.1)
+
+ * No changes since rc.1
+
+Changes in [0.11.1-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.11.1-rc.1) (2018-09-27)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.11.0...v0.11.1-rc.1)
+
+ * make usage of hub compatible with latest version (2.5)
+   [\#747](https://github.com/matrix-org/matrix-js-sdk/pull/747)
+ * Detect when lazy loading has been toggled in client.startClient
+   [\#746](https://github.com/matrix-org/matrix-js-sdk/pull/746)
+ * Add getMediaLimits to client
+   [\#644](https://github.com/matrix-org/matrix-js-sdk/pull/644)
+ * Split npm start into an init and watch script
+   [\#742](https://github.com/matrix-org/matrix-js-sdk/pull/742)
+ * Revert "room name should only take canonical alias into account"
+   [\#738](https://github.com/matrix-org/matrix-js-sdk/pull/738)
+ * fix display name disambiguation with LL
+   [\#737](https://github.com/matrix-org/matrix-js-sdk/pull/737)
+ * Introduce Room.myMembership event
+   [\#735](https://github.com/matrix-org/matrix-js-sdk/pull/735)
+ * room name should only take canonical alias into account
+   [\#733](https://github.com/matrix-org/matrix-js-sdk/pull/733)
+ * state events from context response were not wrapped in a MatrixEvent
+   [\#732](https://github.com/matrix-org/matrix-js-sdk/pull/732)
+ * Reduce amount of promises created when inserting members
+   [\#724](https://github.com/matrix-org/matrix-js-sdk/pull/724)
+ * dont wait for LL members to be stored to resolve the members
+   [\#726](https://github.com/matrix-org/matrix-js-sdk/pull/726)
+ * RoomState.members emitted with wrong argument order for OOB members
+   [\#728](https://github.com/matrix-org/matrix-js-sdk/pull/728)
+
+Changes in [0.11.0](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.11.0) (2018-09-10)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.11.0-rc.1...v0.11.0)
+
+BREAKING CHANGES
+----------------
+ * v0.11.0-rc.1 introduced some breaking changes - see the respective release notes.
+
+No changes since rc.1
+
+Changes in [0.11.0-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.11.0-rc.1) (2018-09-07)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.9...v0.11.0-rc.1)
+
+ * Support for lazy loading members. This should improve performance for
+   users who joined big rooms a lot. Pass to `lazyLoadMembers = true` option when calling `startClient`.
+
+BREAKING CHANGES
+----------------
+
+ * `MatrixClient::startClient` now returns a Promise. No method should be called on the client before that promise resolves. Before this method didn't return anything.
+ * A new `CATCHUP` sync state, emitted by `MatrixClient#"sync"` and returned by `MatrixClient::getSyncState()`, when doing initial sync after the `ERROR` state. See `MatrixClient` documentation for details.
+ * `RoomState::maySendEvent('m.room.message', userId)` & `RoomState::maySendMessage(userId)` do not check the membership of the user anymore, only the power level. To check if the syncing user is allowed to write in a room, use `Room::maySendMessage()` as `RoomState` is not always aware of the syncing user's membership anymore, in case lazy loading of members is enabled.
+
+All Changes
+-----------
+
+ * Only emit CATCHUP if recovering from conn error
+   [\#727](https://github.com/matrix-org/matrix-js-sdk/pull/727)
+ * Fix docstring for sync data.error
+   [\#725](https://github.com/matrix-org/matrix-js-sdk/pull/725)
+ * Re-apply "Don't rely on members to query if syncing user can post to room"
+   [\#723](https://github.com/matrix-org/matrix-js-sdk/pull/723)
+ * Revert "Don't rely on members to query if syncing user can post to room"
+   [\#721](https://github.com/matrix-org/matrix-js-sdk/pull/721)
+ * Don't rely on members to query if syncing user can post to room
+   [\#717](https://github.com/matrix-org/matrix-js-sdk/pull/717)
+ * Fixes for room.guessDMUserId
+   [\#719](https://github.com/matrix-org/matrix-js-sdk/pull/719)
+ * Fix filepanel also filtering main timeline with LL turned on.
+   [\#716](https://github.com/matrix-org/matrix-js-sdk/pull/716)
+ * Remove lazy loaded members when leaving room
+   [\#711](https://github.com/matrix-org/matrix-js-sdk/pull/711)
+ * Fix: show spinner again while recovering from connection error
+   [\#702](https://github.com/matrix-org/matrix-js-sdk/pull/702)
+ * Add method to query LL state in client
+   [\#714](https://github.com/matrix-org/matrix-js-sdk/pull/714)
+ * Fix: also load invited members when lazy loading members
+   [\#707](https://github.com/matrix-org/matrix-js-sdk/pull/707)
+ * Pass through function to discard megolm session
+   [\#704](https://github.com/matrix-org/matrix-js-sdk/pull/704)
+
+Changes in [0.10.9](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.9) (2018-09-03)
+==================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.9-rc.2...v0.10.9)
+
+ * No changes since rc.2
+
+Changes in [0.10.9-rc.2](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.9-rc.2) (2018-08-31)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.9-rc.1...v0.10.9-rc.2)
+
+ * Fix for "otherMember.getAvatarUrl is not a function"
+   [\#708](https://github.com/matrix-org/matrix-js-sdk/pull/708)
+
+Changes in [0.10.9-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.9-rc.1) (2018-08-30)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.8...v0.10.9-rc.1)
+
+ * Fix DM avatar
+   [\#706](https://github.com/matrix-org/matrix-js-sdk/pull/706)
+ * Lazy loading: avoid loading members at initial sync for e2e rooms
+   [\#699](https://github.com/matrix-org/matrix-js-sdk/pull/699)
+ * Improve setRoomEncryption guard against multiple m.room.encryption st…
+   [\#700](https://github.com/matrix-org/matrix-js-sdk/pull/700)
+ * Revert "Lazy loading: don't block on setting up room crypto"
+   [\#698](https://github.com/matrix-org/matrix-js-sdk/pull/698)
+ * Lazy loading: don't block on setting up room crypto
+   [\#696](https://github.com/matrix-org/matrix-js-sdk/pull/696)
+ * Add getVisibleRooms()
+   [\#695](https://github.com/matrix-org/matrix-js-sdk/pull/695)
+ * Add wrapper around getJoinedMemberCount()
+   [\#697](https://github.com/matrix-org/matrix-js-sdk/pull/697)
+ * Api to fetch events via /room/.../event/..
+   [\#694](https://github.com/matrix-org/matrix-js-sdk/pull/694)
+ * Support for room upgrades
+   [\#693](https://github.com/matrix-org/matrix-js-sdk/pull/693)
+ * Lazy loading of room members
+   [\#691](https://github.com/matrix-org/matrix-js-sdk/pull/691)
+
 Changes in [0.10.8](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.8) (2018-08-20)
 ==================================================================================================
 [Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.10.8-rc.1...v0.10.8)

README.md

@@ -30,9 +30,61 @@ In Node.js
     console.log("Public Rooms: %s", JSON.stringify(data));
   });
 ```
+
 See below for how to include libolm to enable end-to-end-encryption. Please check
 [the Node.js terminal app](examples/node) for a more complex example.
 
+To start the client:
+
+```javascript
+await client.startClient({initialSyncLimit: 10});
+```
+
+You can perform a call to `/sync` to get the current state of the client:
+
+```javascript
+client.once('sync', function(state, prevState, res) {
+    if(state === 'PREPARED') {
+        console.log("prepared");
+    } else {
+        console.log(state);
+        process.exit(1);
+    }
+});
+```
+
+To send a message:
+
+```javascript
+var content = {
+    "body": "message text",
+    "msgtype": "m.text"
+};
+client.sendEvent("roomId", "m.room.message", content, "", (err, res) => {
+    console.log(err);
+});
+```
+
+To listen for message events:
+
+```javascript
+client.on("Room.timeline", function(event, room, toStartOfTimeline) {
+  if (event.getType() !== "m.room.message") {
+    return; // only use messages
+  }
+  console.log(event.event.content.body);
+});
+```
+
+By default, the `matrix-js-sdk` client uses the `MatrixInMemoryStore` to store events as they are received. For example to iterate through the currently stored timeline for a room:
+
+```javascript
+Object.keys(client.store.rooms).forEach((roomId) => {
+  client.getRoom(roomId).timeline.forEach(t => {
+      console.log(t.event);
+  });
+});
+```
 
 What does this SDK do?
 ----------------------
@@ -267,13 +319,13 @@ To provide the Olm library in a browser application:
  
 To provide the Olm library in a node.js application:
 
- * ``npm install https://matrix.org/packages/npm/olm/olm-2.2.2.tgz``
+ * ``npm install https://matrix.org/packages/npm/olm/olm-3.0.0.tgz``
    (replace the URL with the latest version you want to use from
     https://matrix.org/packages/npm/olm/)
  * ``global.Olm = require('olm');`` *before* loading ``matrix-js-sdk``.
 
 If you want to package Olm as dependency for your node.js application, you
-can use ``npm install https://matrix.org/packages/npm/olm/olm-2.2.2.tgz 
+can use ``npm install https://matrix.org/packages/npm/olm/olm-3.0.0.tgz
 --save-optional`` (if your application also works without e2e crypto enabled)
 or ``--save`` (if it doesn't) to do so.
 

browser-index.js

@@ -1,5 +1,17 @@
 var matrixcs = require("./lib/matrix");
-matrixcs.request(require("browser-request"));
+const request = require('browser-request');
+const queryString = require('qs');
+
+matrixcs.request(function(opts, fn) {
+    // We manually fix the query string for browser-request because
+    // it doesn't correctly handle cases like ?via=one&via=two. Instead
+    // we mimic `request`'s query string interface to make it all work
+    // as expected.
+    // browser-request will happily take the constructed string as the
+    // query string without trying to modify it further.
+    opts.qs = queryString.stringify(opts.qs || {}, opts.qsStringifyOptions);
+    return request(opts, fn);
+});
 
 // just *accessing* indexedDB throws an exception in firefox with
 // indexeddb disabled.

examples/node/app.js

@@ -202,9 +202,9 @@ function printRoomList() {
             dateStr = new Date(msg.getTs()).toISOString().replace(
                 /T/, ' ').replace(/\..+/, '');
         }
-        var me = roomList[i].getMember(myUserId);
-        if (me) {
-            fmt = fmts[me.membership];
+        var myMembership = roomList[i].getMyMembership();
+        if (myMembership) {
+            fmt = fmts[myMembership];
         }
         var roomName = fixWidth(roomList[i].name, 25);
         print(

jenkins.sh

@@ -5,7 +5,7 @@ set -x
 export NVM_DIR="$HOME/.nvm"
 [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
 
-nvm use 6 || exit $?
+nvm use 10 || exit $?
 npm install || exit $?
 
 RC=0

package.json

@@ -1,6 +1,6 @@
 {
   "name": "matrix-js-sdk",
-  "version": "0.10.8",
+  "version": "0.14.2",
   "description": "Matrix Client-Server SDK for Javascript",
   "main": "index.js",
   "scripts": {
@@ -10,7 +10,9 @@
     "test": "npm run test:build && npm run test:run",
     "check": "npm run test:build && _mocha --recursive specbuild --colors",
     "gendoc": "babel --no-babelrc -d .jsdocbuild src && jsdoc -r .jsdocbuild -P package.json -R README.md -d .jsdoc",
-    "start": "babel -s -w -d lib src",
+    "start": "npm run start:init && npm run start:watch",
+    "start:watch": "babel -s -w --skip-initial-build -d lib src",
+    "start:init": "babel -s -d lib src",
     "clean": "rimraf lib dist",
     "build": "babel -s -d lib src && rimraf dist && mkdir dist && browserify -d browser-index.js | exorcist dist/browser-matrix.js.map > dist/browser-matrix.js && uglifyjs -c -m -o dist/browser-matrix.min.js --source-map dist/browser-matrix.min.js.map --in-source-map dist/browser-matrix.js.map dist/browser-matrix.js",
     "dist": "npm run build",
@@ -19,6 +21,7 @@
     "prepublish": "npm run clean && npm run build && git rev-parse HEAD > git-revision.txt"
   },
   "repository": {
+    "type": "git",
     "url": "https://github.com/matrix-org/matrix-js-sdk"
   },
   "keywords": [
@@ -53,16 +56,19 @@
     "babel-runtime": "^6.26.0",
     "bluebird": "^3.5.0",
     "browser-request": "^0.3.3",
+    "bs58": "^4.0.1",
     "content-type": "^1.0.2",
-    "request": "^2.53.0"
+    "loglevel": "1.6.1",
+    "qs": "^6.5.2",
+    "request": "^2.88.0"
   },
   "devDependencies": {
     "babel-cli": "^6.18.0",
-    "babel-eslint": "^7.1.1",
+    "babel-eslint": "^8.1.1",
     "babel-plugin-transform-async-to-bluebird": "^1.1.1",
     "babel-plugin-transform-runtime": "^6.23.0",
     "babel-preset-es2015": "^6.18.0",
-    "browserify": "^14.0.0",
+    "browserify": "^16.2.3",
     "browserify-shim": "^3.8.13",
     "eslint": "^3.13.1",
     "eslint-config-google": "^0.7.1",
@@ -71,14 +77,14 @@
     "istanbul": "^0.4.5",
     "jsdoc": "^3.5.5",
     "lolex": "^1.5.2",
-    "matrix-mock-request": "^1.2.0",
-    "mocha": "^3.2.0",
-    "mocha-jenkins-reporter": "^0.3.6",
+    "matrix-mock-request": "^1.2.2",
+    "mocha": "^5.2.0",
+    "mocha-jenkins-reporter": "^0.4.0",
     "rimraf": "^2.5.4",
     "source-map-support": "^0.4.11",
     "sourceify": "^0.1.0",
     "uglify-js": "^2.8.26",
-    "watchify": "^3.2.1"
+    "watchify": "^3.11.0"
   },
   "browserify": {
     "transform": [

release.sh

@@ -245,7 +245,7 @@ release_text=`mktemp`
 echo "$tag" > "${release_text}"
 echo >> "${release_text}"
 cat "${latest_changes}" >> "${release_text}"
-hub release create $hubflags $assets -f "${release_text}" "$tag"
+hub release create $hubflags $assets -F "${release_text}" "$tag"
 
 if [ $dodist -eq 0 ]; then
     rm -rf "$builddir"
@@ -281,7 +281,7 @@ fi
 echo "updating master branch"
 git checkout master
 git pull
-git merge --ff-only "$rel_branch"
+git merge "$rel_branch"
 
 # push master  and docs (if generated) to github
 git push origin master

spec/TestClient.js

@@ -102,9 +102,11 @@ TestClient.prototype.start = function() {
 
 /**
  * stop the client
+ * @return {Promise} Resolves once the mock http backend has finished all pending flushes
  */
 TestClient.prototype.stop = function() {
     this.client.stopClient();
+    return this.httpBackend.stop();
 };
 
 /**

spec/integ/devicelist-integ-spec.js

@@ -97,7 +97,7 @@ describe("DeviceList management:", function() {
     });
 
     afterEach(function() {
-        aliceTestClient.stop();
+        return aliceTestClient.stop();
     });
 
     it("Alice shouldn't do a second /query for non-e2e-capable devices", function() {

spec/integ/matrix-client-crypto.spec.js

@@ -410,10 +410,10 @@ describe("MatrixClient crypto", function() {
     });
 
     afterEach(function() {
-        aliTestClient.stop();
         aliTestClient.httpBackend.verifyNoOutstandingExpectation();
-        bobTestClient.stop();
         bobTestClient.httpBackend.verifyNoOutstandingExpectation();
+
+        return Promise.all([aliTestClient.stop(), bobTestClient.stop()]);
     });
 
     it("Bob uploads device keys", function() {

spec/integ/matrix-client-event-emitter.spec.js

@@ -30,6 +30,7 @@ describe("MatrixClient events", function() {
     afterEach(function() {
         httpBackend.verifyNoOutstandingExpectation();
         client.stopClient();
+        return httpBackend.stop();
     });
 
     describe("emissions", function() {

spec/integ/matrix-client-event-timeline.spec.js

@@ -111,6 +111,7 @@ describe("getEventTimeline support", function() {
         if (client) {
             client.stopClient();
         }
+        return httpBackend.stop();
     });
 
     it("timeline support must be enabled to work", function(done) {

spec/integ/matrix-client-methods.spec.js

@@ -41,6 +41,7 @@ describe("MatrixClient", function() {
 
     afterEach(function() {
         httpBackend.verifyNoOutstandingExpectation();
+        return httpBackend.stop();
     });
 
     describe("uploadContent", function() {
@@ -159,7 +160,7 @@ describe("MatrixClient", function() {
     describe("joinRoom", function() {
         it("should no-op if you've already joined a room", function() {
             const roomId = "!foo:bar";
-            const room = new Room(roomId);
+            const room = new Room(roomId, userId);
             room.addLiveEvents([
                 utils.mkMembership({
                     user: userId, room: roomId, mship: "join", event: true,

spec/integ/matrix-client-opts.spec.js

@@ -64,6 +64,7 @@ describe("MatrixClient opts", function() {
 
     afterEach(function() {
         httpBackend.verifyNoOutstandingExpectation();
+        return httpBackend.stop();
     });
 
     describe("without opts.store", function() {
@@ -94,7 +95,7 @@ describe("MatrixClient opts", function() {
             httpBackend.flush("/txn1", 1);
         });
 
-        it("should be able to sync / get new events", function(done) {
+        it("should be able to sync / get new events", async function() {
             const expectedEventTypes = [ // from /initialSync
                 "m.room.message", "m.room.name", "m.room.member", "m.room.member",
                 "m.room.create",
@@ -110,20 +111,16 @@ describe("MatrixClient opts", function() {
             httpBackend.when("GET", "/pushrules").respond(200, {});
             httpBackend.when("POST", "/filter").respond(200, { filter_id: "foo" });
             httpBackend.when("GET", "/sync").respond(200, syncData);
-            client.startClient();
-            httpBackend.flush("/pushrules", 1).then(function() {
-                return httpBackend.flush("/filter", 1);
-            }).then(function() {
-                return Promise.all([
-                    httpBackend.flush("/sync", 1),
-                    utils.syncPromise(client),
-                ]);
-            }).done(function() {
-                expect(expectedEventTypes.length).toEqual(
-                    0, "Expected to see event types: " + expectedEventTypes,
-                );
-                done();
-            });
+            await client.startClient();
+            await httpBackend.flush("/pushrules", 1);
+            await httpBackend.flush("/filter", 1);
+            await Promise.all([
+                httpBackend.flush("/sync", 1),
+                utils.syncPromise(client),
+            ]);
+            expect(expectedEventTypes.length).toEqual(
+                0, "Expected to see event types: " + expectedEventTypes,
+            );
         });
     });
 

spec/integ/matrix-client-retrying.spec.js

@@ -36,6 +36,7 @@ describe("MatrixClient retrying", function() {
 
     afterEach(function() {
         httpBackend.verifyNoOutstandingExpectation();
+        return httpBackend.stop();
     });
 
     xit("should retry according to MatrixScheduler.retryFn", function() {

spec/integ/matrix-client-room-timeline.spec.js

@@ -130,6 +130,7 @@ describe("MatrixClient room timelines", function() {
     afterEach(function() {
         httpBackend.verifyNoOutstandingExpectation();
         client.stopClient();
+        return httpBackend.stop();
     });
 
     describe("local echo events", function() {

spec/integ/matrix-client-syncing.spec.js

@@ -38,6 +38,7 @@ describe("MatrixClient syncing", function() {
     afterEach(function() {
         httpBackend.verifyNoOutstandingExpectation();
         client.stopClient();
+        return httpBackend.stop();
     });
 
     describe("startClient", function() {

spec/integ/megolm-integ.spec.js

@@ -296,7 +296,7 @@ describe("megolm", function() {
     });
 
     afterEach(function() {
-        aliceTestClient.stop();
+        return aliceTestClient.stop();
     });
 
     it("Alice receives a megolm message", function() {
@@ -817,8 +817,14 @@ describe("megolm", function() {
                 };
             });
 
+            // Grab the event that we'll need to resend
+            const room = aliceTestClient.client.getRoom(ROOM_ID);
+            const pendingEvents = room.getPendingEvents();
+            expect(pendingEvents.length).toEqual(1);
+            const unsentEvent = pendingEvents[0];
+
             return Promise.all([
-                aliceTestClient.client.sendTextMessage(ROOM_ID, 'test'),
+                aliceTestClient.client.resendEvent(unsentEvent, room),
 
                 // the crypto stuff can take a while, so give the requests a whole second.
                 aliceTestClient.httpBackend.flushAllExpected({

spec/unit/crypto.spec.js

@@ -1,20 +1,118 @@
-
-"use strict";
 import 'source-map-support/register';
+import Crypto from '../../lib/crypto';
+import expect from 'expect';
+
+import WebStorageSessionStore from '../../lib/store/session/webstorage';
+import MemoryCryptoStore from '../../lib/crypto/store/memory-crypto-store.js';
+import MockStorageApi from '../MockStorageApi';
+
+const EventEmitter = require("events").EventEmitter;
 
 const sdk = require("../..");
-let Crypto;
-if (sdk.CRYPTO_ENABLED) {
-    Crypto = require("../../lib/crypto");
-}
 
-import expect from 'expect';
+const Olm = global.Olm;
 
 describe("Crypto", function() {
     if (!sdk.CRYPTO_ENABLED) {
         return;
     }
+
+    beforeEach(function(done) {
+        Olm.init().then(done);
+    });
+
     it("Crypto exposes the correct olm library version", function() {
-        expect(Crypto.getOlmVersion()[0]).toEqual(2);
+        expect(Crypto.getOlmVersion()[0]).toEqual(3);
+    });
+
+
+    describe('Session management', function() {
+        const otkResponse = {
+            one_time_keys: {
+                '@alice:home.server': {
+                    aliceDevice: {
+                        'signed_curve25519:FLIBBLE': {
+                            key: 'YmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmI',
+                            signatures: {
+                                '@alice:home.server': {
+                                    'ed25519:aliceDevice': 'totally a valid signature',
+                                },
+                            },
+                        },
+                    },
+                },
+            },
+        };
+        let crypto;
+        let mockBaseApis;
+        let mockRoomList;
+
+        let fakeEmitter;
+
+        beforeEach(async function() {
+            const mockStorage = new MockStorageApi();
+            const sessionStore = new WebStorageSessionStore(mockStorage);
+            const cryptoStore = new MemoryCryptoStore(mockStorage);
+
+            cryptoStore.storeEndToEndDeviceData({
+                devices: {
+                    '@bob:home.server': {
+                        'BOBDEVICE': {
+                            keys: {
+                                'curve25519:BOBDEVICE': 'this is a key',
+                            },
+                        },
+                    },
+                },
+                trackingStatus: {},
+            });
+
+            mockBaseApis = {
+                sendToDevice: expect.createSpy(),
+                getKeyBackupVersion: expect.createSpy(),
+            };
+            mockRoomList = {};
+
+            fakeEmitter = new EventEmitter();
+
+            crypto = new Crypto(
+                mockBaseApis,
+                sessionStore,
+                "@alice:home.server",
+                "FLIBBLE",
+                sessionStore,
+                cryptoStore,
+                mockRoomList,
+            );
+            crypto.registerEventHandlers(fakeEmitter);
+            await crypto.init();
+        });
+
+        afterEach(async function() {
+            await crypto.stop();
+        });
+
+        it("restarts wedged Olm sessions", async function() {
+            const prom = new Promise((resolve) => {
+                mockBaseApis.claimOneTimeKeys = function() {
+                    resolve();
+                    return otkResponse;
+                };
+            });
+
+            fakeEmitter.emit('toDeviceEvent', {
+                getType: expect.createSpy().andReturn('m.room.message'),
+                getContent: expect.createSpy().andReturn({
+                    msgtype: 'm.bad.encrypted',
+                }),
+                getWireContent: expect.createSpy().andReturn({
+                    algorithm: 'm.olm.v1.curve25519-aes-sha2',
+                    sender_key: 'this is a key',
+                }),
+                getSender: expect.createSpy().andReturn('@bob:home.server'),
+            });
+
+            await prom;
+        });
     });
 });

spec/unit/crypto/DeviceList.spec.js

@@ -59,16 +59,25 @@ describe('DeviceList', function() {
     let downloadSpy;
     let sessionStore;
     let cryptoStore;
+    let deviceLists = [];
 
     beforeEach(function() {
         testUtils.beforeEach(this); // eslint-disable-line no-invalid-this
 
+        deviceLists = [];
+
         downloadSpy = expect.createSpy();
         const mockStorage = new MockStorageApi();
         sessionStore = new WebStorageSessionStore(mockStorage);
         cryptoStore = new MemoryCryptoStore();
     });
 
+    afterEach(function() {
+        for (const dl of deviceLists) {
+            dl.stop();
+        }
+    });
+
     function createTestDeviceList() {
         const baseApis = {
             downloadKeysForUsers: downloadSpy,
@@ -76,7 +85,9 @@ describe('DeviceList', function() {
         const mockOlm = {
             verifySignature: function(key, message, signature) {},
         };
-        return new DeviceList(baseApis, cryptoStore, sessionStore, mockOlm);
+        const dl = new DeviceList(baseApis, cryptoStore, sessionStore, mockOlm);
+        deviceLists.push(dl);
+        return dl;
     }
 
     it("should successfully download and store device keys", function() {

spec/unit/crypto/algorithms/megolm.spec.js

@@ -13,20 +13,17 @@ import WebStorageSessionStore from '../../../../lib/store/session/webstorage';
 import MemoryCryptoStore from '../../../../lib/crypto/store/memory-crypto-store.js';
 import MockStorageApi from '../../../MockStorageApi';
 import testUtils from '../../../test-utils';
-
-// Crypto and OlmDevice won't import unless we have global.Olm
-let OlmDevice;
-let Crypto;
-if (global.Olm) {
-    OlmDevice = require('../../../../lib/crypto/OlmDevice');
-    Crypto = require('../../../../lib/crypto');
-}
+import OlmDevice from '../../../../lib/crypto/OlmDevice';
+import Crypto from '../../../../lib/crypto';
 
 const MatrixEvent = sdk.MatrixEvent;
 const MegolmDecryption = algorithms.DECRYPTION_CLASSES['m.megolm.v1.aes-sha2'];
+const MegolmEncryption = algorithms.ENCRYPTION_CLASSES['m.megolm.v1.aes-sha2'];
 
 const ROOM_ID = '!ROOM:ID';
 
+const Olm = global.Olm;
+
 describe("MegolmDecryption", function() {
     if (!global.Olm) {
         console.warn('Not running megolm unit tests: libolm not present');
@@ -38,9 +35,11 @@ describe("MegolmDecryption", function() {
     let mockCrypto;
     let mockBaseApis;
 
-    beforeEach(function() {
+    beforeEach(async function() {
         testUtils.beforeEach(this); // eslint-disable-line no-invalid-this
 
+        await Olm.init();
+
         mockCrypto = testUtils.mock(Crypto, 'Crypto');
         mockBaseApis = {};
 
@@ -69,7 +68,7 @@ describe("MegolmDecryption", function() {
 
     describe('receives some keys:', function() {
         let groupSession;
-        beforeEach(function() {
+        beforeEach(async function() {
             groupSession = new global.Olm.OutboundGroupSession();
             groupSession.create();
 
@@ -98,7 +97,7 @@ describe("MegolmDecryption", function() {
                 },
             };
 
-            return event.attemptDecryption(mockCrypto).then(() => {
+            await event.attemptDecryption(mockCrypto).then(() => {
                 megolmDecryption.onRoomKeyEvent(event);
             });
         });
@@ -266,5 +265,92 @@ describe("MegolmDecryption", function() {
                 // test is successful if no exception is thrown
             });
         });
+
+        it("re-uses sessions for sequential messages", async function() {
+            const mockStorage = new MockStorageApi();
+            const sessionStore = new WebStorageSessionStore(mockStorage);
+            const cryptoStore = new MemoryCryptoStore(mockStorage);
+
+            const olmDevice = new OlmDevice(sessionStore, cryptoStore);
+            olmDevice.verifySignature = expect.createSpy();
+            await olmDevice.init();
+
+            mockBaseApis.claimOneTimeKeys = expect.createSpy().andReturn(Promise.resolve({
+                one_time_keys: {
+                    '@alice:home.server': {
+                        aliceDevice: {
+                            'signed_curve25519:flooble': {
+                                key: 'YmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmI',
+                                signatures: {
+                                    '@alice:home.server': {
+                                        'ed25519:aliceDevice': 'totally valid',
+                                    },
+                                },
+                            },
+                        },
+                    },
+                },
+            }));
+            mockBaseApis.sendToDevice = expect.createSpy().andReturn(Promise.resolve());
+
+            mockCrypto.downloadKeys.andReturn(Promise.resolve({
+                '@alice:home.server': {
+                    aliceDevice: {
+                        deviceId: 'aliceDevice',
+                        isBlocked: expect.createSpy().andReturn(false),
+                        isUnverified: expect.createSpy().andReturn(false),
+                        getIdentityKey: expect.createSpy().andReturn(
+                            'YWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWE',
+                        ),
+                        getFingerprint: expect.createSpy().andReturn(''),
+                    },
+                },
+            }));
+
+            const megolmEncryption = new MegolmEncryption({
+                userId: '@user:id',
+                crypto: mockCrypto,
+                olmDevice: olmDevice,
+                baseApis: mockBaseApis,
+                roomId: ROOM_ID,
+                config: {
+                    rotation_period_ms: 9999999999999,
+                },
+            });
+            const mockRoom = {
+                getEncryptionTargetMembers: expect.createSpy().andReturn(
+                    [{userId: "@alice:home.server"}],
+                ),
+                getBlacklistUnverifiedDevices: expect.createSpy().andReturn(false),
+            };
+            const ct1 = await megolmEncryption.encryptMessage(mockRoom, "a.fake.type", {
+                body: "Some text",
+            });
+            expect(mockRoom.getEncryptionTargetMembers).toHaveBeenCalled();
+
+            // this should have claimed a key for alice as it's starting a new session
+            expect(mockBaseApis.claimOneTimeKeys).toHaveBeenCalled(
+                [['@alice:home.server', 'aliceDevice']], 'signed_curve25519',
+            );
+            expect(mockCrypto.downloadKeys).toHaveBeenCalledWith(
+                ['@alice:home.server'], false,
+            );
+            expect(mockBaseApis.sendToDevice).toHaveBeenCalled();
+            expect(mockBaseApis.claimOneTimeKeys).toHaveBeenCalled(
+                [['@alice:home.server', 'aliceDevice']], 'signed_curve25519',
+            );
+
+            mockBaseApis.claimOneTimeKeys.reset();
+
+            const ct2 = await megolmEncryption.encryptMessage(mockRoom, "a.fake.type", {
+                body: "Some more text",
+            });
+
+            // this should *not* have claimed a key as it should be using the same session
+            expect(mockBaseApis.claimOneTimeKeys).toNotHaveBeenCalled();
+
+            // likewise they should show the same session ID
+            expect(ct2.session_id).toEqual(ct1.session_id);
+        });
     });
 });

spec/unit/crypto/algorithms/olm.spec.js

@@ -0,0 +1,90 @@
+/*
+Copyright 2018 New Vector Ltd
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+try {
+    global.Olm = require('olm');
+} catch (e) {
+    console.warn("unable to run megolm tests: libolm not available");
+}
+
+import expect from 'expect';
+import WebStorageSessionStore from '../../../../lib/store/session/webstorage';
+import MemoryCryptoStore from '../../../../lib/crypto/store/memory-crypto-store.js';
+import MockStorageApi from '../../../MockStorageApi';
+import testUtils from '../../../test-utils';
+
+import OlmDevice from '../../../../lib/crypto/OlmDevice';
+
+function makeOlmDevice() {
+    const mockStorage = new MockStorageApi();
+    const sessionStore = new WebStorageSessionStore(mockStorage);
+    const cryptoStore = new MemoryCryptoStore(mockStorage);
+    const olmDevice = new OlmDevice(sessionStore, cryptoStore);
+    return olmDevice;
+}
+
+async function setupSession(initiator, opponent) {
+    await opponent.generateOneTimeKeys(1);
+    const keys = await opponent.getOneTimeKeys();
+    const firstKey = Object.values(keys['curve25519'])[0];
+
+    const sid = await initiator.createOutboundSession(
+        opponent.deviceCurve25519Key, firstKey,
+    );
+    return sid;
+}
+
+describe("OlmDecryption", function() {
+    if (!global.Olm) {
+        console.warn('Not running megolm unit tests: libolm not present');
+        return;
+    }
+
+    let aliceOlmDevice;
+    let bobOlmDevice;
+
+    beforeEach(async function() {
+        testUtils.beforeEach(this); // eslint-disable-line no-invalid-this
+
+        await global.Olm.init();
+
+        aliceOlmDevice = makeOlmDevice();
+        bobOlmDevice = makeOlmDevice();
+        await aliceOlmDevice.init();
+        await bobOlmDevice.init();
+    });
+
+    describe('olm', function() {
+        it("can decrypt messages", async function() {
+            const sid = await setupSession(aliceOlmDevice, bobOlmDevice);
+
+            const ciphertext = await aliceOlmDevice.encryptMessage(
+                bobOlmDevice.deviceCurve25519Key,
+                sid,
+                "The olm or proteus is an aquatic salamander in the family Proteidae",
+            );
+
+            const result = await bobOlmDevice.createInboundSession(
+                aliceOlmDevice.deviceCurve25519Key,
+                ciphertext.type,
+                ciphertext.body,
+            );
+            expect(result.payload).toEqual(
+                "The olm or proteus is an aquatic salamander in the family Proteidae",
+            );
+        });
+    });
+});

spec/unit/crypto/backup.spec.js

@@ -0,0 +1,470 @@
+/*
+Copyright 2018 New Vector Ltd
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+try {
+    global.Olm = require('olm');
+} catch (e) {
+    console.warn("unable to run megolm backup tests: libolm not available");
+}
+
+import expect from 'expect';
+import Promise from 'bluebird';
+
+import sdk from '../../..';
+import algorithms from '../../../lib/crypto/algorithms';
+import WebStorageSessionStore from '../../../lib/store/session/webstorage';
+import MemoryCryptoStore from '../../../lib/crypto/store/memory-crypto-store.js';
+import MockStorageApi from '../../MockStorageApi';
+import testUtils from '../../test-utils';
+
+import OlmDevice from '../../../lib/crypto/OlmDevice';
+import Crypto from '../../../lib/crypto';
+
+const Olm = global.Olm;
+
+const MatrixClient = sdk.MatrixClient;
+const MatrixEvent = sdk.MatrixEvent;
+const MegolmDecryption = algorithms.DECRYPTION_CLASSES['m.megolm.v1.aes-sha2'];
+
+const ROOM_ID = '!ROOM:ID';
+
+const SESSION_ID = 'o+21hSjP+mgEmcfdslPsQdvzWnkdt0Wyo00Kp++R8Kc';
+const ENCRYPTED_EVENT = new MatrixEvent({
+    type: 'm.room.encrypted',
+    room_id: '!ROOM:ID',
+    content: {
+        algorithm: 'm.megolm.v1.aes-sha2',
+        sender_key: 'SENDER_CURVE25519',
+        session_id: SESSION_ID,
+        ciphertext: 'AwgAEjD+VwXZ7PoGPRS/H4kwpAsMp/g+WPvJVtPEKE8fmM9IcT/N'
+            + 'CiwPb8PehecDKP0cjm1XO88k6Bw3D17aGiBHr5iBoP7oSw8CXULXAMTkBl'
+            + 'mkufRQq2+d0Giy1s4/Cg5n13jSVrSb2q7VTSv1ZHAFjUCsLSfR0gxqcQs',
+    },
+    event_id: '$event1',
+    origin_server_ts: 1507753886000,
+});
+
+const KEY_BACKUP_DATA = {
+    first_message_index: 0,
+    forwarded_count: 0,
+    is_verified: false,
+    session_data: {
+        ciphertext: '2z2M7CZ+azAiTHN1oFzZ3smAFFt+LEOYY6h3QO3XXGdw'
+            + '6YpNn/gpHDO6I/rgj1zNd4FoTmzcQgvKdU8kN20u5BWRHxaHTZ'
+            + 'Slne5RxE6vUdREsBgZePglBNyG0AogR/PVdcrv/v18Y6rLM5O9'
+            + 'SELmwbV63uV9Kuu/misMxoqbuqEdG7uujyaEKtjlQsJ5MGPQOy'
+            + 'Syw7XrnesSwF6XWRMxcPGRV0xZr3s9PI350Wve3EncjRgJ9IGF'
+            + 'ru1bcptMqfXgPZkOyGvrphHoFfoK7nY3xMEHUiaTRfRIjq8HNV'
+            + '4o8QY1qmWGnxNBQgOlL8MZlykjg3ULmQ3DtFfQPj/YYGS3jzxv'
+            + 'C+EBjaafmsg+52CTeK3Rswu72PX450BnSZ1i3If4xWAUKvjTpe'
+            + 'Ug5aDLqttOv1pITolTJDw5W/SD+b5rjEKg1CFCHGEGE9wwV3Nf'
+            + 'QHVCQL+dfpd7Or0poy4dqKMAi3g0o3Tg7edIF8d5rREmxaALPy'
+            + 'iie8PHD8mj/5Y0GLqrac4CD6+Mop7eUTzVovprjg',
+        mac: '5lxYBHQU80M',
+        ephemeral: '/Bn0A4UMFwJaDDvh0aEk1XZj3k1IfgCxgFY9P9a0b14',
+    },
+};
+
+function makeTestClient(sessionStore, cryptoStore) {
+    const scheduler = [
+        "getQueueForEvent", "queueEvent", "removeEventFromQueue",
+        "setProcessFunction",
+    ].reduce((r, k) => {r[k] = expect.createSpy(); return r;}, {});
+    const store = [
+        "getRoom", "getRooms", "getUser", "getSyncToken", "scrollback",
+        "save", "wantsSave", "setSyncToken", "storeEvents", "storeRoom",
+        "storeUser", "getFilterIdByName", "setFilterIdByName", "getFilter",
+        "storeFilter", "getSyncAccumulator", "startup", "deleteAllData",
+    ].reduce((r, k) => {r[k] = expect.createSpy(); return r;}, {});
+    store.getSavedSync = expect.createSpy().andReturn(Promise.resolve(null));
+    store.getSavedSyncToken = expect.createSpy().andReturn(Promise.resolve(null));
+    store.setSyncData = expect.createSpy().andReturn(Promise.resolve(null));
+    return new MatrixClient({
+        baseUrl: "https://my.home.server",
+        idBaseUrl: "https://identity.server",
+        accessToken: "my.access.token",
+        request: function() {}, // NOP
+        store: store,
+        scheduler: scheduler,
+        userId: "@alice:bar",
+        deviceId: "device",
+        sessionStore: sessionStore,
+        cryptoStore: cryptoStore,
+    });
+}
+
+describe("MegolmBackup", function() {
+    if (!global.Olm) {
+        console.warn('Not running megolm backup unit tests: libolm not present');
+        return;
+    }
+
+    let olmDevice;
+    let mockOlmLib;
+    let mockCrypto;
+    let mockStorage;
+    let sessionStore;
+    let cryptoStore;
+    let megolmDecryption;
+    beforeEach(async function() {
+        await Olm.init();
+        testUtils.beforeEach(this); // eslint-disable-line no-invalid-this
+
+        mockCrypto = testUtils.mock(Crypto, 'Crypto');
+        mockCrypto.backupKey = new Olm.PkEncryption();
+        mockCrypto.backupKey.set_recipient_key(
+            "hSDwCYkwp1R0i33ctD73Wg2/Og0mOBr066SpjqqbTmo",
+        );
+        mockCrypto.backupInfo = {
+            algorithm: "m.megolm_backup.v1",
+            version: 1,
+            auth_data: {
+                public_key: "hSDwCYkwp1R0i33ctD73Wg2/Og0mOBr066SpjqqbTmo",
+            },
+        };
+
+        mockStorage = new MockStorageApi();
+        sessionStore = new WebStorageSessionStore(mockStorage);
+        cryptoStore = new MemoryCryptoStore(mockStorage);
+
+        olmDevice = new OlmDevice(sessionStore, cryptoStore);
+
+        // we stub out the olm encryption bits
+        mockOlmLib = {};
+        mockOlmLib.ensureOlmSessionsForDevices = expect.createSpy();
+        mockOlmLib.encryptMessageForDevice =
+            expect.createSpy().andReturn(Promise.resolve());
+    });
+
+    describe("backup", function() {
+        let mockBaseApis;
+        let realSetTimeout;
+
+        beforeEach(function() {
+            mockBaseApis = {};
+
+            megolmDecryption = new MegolmDecryption({
+                userId: '@user:id',
+                crypto: mockCrypto,
+                olmDevice: olmDevice,
+                baseApis: mockBaseApis,
+                roomId: ROOM_ID,
+            });
+
+            megolmDecryption.olmlib = mockOlmLib;
+
+            // clobber the setTimeout function to run 100x faster.
+            // ideally we would use lolex, but we have no oportunity
+            // to tick the clock between the first try and the retry.
+            realSetTimeout = global.setTimeout;
+            global.setTimeout = function(f, n) {
+                return realSetTimeout(f, n/100);
+            };
+        });
+
+        afterEach(function() {
+            global.setTimeout = realSetTimeout;
+        });
+
+        it('automatically calls the key back up', function() {
+            const groupSession = new Olm.OutboundGroupSession();
+            groupSession.create();
+
+            // construct a fake decrypted key event via the use of a mocked
+            // 'crypto' implementation.
+            const event = new MatrixEvent({
+                type: 'm.room.encrypted',
+            });
+            const decryptedData = {
+                clearEvent: {
+                    type: 'm.room_key',
+                    content: {
+                        algorithm: 'm.megolm.v1.aes-sha2',
+                        room_id: ROOM_ID,
+                        session_id: groupSession.session_id(),
+                        session_key: groupSession.session_key(),
+                    },
+                },
+                senderCurve25519Key: "SENDER_CURVE25519",
+                claimedEd25519Key: "SENDER_ED25519",
+            };
+
+            mockCrypto.decryptEvent = function() {
+                return Promise.resolve(decryptedData);
+            };
+            mockCrypto.cancelRoomKeyRequest = function() {};
+
+            mockCrypto.backupGroupSession = expect.createSpy();
+
+            return event.attemptDecryption(mockCrypto).then(() => {
+                return megolmDecryption.onRoomKeyEvent(event);
+            }).then(() => {
+                expect(mockCrypto.backupGroupSession).toHaveBeenCalled();
+            });
+        });
+
+        it('sends backups to the server', function() {
+            const groupSession = new Olm.OutboundGroupSession();
+            groupSession.create();
+            const ibGroupSession = new Olm.InboundGroupSession();
+            ibGroupSession.create(groupSession.session_key());
+
+            const client = makeTestClient(sessionStore, cryptoStore);
+
+            megolmDecryption = new MegolmDecryption({
+                userId: '@user:id',
+                crypto: mockCrypto,
+                olmDevice: olmDevice,
+                baseApis: client,
+                roomId: ROOM_ID,
+            });
+
+            megolmDecryption.olmlib = mockOlmLib;
+
+            return client.initCrypto()
+                .then(() => {
+                    return cryptoStore.doTxn(
+                        "readwrite",
+                        [cryptoStore.STORE_SESSION],
+                        (txn) => {
+                            cryptoStore.addEndToEndInboundGroupSession(
+                                "F0Q2NmyJNgUVj9DGsb4ZQt3aVxhVcUQhg7+gvW0oyKI",
+                                groupSession.session_id(),
+                                {
+                                    forwardingCurve25519KeyChain: undefined,
+                                    keysClaimed: {
+                                        ed25519: "SENDER_ED25519",
+                                    },
+                                    room_id: ROOM_ID,
+                                    session: ibGroupSession.pickle(olmDevice._pickleKey),
+                                },
+                                txn);
+                        });
+                })
+                .then(() => {
+                    client.enableKeyBackup({
+                        algorithm: "m.megolm_backup.v1",
+                        version: 1,
+                        auth_data: {
+                            public_key: "hSDwCYkwp1R0i33ctD73Wg2/Og0mOBr066SpjqqbTmo",
+                        },
+                    });
+                    let numCalls = 0;
+                    return new Promise((resolve, reject) => {
+                        client._http.authedRequest = function(
+                            callback, method, path, queryParams, data, opts,
+                        ) {
+                            ++numCalls;
+                            expect(numCalls).toBeLessThanOrEqualTo(1);
+                            if (numCalls >= 2) {
+                                // exit out of retry loop if there's something wrong
+                                reject(new Error("authedRequest called too many timmes"));
+                                return Promise.resolve({});
+                            }
+                            expect(method).toBe("PUT");
+                            expect(path).toBe("/room_keys/keys");
+                            expect(queryParams.version).toBe(1);
+                            expect(data.rooms[ROOM_ID].sessions).toExist();
+                            expect(data.rooms[ROOM_ID].sessions).toIncludeKey(
+                                groupSession.session_id(),
+                            );
+                            resolve();
+                            return Promise.resolve({});
+                        };
+                        client._crypto.backupGroupSession(
+                            "roomId",
+                            "F0Q2NmyJNgUVj9DGsb4ZQt3aVxhVcUQhg7+gvW0oyKI",
+                            [],
+                            groupSession.session_id(),
+                            groupSession.session_key(),
+                        );
+                    }).then(() => {
+                        expect(numCalls).toBe(1);
+                    });
+                });
+        });
+
+        it('retries when a backup fails', function() {
+            const groupSession = new Olm.OutboundGroupSession();
+            groupSession.create();
+            const ibGroupSession = new Olm.InboundGroupSession();
+            ibGroupSession.create(groupSession.session_key());
+
+            const scheduler = [
+                "getQueueForEvent", "queueEvent", "removeEventFromQueue",
+                "setProcessFunction",
+            ].reduce((r, k) => {r[k] = expect.createSpy(); return r;}, {});
+            const store = [
+                "getRoom", "getRooms", "getUser", "getSyncToken", "scrollback",
+                "save", "wantsSave", "setSyncToken", "storeEvents", "storeRoom",
+                "storeUser", "getFilterIdByName", "setFilterIdByName", "getFilter",
+                "storeFilter", "getSyncAccumulator", "startup", "deleteAllData",
+            ].reduce((r, k) => {r[k] = expect.createSpy(); return r;}, {});
+            store.getSavedSync = expect.createSpy().andReturn(Promise.resolve(null));
+            store.getSavedSyncToken = expect.createSpy().andReturn(Promise.resolve(null));
+            store.setSyncData = expect.createSpy().andReturn(Promise.resolve(null));
+            const client = new MatrixClient({
+                baseUrl: "https://my.home.server",
+                idBaseUrl: "https://identity.server",
+                accessToken: "my.access.token",
+                request: function() {}, // NOP
+                store: store,
+                scheduler: scheduler,
+                userId: "@alice:bar",
+                deviceId: "device",
+                sessionStore: sessionStore,
+                cryptoStore: cryptoStore,
+            });
+
+            megolmDecryption = new MegolmDecryption({
+                userId: '@user:id',
+                crypto: mockCrypto,
+                olmDevice: olmDevice,
+                baseApis: client,
+                roomId: ROOM_ID,
+            });
+
+            megolmDecryption.olmlib = mockOlmLib;
+
+            return client.initCrypto()
+                .then(() => {
+                    return cryptoStore.doTxn(
+                        "readwrite",
+                        [cryptoStore.STORE_SESSION],
+                        (txn) => {
+                            cryptoStore.addEndToEndInboundGroupSession(
+                                "F0Q2NmyJNgUVj9DGsb4ZQt3aVxhVcUQhg7+gvW0oyKI",
+                                groupSession.session_id(),
+                                {
+                                    forwardingCurve25519KeyChain: undefined,
+                                    keysClaimed: {
+                                        ed25519: "SENDER_ED25519",
+                                    },
+                                    room_id: ROOM_ID,
+                                    session: ibGroupSession.pickle(olmDevice._pickleKey),
+                                },
+                                txn);
+                        });
+                })
+                .then(() => {
+                    client.enableKeyBackup({
+                        algorithm: "foobar",
+                        version: 1,
+                        auth_data: {
+                            public_key: "hSDwCYkwp1R0i33ctD73Wg2/Og0mOBr066SpjqqbTmo",
+                        },
+                    });
+                    let numCalls = 0;
+                    return new Promise((resolve, reject) => {
+                        client._http.authedRequest = function(
+                            callback, method, path, queryParams, data, opts,
+                        ) {
+                            ++numCalls;
+                            expect(numCalls).toBeLessThanOrEqualTo(2);
+                            if (numCalls >= 3) {
+                                // exit out of retry loop if there's something wrong
+                                reject(new Error("authedRequest called too many timmes"));
+                                return Promise.resolve({});
+                            }
+                            expect(method).toBe("PUT");
+                            expect(path).toBe("/room_keys/keys");
+                            expect(queryParams.version).toBe(1);
+                            expect(data.rooms[ROOM_ID].sessions).toExist();
+                            expect(data.rooms[ROOM_ID].sessions).toIncludeKey(
+                                groupSession.session_id(),
+                            );
+                            if (numCalls > 1) {
+                                resolve();
+                                return Promise.resolve({});
+                            } else {
+                                return Promise.reject(
+                                    new Error("this is an expected failure"),
+                                );
+                            }
+                        };
+                        client._crypto.backupGroupSession(
+                            "roomId",
+                            "F0Q2NmyJNgUVj9DGsb4ZQt3aVxhVcUQhg7+gvW0oyKI",
+                            [],
+                            groupSession.session_id(),
+                            groupSession.session_key(),
+                        );
+                    }).then(() => {
+                        expect(numCalls).toBe(2);
+                    });
+                });
+        });
+    });
+
+    describe("restore", function() {
+        let client;
+
+        beforeEach(function() {
+            client = makeTestClient(sessionStore, cryptoStore);
+
+            megolmDecryption = new MegolmDecryption({
+                userId: '@user:id',
+                crypto: mockCrypto,
+                olmDevice: olmDevice,
+                baseApis: client,
+                roomId: ROOM_ID,
+            });
+
+            megolmDecryption.olmlib = mockOlmLib;
+
+            return client.initCrypto();
+        });
+
+        afterEach(function() {
+            client.stopClient();
+        });
+
+        it('can restore from backup', function() {
+            client._http.authedRequest = function() {
+                return Promise.resolve(KEY_BACKUP_DATA);
+            };
+            return client.restoreKeyBackupWithRecoveryKey(
+                "EsTc LW2K PGiF wKEA 3As5 g5c4 BXwk qeeJ ZJV8 Q9fu gUMN UE4d",
+                ROOM_ID,
+                SESSION_ID,
+            ).then(() => {
+                return megolmDecryption.decryptEvent(ENCRYPTED_EVENT);
+            }).then((res) => {
+                expect(res.clearEvent.content).toEqual('testytest');
+            });
+        });
+
+        it('can restore backup by room', function() {
+            client._http.authedRequest = function() {
+                return Promise.resolve({
+                    rooms: {
+                        [ROOM_ID]: {
+                            sessions: {
+                                [SESSION_ID]: KEY_BACKUP_DATA,
+                            },
+                        },
+                    },
+                });
+            };
+            return client.restoreKeyBackupWithRecoveryKey(
+                "EsTc LW2K PGiF wKEA 3As5 g5c4 BXwk qeeJ ZJV8 Q9fu gUMN UE4d",
+            ).then(() => {
+                return megolmDecryption.decryptEvent(ENCRYPTED_EVENT);
+            }).then((res) => {
+                expect(res.clearEvent.content).toEqual('testytest');
+            });
+        });
+    });
+});

spec/unit/matrix-client.spec.js

@@ -139,6 +139,9 @@ describe("MatrixClient", function() {
         store.getSavedSync = expect.createSpy().andReturn(Promise.resolve(null));
         store.getSavedSyncToken = expect.createSpy().andReturn(Promise.resolve(null));
         store.setSyncData = expect.createSpy().andReturn(Promise.resolve(null));
+        store.getClientOptions = expect.createSpy().andReturn(Promise.resolve(null));
+        store.storeClientOptions = expect.createSpy().andReturn(Promise.resolve(null));
+        store.isNewlyCreated = expect.createSpy().andReturn(Promise.resolve(true));
         client = new MatrixClient({
             baseUrl: "https://my.home.server",
             idBaseUrl: identityServerUrl,
@@ -182,7 +185,7 @@ describe("MatrixClient", function() {
         });
     });
 
-    it("should not POST /filter if a matching filter already exists", function(done) {
+    it("should not POST /filter if a matching filter already exists", async function() {
         httpLookups = [];
         httpLookups.push(PUSH_RULES_RESPONSE);
         httpLookups.push(SYNC_RESPONSE);
@@ -191,15 +194,19 @@ describe("MatrixClient", function() {
         const filter = new sdk.Filter(0, filterId);
         filter.setDefinition({"room": {"timeline": {"limit": 8}}});
         store.getFilter.andReturn(filter);
-        client.startClient();
-
-        client.on("sync", function syncListener(state) {
-            if (state === "SYNCING") {
-                expect(httpLookups.length).toEqual(0);
-                client.removeListener("sync", syncListener);
-                done();
-            }
+        const syncPromise = new Promise((resolve, reject) => {
+            client.on("sync", function syncListener(state) {
+                if (state === "SYNCING") {
+                    expect(httpLookups.length).toEqual(0);
+                    client.removeListener("sync", syncListener);
+                    resolve();
+                } else if (state === "ERROR") {
+                    reject(new Error("sync error"));
+                }
+            });
         });
+        await client.startClient();
+        await syncPromise;
     });
 
     describe("getSyncState", function() {
@@ -207,15 +214,18 @@ describe("MatrixClient", function() {
             expect(client.getSyncState()).toBe(null);
         });
 
-        it("should return the same sync state as emitted sync events", function(done) {
-            client.on("sync", function syncListener(state) {
-                expect(state).toEqual(client.getSyncState());
-                if (state === "SYNCING") {
-                    client.removeListener("sync", syncListener);
-                    done();
-                }
+        it("should return the same sync state as emitted sync events", async function() {
+            const syncingPromise = new Promise((resolve) => {
+                client.on("sync", function syncListener(state) {
+                    expect(state).toEqual(client.getSyncState());
+                    if (state === "SYNCING") {
+                        client.removeListener("sync", syncListener);
+                        resolve();
+                    }
+                });
             });
-            client.startClient();
+            await client.startClient();
+            await syncingPromise;
         });
     });
 
@@ -258,8 +268,8 @@ describe("MatrixClient", function() {
     });
 
     describe("retryImmediately", function() {
-        it("should return false if there is no request waiting", function() {
-            client.startClient();
+        it("should return false if there is no request waiting", async function() {
+            await client.startClient();
             expect(client.retryImmediately()).toBe(false);
         });
 
@@ -380,7 +390,7 @@ describe("MatrixClient", function() {
             client.startClient();
         });
 
-        it("should transition ERROR -> PREPARED after /sync if prev failed",
+        it("should transition ERROR -> CATCHUP after /sync if prev failed",
         function(done) {
             const expectedStates = [];
             acceptKeepalives = false;
@@ -403,7 +413,7 @@ describe("MatrixClient", function() {
 
             expectedStates.push(["RECONNECTING", null]);
             expectedStates.push(["ERROR", "RECONNECTING"]);
-            expectedStates.push(["PREPARED", "ERROR"]);
+            expectedStates.push(["CATCHUP", "ERROR"]);
             client.on("sync", syncChecker(expectedStates, done));
             client.startClient();
         });

spec/unit/room-member.spec.js

@@ -192,6 +192,15 @@ describe("RoomMember", function() {
         });
     });
 
+    describe("isOutOfBand", function() {
+        it("should be set by markOutOfBand", function() {
+            const member = new RoomMember();
+            expect(member.isOutOfBand()).toEqual(false);
+            member.markOutOfBand();
+            expect(member.isOutOfBand()).toEqual(true);
+        });
+    });
+
     describe("setMembershipEvent", function() {
         const joinEvent = utils.mkMembership({
             event: true,

spec/unit/room-state.spec.js

@@ -11,6 +11,9 @@ describe("RoomState", function() {
     const roomId = "!foo:bar";
     const userA = "@alice:bar";
     const userB = "@bob:bar";
+    const userC = "@cleo:bar";
+    const userLazy = "@lazy:bar";
+
     let state;
 
     beforeEach(function() {
@@ -78,8 +81,8 @@ describe("RoomState", function() {
     });
 
     describe("getSentinelMember", function() {
-        it("should return null if there is no member", function() {
-            expect(state.getSentinelMember("@no-one:here")).toEqual(null);
+        it("should return a member with the user id as name", function() {
+            expect(state.getSentinelMember("@no-one:here").name).toEqual("@no-one:here");
         });
 
         it("should return a member which doesn't change when the state is updated",
@@ -223,7 +226,6 @@ describe("RoomState", function() {
 
         it("should call setPowerLevelEvent on a new RoomMember if power levels exist",
         function() {
-            const userC = "@cleo:bar";
             const memberEvent = utils.mkMembership({
                 mship: "join", user: userC, room: roomId, event: true,
             });
@@ -263,6 +265,114 @@ describe("RoomState", function() {
         });
     });
 
+    describe("setOutOfBandMembers", function() {
+        it("should add a new member", function() {
+            const oobMemberEvent = utils.mkMembership({
+                user: userLazy, mship: "join", room: roomId, event: true,
+            });
+            state.markOutOfBandMembersStarted();
+            state.setOutOfBandMembers([oobMemberEvent]);
+            const member = state.getMember(userLazy);
+            expect(member.userId).toEqual(userLazy);
+            expect(member.isOutOfBand()).toEqual(true);
+        });
+
+        it("should have no effect when not in correct status", function() {
+            state.setOutOfBandMembers([utils.mkMembership({
+                user: userLazy, mship: "join", room: roomId, event: true,
+            })]);
+            expect(state.getMember(userLazy)).toBeFalsy();
+        });
+
+        it("should emit newMember when adding a member", function() {
+            const userLazy = "@oob:hs";
+            const oobMemberEvent = utils.mkMembership({
+                user: userLazy, mship: "join", room: roomId, event: true,
+            });
+            let eventReceived = false;
+            state.once('RoomState.newMember', (_, __, member) => {
+                expect(member.userId).toEqual(userLazy);
+                eventReceived = true;
+            });
+            state.markOutOfBandMembersStarted();
+            state.setOutOfBandMembers([oobMemberEvent]);
+            expect(eventReceived).toEqual(true);
+        });
+
+        it("should never overwrite existing members", function() {
+            const oobMemberEvent = utils.mkMembership({
+                user: userA, mship: "join", room: roomId, event: true,
+            });
+            state.markOutOfBandMembersStarted();
+            state.setOutOfBandMembers([oobMemberEvent]);
+            const memberA = state.getMember(userA);
+            expect(memberA.events.member.getId()).toNotEqual(oobMemberEvent.getId());
+            expect(memberA.isOutOfBand()).toEqual(false);
+        });
+
+        it("should emit members when updating a member", function() {
+            const doesntExistYetUserId = "@doesntexistyet:hs";
+            const oobMemberEvent = utils.mkMembership({
+                user: doesntExistYetUserId, mship: "join", room: roomId, event: true,
+            });
+            let eventReceived = false;
+            state.once('RoomState.members', (_, __, member) => {
+                expect(member.userId).toEqual(doesntExistYetUserId);
+                eventReceived = true;
+            });
+
+            state.markOutOfBandMembersStarted();
+            state.setOutOfBandMembers([oobMemberEvent]);
+            expect(eventReceived).toEqual(true);
+        });
+    });
+
+    describe("clone", function() {
+        it("should contain same information as original", function() {
+            // include OOB members in copy
+            state.markOutOfBandMembersStarted();
+            state.setOutOfBandMembers([utils.mkMembership({
+                user: userLazy, mship: "join", room: roomId, event: true,
+            })]);
+            const copy = state.clone();
+            // check individual members
+            [userA, userB, userLazy].forEach((userId) => {
+                const member = state.getMember(userId);
+                const memberCopy = copy.getMember(userId);
+                expect(member.name).toEqual(memberCopy.name);
+                expect(member.isOutOfBand()).toEqual(memberCopy.isOutOfBand());
+            });
+            // check member keys
+            expect(Object.keys(state.members)).toEqual(Object.keys(copy.members));
+            // check join count
+            expect(state.getJoinedMemberCount()).toEqual(copy.getJoinedMemberCount());
+        });
+
+        it("should mark old copy as not waiting for out of band anymore", function() {
+            state.markOutOfBandMembersStarted();
+            const copy = state.clone();
+            copy.setOutOfBandMembers([utils.mkMembership({
+                user: userA, mship: "join", room: roomId, event: true,
+            })]);
+            // should have no effect as it should be marked in status finished just like copy
+            state.setOutOfBandMembers([utils.mkMembership({
+                user: userLazy, mship: "join", room: roomId, event: true,
+            })]);
+            expect(state.getMember(userLazy)).toBeFalsy();
+        });
+
+        it("should return copy independent of original", function() {
+            const copy = state.clone();
+            copy.setStateEvents([utils.mkMembership({
+                user: userLazy, mship: "join", room: roomId, event: true,
+            })]);
+
+            expect(state.getMember(userLazy)).toBeFalsy();
+            expect(state.getJoinedMemberCount()).toEqual(2);
+            expect(copy.getJoinedMemberCount()).toEqual(3);
+        });
+    });
+
     describe("setTypingEvent", function() {
         it("should call setTypingEvent on each RoomMember", function() {
             const typingEvent = utils.mkEvent({
@@ -285,13 +395,6 @@ describe("RoomState", function() {
     });
 
     describe("maySendStateEvent", function() {
-        it("should say non-joined members may not send state",
-        function() {
-            expect(state.maySendStateEvent(
-                'm.room.name', "@nobody:nowhere",
-            )).toEqual(false);
-        });
-
         it("should say any member may send state with no power level event",
         function() {
             expect(state.maySendStateEvent('m.room.name', userA)).toEqual(true);
@@ -367,15 +470,117 @@ describe("RoomState", function() {
         });
     });
 
-    describe("maySendEvent", function() {
-        it("should say non-joined members may not send events",
-        function() {
-            expect(state.maySendEvent(
-                'm.room.message', "@nobody:nowhere",
-            )).toEqual(false);
-            expect(state.maySendMessage("@nobody:nowhere")).toEqual(false);
+    describe("getJoinedMemberCount", function() {
+        beforeEach(() => {
+            state = new RoomState(roomId);
         });
 
+        it("should update after adding joined member", function() {
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "join",
+                    user: userA, room: roomId}),
+            ]);
+            expect(state.getJoinedMemberCount()).toEqual(1);
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "join",
+                    user: userC, room: roomId}),
+            ]);
+            expect(state.getJoinedMemberCount()).toEqual(2);
+        });
+    });
+
+    describe("getInvitedMemberCount", function() {
+        beforeEach(() => {
+            state = new RoomState(roomId);
+        });
+
+        it("should update after adding invited member", function() {
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "invite",
+                    user: userA, room: roomId}),
+            ]);
+            expect(state.getInvitedMemberCount()).toEqual(1);
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "invite",
+                    user: userC, room: roomId}),
+            ]);
+            expect(state.getInvitedMemberCount()).toEqual(2);
+        });
+    });
+
+    describe("setJoinedMemberCount", function() {
+        beforeEach(() => {
+            state = new RoomState(roomId);
+        });
+
+        it("should, once used, override counting members from state", function() {
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "join",
+                    user: userA, room: roomId}),
+            ]);
+            expect(state.getJoinedMemberCount()).toEqual(1);
+            state.setJoinedMemberCount(100);
+            expect(state.getJoinedMemberCount()).toEqual(100);
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "join",
+                    user: userC, room: roomId}),
+            ]);
+            expect(state.getJoinedMemberCount()).toEqual(100);
+        });
+
+        it("should, once used, override counting members from state, " +
+        "also after clone", function() {
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "join",
+                    user: userA, room: roomId}),
+            ]);
+            state.setJoinedMemberCount(100);
+            const copy = state.clone();
+            copy.setStateEvents([
+                utils.mkMembership({event: true, mship: "join",
+                    user: userC, room: roomId}),
+            ]);
+            expect(state.getJoinedMemberCount()).toEqual(100);
+        });
+    });
+
+    describe("setInvitedMemberCount", function() {
+        beforeEach(() => {
+            state = new RoomState(roomId);
+        });
+
+        it("should, once used, override counting members from state", function() {
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "invite",
+                    user: userB, room: roomId}),
+            ]);
+            expect(state.getInvitedMemberCount()).toEqual(1);
+            state.setInvitedMemberCount(100);
+            expect(state.getInvitedMemberCount()).toEqual(100);
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "invite",
+                    user: userC, room: roomId}),
+            ]);
+            expect(state.getInvitedMemberCount()).toEqual(100);
+        });
+
+        it("should, once used, override counting members from state, " +
+        "also after clone", function() {
+            state.setStateEvents([
+                utils.mkMembership({event: true, mship: "invite",
+                    user: userB, room: roomId}),
+            ]);
+            state.setInvitedMemberCount(100);
+            const copy = state.clone();
+            copy.setStateEvents([
+                utils.mkMembership({event: true, mship: "invite",
+                    user: userC, room: roomId}),
+            ]);
+            expect(state.getInvitedMemberCount()).toEqual(100);
+        });
+    });
+
+    describe("maySendEvent", function() {
         it("should say any member may send events with no power level event",
         function() {
             expect(state.maySendEvent('m.room.message', userA)).toEqual(true);

spec/unit/room.spec.js

@@ -67,13 +67,14 @@ describe("Room", function() {
 
     describe("getMember", function() {
         beforeEach(function() {
-            // clobber members property with test data
-            room.currentState.members = {
-                "@alice:bar": {
-                    userId: userA,
-                    roomId: roomId,
-                },
-            };
+            room.currentState.getMember.andCall(function(userId) {
+                return {
+                    "@alice:bar": {
+                        userId: userA,
+                        roomId: roomId,
+                    },
+                }[userId];
+            });
         });
 
         it("should return null if the member isn't in current state", function() {
@@ -386,7 +387,7 @@ describe("Room", function() {
         let events = null;
 
         beforeEach(function() {
-            room = new Room(roomId, {timelineSupport: timelineSupport});
+            room = new Room(roomId, null, null, {timelineSupport: timelineSupport});
             // set events each time to avoid resusing Event objects (which
             // doesn't work because they get frozen)
             events = [
@@ -468,7 +469,7 @@ describe("Room", function() {
 
     describe("compareEventOrdering", function() {
         beforeEach(function() {
-            room = new Room(roomId, {timelineSupport: true});
+            room = new Room(roomId, null, null, {timelineSupport: true});
         });
 
         const events = [
@@ -570,72 +571,75 @@ describe("Room", function() {
     describe("hasMembershipState", function() {
         it("should return true for a matching userId and membership",
         function() {
-            room.currentState.members = {
-                "@alice:bar": { userId: "@alice:bar", membership: "join" },
-                "@bob:bar": { userId: "@bob:bar", membership: "invite" },
-            };
+            room.currentState.getMember.andCall(function(userId) {
+                return {
+                    "@alice:bar": { userId: "@alice:bar", membership: "join" },
+                    "@bob:bar": { userId: "@bob:bar", membership: "invite" },
+                }[userId];
+            });
             expect(room.hasMembershipState("@bob:bar", "invite")).toBe(true);
         });
 
         it("should return false if match membership but no match userId",
         function() {
-            room.currentState.members = {
-                "@alice:bar": { userId: "@alice:bar", membership: "join" },
-            };
+            room.currentState.getMember.andCall(function(userId) {
+                return {
+                    "@alice:bar": { userId: "@alice:bar", membership: "join" },
+                }[userId];
+            });
             expect(room.hasMembershipState("@bob:bar", "join")).toBe(false);
         });
 
         it("should return false if match userId but no match membership",
         function() {
-            room.currentState.members = {
-                "@alice:bar": { userId: "@alice:bar", membership: "join" },
-            };
+            room.currentState.getMember.andCall(function(userId) {
+                return {
+                    "@alice:bar": { userId: "@alice:bar", membership: "join" },
+                }[userId];
+            });
             expect(room.hasMembershipState("@alice:bar", "ban")).toBe(false);
         });
 
         it("should return false if no match membership or userId",
         function() {
-            room.currentState.members = {
-                "@alice:bar": { userId: "@alice:bar", membership: "join" },
-            };
+            room.currentState.getMember.andCall(function(userId) {
+                return {
+                    "@alice:bar": { userId: "@alice:bar", membership: "join" },
+                }[userId];
+            });
             expect(room.hasMembershipState("@bob:bar", "invite")).toBe(false);
         });
 
         it("should return false if no members exist",
         function() {
-            room.currentState.members = {};
             expect(room.hasMembershipState("@foo:bar", "join")).toBe(false);
         });
     });
 
     describe("recalculate", function() {
-        let stateLookup = {
-            // event.type + "$" event.state_key : MatrixEvent
-        };
-
         const setJoinRule = function(rule) {
-            stateLookup["m.room.join_rules$"] = utils.mkEvent({
+            room.addLiveEvents([utils.mkEvent({
                 type: "m.room.join_rules", room: roomId, user: userA, content: {
                     join_rule: rule,
                 }, event: true,
-            });
+            })]);
         };
         const setAliases = function(aliases, stateKey) {
             if (!stateKey) {
                 stateKey = "flibble";
             }
-            stateLookup["m.room.aliases$" + stateKey] = utils.mkEvent({
+            room.addLiveEvents([utils.mkEvent({
                 type: "m.room.aliases", room: roomId, skey: stateKey, content: {
                     aliases: aliases,
                 }, event: true,
-            });
+            })]);
         };
         const setRoomName = function(name) {
-            stateLookup["m.room.name$"] = utils.mkEvent({
+            room.addLiveEvents([utils.mkEvent({
                 type: "m.room.name", room: roomId, user: userA, content: {
                     name: name,
                 }, event: true,
-            });
+            })]);
         };
         const addMember = function(userId, state, opts) {
             if (!state) {
@@ -647,56 +651,14 @@ describe("Room", function() {
             opts.user = opts.user || userId;
             opts.skey = userId;
             opts.event = true;
-            stateLookup["m.room.member$" + userId] = utils.mkMembership(opts);
+            const event = utils.mkMembership(opts);
+            room.addLiveEvents([event]);
+            return event;
         };
 
         beforeEach(function() {
-            stateLookup = {};
-            room.currentState.getStateEvents.andCall(function(type, key) {
-                if (key === undefined) {
-                    const prefix = type + "$";
-                    const list = [];
-                    for (const stateBlob in stateLookup) {
-                        if (!stateLookup.hasOwnProperty(stateBlob)) {
-                            continue;
-                        }
-                        if (stateBlob.indexOf(prefix) === 0) {
-                            list.push(stateLookup[stateBlob]);
-                        }
-                    }
-                    return list;
-                } else {
-                    return stateLookup[type + "$" + key];
-                }
-            });
-            room.currentState.getMembers.andCall(function() {
-                const memberEvents = room.currentState.getStateEvents("m.room.member");
-                const members = [];
-                for (let i = 0; i < memberEvents.length; i++) {
-                    members.push({
-                        name: memberEvents[i].event.content &&
-                                memberEvents[i].event.content.displayname ?
-                                memberEvents[i].event.content.displayname :
-                                memberEvents[i].getStateKey(),
-                        userId: memberEvents[i].getStateKey(),
-                        events: { member: memberEvents[i] },
-                    });
-                }
-                return members;
-            });
-            room.currentState.getMember.andCall(function(userId) {
-                const memberEvent = room.currentState.getStateEvents(
-                    "m.room.member", userId,
-                );
-                return {
-                    name: memberEvent.event.content &&
-                            memberEvent.event.content.displayname ?
-                            memberEvent.event.content.displayname :
-                            memberEvent.getStateKey(),
-                    userId: memberEvent.getStateKey(),
-                    events: { member: memberEvent },
-                };
-            });
+            // no mocking
+            room = new Room(roomId, null, userA);
         });
 
         describe("Room.recalculate => Stripped State Events", function() {
@@ -704,8 +666,8 @@ describe("Room", function() {
             "room is an invite room", function() {
                 const roomName = "flibble";
 
-                addMember(userA, "invite");
-                stateLookup["m.room.member$" + userA].event.invite_room_state = [
+                const event = addMember(userA, "invite");
+                event.event.invite_room_state = [
                     {
                         type: "m.room.name",
                         state_key: "",
@@ -715,30 +677,108 @@ describe("Room", function() {
                     },
                 ];
 
-                room.recalculate(userA);
-                expect(room.currentState.setStateEvents).toHaveBeenCalled();
-                // first call, first arg (which is an array), first element in array
-                const fakeEvent = room.currentState.setStateEvents.calls[0].
-                      arguments[0][0];
-                expect(fakeEvent.getContent()).toEqual({
-                    name: roomName,
-                });
+                room.recalculate();
+                expect(room.name).toEqual(roomName);
             });
 
             it("should not clobber state events if it isn't an invite room", function() {
-                addMember(userA, "join");
-                stateLookup["m.room.member$" + userA].event.invite_room_state = [
+                const event = addMember(userA, "join");
+                const roomName = "flibble";
+                setRoomName(roomName);
+                const roomNameToIgnore = "ignoreme";
+                event.event.invite_room_state = [
                     {
                         type: "m.room.name",
                         state_key: "",
                         content: {
-                            name: "flibble",
+                            name: roomNameToIgnore,
                         },
                     },
                 ];
 
-                room.recalculate(userA);
-                expect(room.currentState.setStateEvents).toNotHaveBeenCalled();
+                room.recalculate();
+                expect(room.name).toEqual(roomName);
+            });
+        });
+
+        describe("Room.recalculate => Room Name using room summary", function() {
+            it("should use room heroes if available", function() {
+                addMember(userA, "invite");
+                addMember(userB);
+                addMember(userC);
+                addMember(userD);
+                room.setSummary({
+                    "m.heroes": [userB, userC, userD],
+                });
+
+                room.recalculate();
+                expect(room.name).toEqual(`${userB} and 2 others`);
+            });
+
+            it("missing hero member state reverts to mxid", function() {
+                room.setSummary({
+                    "m.heroes": [userB],
+                    "m.joined_member_count": 2,
+                });
+
+                room.recalculate();
+                expect(room.name).toEqual(userB);
+            });
+
+            it("uses hero name from state", function() {
+                const name = "Mr B";
+                addMember(userA, "invite");
+                addMember(userB, "join", {name});
+                room.setSummary({
+                    "m.heroes": [userB],
+                });
+
+                room.recalculate();
+                expect(room.name).toEqual(name);
+            });
+
+            it("uses counts from summary", function() {
+                const name = "Mr B";
+                addMember(userB, "join", {name});
+                room.setSummary({
+                    "m.heroes": [userB],
+                    "m.joined_member_count": 50,
+                    "m.invited_member_count": 50,
+                });
+                room.recalculate();
+                expect(room.name).toEqual(`${name} and 98 others`);
+            });
+
+            it("relies on heroes in case of absent counts", function() {
+                const nameB = "Mr Bean";
+                const nameC = "Mel C";
+                addMember(userB, "join", {name: nameB});
+                addMember(userC, "join", {name: nameC});
+                room.setSummary({
+                    "m.heroes": [userB, userC],
+                });
+                room.recalculate();
+                expect(room.name).toEqual(`${nameB} and ${nameC}`);
+            });
+
+            it("uses only heroes", function() {
+                const nameB = "Mr Bean";
+                addMember(userB, "join", {name: nameB});
+                addMember(userC, "join");
+                room.setSummary({
+                    "m.heroes": [userB],
+                });
+                room.recalculate();
+                expect(room.name).toEqual(nameB);
+            });
+
+            it("reverts to empty room in case of self chat", function() {
+                room.setSummary({
+                    "m.heroes": [],
+                    "m.invited_member_count": 1,
+                });
+                room.recalculate();
+                expect(room.name).toEqual("Empty room");
             });
         });
 
@@ -751,7 +791,7 @@ describe("Room", function() {
                 addMember(userB);
                 addMember(userC);
                 addMember(userD);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 // we expect at least 1 member to be mentioned
                 const others = [userB, userC, userD];
@@ -772,7 +812,7 @@ describe("Room", function() {
                 addMember(userA);
                 addMember(userB);
                 addMember(userC);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name.indexOf(userB)).toNotEqual(-1, name);
                 expect(name.indexOf(userC)).toNotEqual(-1, name);
@@ -785,7 +825,7 @@ describe("Room", function() {
                 addMember(userA);
                 addMember(userB);
                 addMember(userC);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name.indexOf(userB)).toNotEqual(-1, name);
                 expect(name.indexOf(userC)).toNotEqual(-1, name);
@@ -797,7 +837,7 @@ describe("Room", function() {
                 setJoinRule("public");
                 addMember(userA);
                 addMember(userB);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name.indexOf(userB)).toNotEqual(-1, name);
             });
@@ -808,7 +848,7 @@ describe("Room", function() {
                 setJoinRule("invite");
                 addMember(userA);
                 addMember(userB);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name.indexOf(userB)).toNotEqual(-1, name);
             });
@@ -818,7 +858,7 @@ describe("Room", function() {
                 setJoinRule("invite");
                 addMember(userA, "invite", {user: userB});
                 addMember(userB);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name.indexOf(userB)).toNotEqual(-1, name);
             });
@@ -828,7 +868,7 @@ describe("Room", function() {
                 const alias = "#room_alias:here";
                 setJoinRule("invite");
                 setAliases([alias, "#another:one"]);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name).toEqual(alias);
             });
@@ -838,7 +878,7 @@ describe("Room", function() {
                 const alias = "#room_alias:here";
                 setJoinRule("public");
                 setAliases([alias, "#another:one"]);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name).toEqual(alias);
             });
@@ -848,7 +888,7 @@ describe("Room", function() {
                 const roomName = "A mighty name indeed";
                 setJoinRule("invite");
                 setRoomName(roomName);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name).toEqual(roomName);
             });
@@ -858,25 +898,23 @@ describe("Room", function() {
                 const roomName = "A mighty name indeed";
                 setJoinRule("public");
                 setRoomName(roomName);
-                room.recalculate(userA);
-                const name = room.name;
-                expect(name).toEqual(roomName);
+                room.recalculate();
+                expect(room.name).toEqual(roomName);
             });
 
             it("should return 'Empty room' for private (invite join_rules) rooms if" +
             " a room name and alias don't exist and it is a self-chat.", function() {
                 setJoinRule("invite");
                 addMember(userA);
-                room.recalculate(userA);
-                const name = room.name;
-                expect(name).toEqual("Empty room");
+                room.recalculate();
+                expect(room.name).toEqual("Empty room");
             });
 
             it("should return 'Empty room' for public (public join_rules) rooms if a" +
             " room name and alias don't exist and it is a self-chat.", function() {
                 setJoinRule("public");
                 addMember(userA);
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name).toEqual("Empty room");
             });
@@ -884,7 +922,7 @@ describe("Room", function() {
             it("should return 'Empty room' if there is no name, " +
                "alias or members in the room.",
             function() {
-                room.recalculate(userA);
+                room.recalculate();
                 const name = room.name;
                 expect(name).toEqual("Empty room");
             });
@@ -893,9 +931,9 @@ describe("Room", function() {
                "available",
             function() {
                 setJoinRule("invite");
-                addMember(userA, 'join', {name: "Alice"});
-                addMember(userB, "invite", {user: userA});
-                room.recalculate(userB);
+                addMember(userB, 'join', {name: "Alice"});
+                addMember(userA, "invite", {user: userA});
+                room.recalculate();
                 const name = room.name;
                 expect(name).toEqual("Alice");
             });
@@ -903,11 +941,11 @@ describe("Room", function() {
             it("should return inviter mxid if display name not available",
             function() {
                 setJoinRule("invite");
-                addMember(userA);
-                addMember(userB, "invite", {user: userA});
-                room.recalculate(userB);
+                addMember(userB);
+                addMember(userA, "invite", {user: userA});
+                room.recalculate();
                 const name = room.name;
-                expect(name).toEqual(userA);
+                expect(name).toEqual(userB);
             });
         });
     });
@@ -1154,7 +1192,7 @@ describe("Room", function() {
     describe("addPendingEvent", function() {
         it("should add pending events to the pendingEventList if " +
                       "pendingEventOrdering == 'detached'", function() {
-            const room = new Room(roomId, {
+            const room = new Room(roomId, null, userA, {
                 pendingEventOrdering: "detached",
             });
             const eventA = utils.mkMessage({
@@ -1180,7 +1218,7 @@ describe("Room", function() {
 
         it("should add pending events to the timeline if " +
                       "pendingEventOrdering == 'chronological'", function() {
-            room = new Room(roomId, {
+            room = new Room(roomId, null, userA, {
                 pendingEventOrdering: "chronological",
             });
             const eventA = utils.mkMessage({
@@ -1204,7 +1242,7 @@ describe("Room", function() {
 
     describe("updatePendingEvent", function() {
         it("should remove cancelled events from the pending list", function() {
-            const room = new Room(roomId, {
+            const room = new Room(roomId, null, userA, {
                 pendingEventOrdering: "detached",
             });
             const eventA = utils.mkMessage({
@@ -1240,7 +1278,7 @@ describe("Room", function() {
 
 
         it("should remove cancelled events from the timeline", function() {
-            const room = new Room(roomId);
+            const room = new Room(roomId, null, userA);
             const eventA = utils.mkMessage({
                 room: roomId, user: userA, event: true,
             });
@@ -1272,4 +1310,153 @@ describe("Room", function() {
             expect(callCount).toEqual(1);
         });
     });
+
+    describe("loadMembersIfNeeded", function() {
+        function createClientMock(serverResponse, storageResponse = null) {
+            return {
+                getEventMapper: function() {
+                    // events should already be MatrixEvents
+                    return function(event) {return event;};
+                },
+                isRoomEncrypted: function() {
+                    return false;
+                },
+                _http: {
+                    serverResponse,
+                    authedRequest: function() {
+                        if (this.serverResponse instanceof Error) {
+                            return Promise.reject(this.serverResponse);
+                        } else {
+                            return Promise.resolve({chunk: this.serverResponse});
+                        }
+                    },
+                },
+                store: {
+                    storageResponse,
+                    storedMembers: null,
+                    getOutOfBandMembers: function() {
+                        if (this.storageResponse instanceof Error) {
+                            return Promise.reject(this.storageResponse);
+                        } else {
+                            return Promise.resolve(this.storageResponse);
+                        }
+                    },
+                    setOutOfBandMembers: function(roomId, memberEvents) {
+                        this.storedMembers = memberEvents;
+                        return Promise.resolve();
+                    },
+                    getSyncToken: () => "sync_token",
+                },
+            };
+        }
+
+        const memberEvent = utils.mkMembership({
+            user: "@user_a:bar", mship: "join",
+            room: roomId, event: true, name: "User A",
+        });
+
+        it("should load members from server on first call", async function() {
+            const client = createClientMock([memberEvent]);
+            const room = new Room(roomId, client, null, {lazyLoadMembers: true});
+            await room.loadMembersIfNeeded();
+            const memberA = room.getMember("@user_a:bar");
+            expect(memberA.name).toEqual("User A");
+            const storedMembers = client.store.storedMembers;
+            expect(storedMembers.length).toEqual(1);
+            expect(storedMembers[0].event_id).toEqual(memberEvent.getId());
+        });
+
+        it("should take members from storage if available", async function() {
+            const memberEvent2 = utils.mkMembership({
+                user: "@user_a:bar", mship: "join",
+                room: roomId, event: true, name: "Ms A",
+            });
+            const client = createClientMock([memberEvent2], [memberEvent]);
+            const room = new Room(roomId, client, null, {lazyLoadMembers: true});
+
+            await room.loadMembersIfNeeded();
+
+            const memberA = room.getMember("@user_a:bar");
+            expect(memberA.name).toEqual("User A");
+        });
+
+        it("should allow retry on error", async function() {
+            const client = createClientMock(new Error("server says no"));
+            const room = new Room(roomId, client, null, {lazyLoadMembers: true});
+            let hasThrown = false;
+            try {
+                await room.loadMembersIfNeeded();
+            } catch(err) {
+                hasThrown = true;
+            }
+            expect(hasThrown).toEqual(true);
+
+            client._http.serverResponse = [memberEvent];
+            await room.loadMembersIfNeeded();
+            const memberA = room.getMember("@user_a:bar");
+            expect(memberA.name).toEqual("User A");
+        });
+    });
+
+    describe("getMyMembership", function() {
+        it("should return synced membership if membership isn't available yet",
+        function() {
+            const room = new Room(roomId, null, userA);
+            room.updateMyMembership("invite");
+            expect(room.getMyMembership()).toEqual("invite");
+        });
+        it("should emit a Room.myMembership event on a change",
+        function() {
+            const room = new Room(roomId, null, userA);
+            const events = [];
+            room.on("Room.myMembership", (_room, membership, oldMembership) => {
+                events.push({membership, oldMembership});
+            });
+            room.updateMyMembership("invite");
+            expect(room.getMyMembership()).toEqual("invite");
+            expect(events[0]).toEqual({membership: "invite", oldMembership: null});
+            events.splice(0);   //clear
+            room.updateMyMembership("invite");
+            expect(events.length).toEqual(0);
+            room.updateMyMembership("join");
+            expect(room.getMyMembership()).toEqual("join");
+            expect(events[0]).toEqual({membership: "join", oldMembership: "invite"});
+        });
+    });
+
+    describe("guessDMUserId", function() {
+        it("should return first hero id",
+        function() {
+            const room = new Room(roomId, null, userA);
+            room.setSummary({'m.heroes': [userB]});
+            expect(room.guessDMUserId()).toEqual(userB);
+        });
+        it("should return first member that isn't self",
+        function() {
+            const room = new Room(roomId, null, userA);
+            room.addLiveEvents([utils.mkMembership({
+                user: userB, mship: "join",
+                room: roomId, event: true,
+            })]);
+            expect(room.guessDMUserId()).toEqual(userB);
+        });
+        it("should return self if only member present",
+        function() {
+            const room = new Room(roomId, null, userA);
+            expect(room.guessDMUserId()).toEqual(userA);
+        });
+    });
+
+    describe("maySendMessage", function() {
+        it("should return false if synced membership not join",
+        function() {
+            const room = new Room(roomId, null, userA);
+            room.updateMyMembership("invite");
+            expect(room.maySendMessage()).toEqual(false);
+            room.updateMyMembership("leave");
+            expect(room.maySendMessage()).toEqual(false);
+            room.updateMyMembership("join");
+            expect(room.maySendMessage()).toEqual(true);
+        });
+    });
 });

spec/unit/sync-accumulator.spec.js

@@ -52,6 +52,11 @@ describe("SyncAccumulator", function() {
                                 member("bob", "join"),
                             ],
                         },
+                        summary: {
+                            "m.heroes": undefined,
+                            "m.joined_member_count": undefined,
+                            "m.invited_member_count": undefined,
+                        },
                         timeline: {
                             events: [msg("alice", "hi")],
                             prev_batch: "something",
@@ -318,6 +323,58 @@ describe("SyncAccumulator", function() {
             },
         });
     });
+
+    describe("summary field", function() {
+        function createSyncResponseWithSummary(summary) {
+            return {
+                next_batch: "abc",
+                rooms: {
+                    invite: {},
+                    leave: {},
+                    join: {
+                        "!foo:bar": {
+                            account_data: { events: [] },
+                            ephemeral: { events: [] },
+                            unread_notifications: {},
+                            state: {
+                                events: [],
+                            },
+                            summary: summary,
+                            timeline: {
+                                events: [],
+                                prev_batch: "something",
+                            },
+                        },
+                    },
+                },
+            };
+        }
+
+        it("should copy summary properties", function() {
+            sa.accumulate(createSyncResponseWithSummary({
+                "m.heroes": ["@alice:bar"],
+                "m.invited_member_count": 2,
+            }));
+            const summary = sa.getJSON().roomsData.join["!foo:bar"].summary;
+            expect(summary["m.invited_member_count"]).toEqual(2);
+            expect(summary["m.heroes"]).toEqual(["@alice:bar"]);
+        });
+
+        it("should accumulate summary properties", function() {
+            sa.accumulate(createSyncResponseWithSummary({
+                "m.heroes": ["@alice:bar"],
+                "m.invited_member_count": 2,
+            }));
+            sa.accumulate(createSyncResponseWithSummary({
+                "m.heroes": ["@bob:bar"],
+                "m.joined_member_count": 5,
+            }));
+            const summary = sa.getJSON().roomsData.join["!foo:bar"].summary;
+            expect(summary["m.invited_member_count"]).toEqual(2);
+            expect(summary["m.joined_member_count"]).toEqual(5);
+            expect(summary["m.heroes"]).toEqual(["@bob:bar"]);
+        });
+    });
 });
 
 function syncSkeleton(joinObj) {

src/base-apis.js

@@ -262,7 +262,19 @@ MatrixBaseApis.prototype.login = function(loginType, data, callback) {
     utils.extend(login_data, data);
 
     return this._http.authedRequest(
-        callback, "POST", "/login", undefined, login_data,
+        (error, response) => {
+            if (loginType === "m.login.password" && response &&
+                response.access_token && response.user_id) {
+                this._http.opts.accessToken = response.access_token;
+                this.credentials = {
+                    userId: response.user_id,
+                };
+            }
+
+            if (callback) {
+                callback(error, response);
+            }
+        }, "POST", "/login", undefined, login_data,
     );
 };
 
@@ -298,9 +310,23 @@ MatrixBaseApis.prototype.loginWithSAML2 = function(relayState, callback) {
  * @return {string} The HS URL to hit to begin the CAS login process.
  */
 MatrixBaseApis.prototype.getCasLoginUrl = function(redirectUrl) {
-    return this._http.getUrl("/login/cas/redirect", {
+    return this.getSsoLoginUrl(redirectUrl, "cas");
+};
+
+/**
+ * @param {string} redirectUrl The URL to redirect to after the HS
+ *     authenticates with the SSO.
+ * @param {string} loginType The type of SSO login we are doing (sso or cas).
+ *     Defaults to 'sso'.
+ * @return {string} The HS URL to hit to begin the SSO login process.
+ */
+MatrixBaseApis.prototype.getSsoLoginUrl = function(redirectUrl, loginType) {
+    if (loginType === undefined) {
+        loginType = "sso";
+    }
+    return this._http.getUrl("/login/"+loginType+"/redirect", {
         "redirectUrl": redirectUrl,
-    }, httpApi.PREFIX_UNSTABLE);
+    }, httpApi.PREFIX_R0);
 };
 
 /**
@@ -417,6 +443,69 @@ MatrixBaseApis.prototype.roomState = function(roomId, callback) {
     return this._http.authedRequest(callback, "GET", path);
 };
 
+/**
+ * Get an event in a room by its event id.
+ * @param {string} roomId
+ * @param {string} eventId
+ * @param {module:client.callback} callback Optional.
+ *
+ * @return {Promise} Resolves to an object containing the event.
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.fetchRoomEvent = function(roomId, eventId, callback) {
+    const path = utils.encodeUri(
+        "/rooms/$roomId/event/$eventId", {
+            $roomId: roomId,
+            $eventId: eventId,
+        },
+    );
+    return this._http.authedRequest(callback, "GET", path);
+};
+
+/**
+ * @param {string} roomId
+ * @param {string} includeMembership the membership type to include in the response
+ * @param {string} excludeMembership the membership type to exclude from the response
+ * @param {string} atEventId the id of the event for which moment in the timeline the members should be returned for
+ * @param {module:client.callback} callback Optional.
+ * @return {module:client.Promise} Resolves: dictionary of userid to profile information
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.members =
+function(roomId, includeMembership, excludeMembership, atEventId, callback) {
+    const queryParams = {};
+    if (includeMembership) {
+        queryParams.membership = includeMembership;
+    }
+    if (excludeMembership) {
+        queryParams.not_membership = excludeMembership;
+    }
+    if (atEventId) {
+        queryParams.at = atEventId;
+    }
+
+    const queryString = utils.encodeParams(queryParams);
+
+    const path = utils.encodeUri("/rooms/$roomId/members?" + queryString,
+        {$roomId: roomId});
+    return this._http.authedRequest(callback, "GET", path);
+};
+
+/**
+ * Upgrades a room to a new protocol version
+ * @param {string} roomId
+ * @param {string} newVersion The target version to upgrade to
+ * @return {module:client.Promise} Resolves: Object with key 'replacement_room'
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.upgradeRoom = function(roomId, newVersion) {
+    const path = utils.encodeUri("/rooms/$roomId/upgrade", {$roomId: roomId});
+    return this._http.authedRequest(
+        undefined, "POST", path, undefined, {new_version: newVersion},
+    );
+};
+
+
 /**
  * @param {string} groupId
  * @return {module:client.Promise} Resolves: Group summary object
@@ -864,6 +953,28 @@ MatrixBaseApis.prototype.setRoomReadMarkersHttpRequest =
     );
 };
 
+/**
+ * @return {module:client.Promise} Resolves: A list of the user's current rooms
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.getJoinedRooms = function() {
+    const path = utils.encodeUri("/joined_rooms");
+    return this._http.authedRequest(undefined, "GET", path);
+};
+
+/**
+ * Retrieve membership info. for a room.
+ * @param {string} roomId ID of the room to get membership for
+ * @return {module:client.Promise} Resolves: A list of currently joined users
+ *                                 and their profile data.
+ * @return {module:http-api.MatrixError} Rejects: with an error response.
+ */
+MatrixBaseApis.prototype.getJoinedRoomMembers = function(roomId) {
+    const path = utils.encodeUri("/rooms/$roomId/joined_members", {
+        $roomId: roomId,
+    });
+    return this._http.authedRequest(undefined, "GET", path);
+};
 
 // Room Directory operations
 // =========================

src/client.js

@@ -41,18 +41,46 @@ const SyncApi = require("./sync");
 const MatrixBaseApis = require("./base-apis");
 const MatrixError = httpApi.MatrixError;
 const ContentHelpers = require("./content-helpers");
+const olmlib = require("./crypto/olmlib");
 
 import ReEmitter from './ReEmitter';
 import RoomList from './crypto/RoomList';
 
-const SCROLLBACK_DELAY_MS = 3000;
-let CRYPTO_ENABLED = false;
+import Crypto from './crypto';
+import { isCryptoAvailable } from './crypto';
+import { encodeRecoveryKey, decodeRecoveryKey } from './crypto/recoverykey';
+import { keyForNewBackup, keyForExistingBackup } from './crypto/backup_password';
+import { randomString } from './randomstring';
 
-try {
-    var Crypto = require("./crypto");
-    CRYPTO_ENABLED = true;
-} catch (e) {
-    console.warn("Unable to load crypto module: crypto will be disabled: " + e);
+// Disable warnings for now: we use deprecated bluebird functions
+// and need to migrate, but they spam the console with warnings.
+Promise.config({warnings: false});
+
+
+const SCROLLBACK_DELAY_MS = 3000;
+const CRYPTO_ENABLED = isCryptoAvailable();
+
+function keysFromRecoverySession(sessions, decryptionKey, roomId) {
+    const keys = [];
+    for (const [sessionId, sessionData] of Object.entries(sessions)) {
+        try {
+            const decrypted = keyFromRecoverySession(sessionData, decryptionKey);
+            decrypted.session_id = sessionId;
+            decrypted.room_id = roomId;
+            keys.push(decrypted);
+        } catch (e) {
+            console.log("Failed to decrypt session from backup");
+        }
+    }
+    return keys;
+}
+
+function keyFromRecoverySession(session, decryptionKey) {
+    return JSON.parse(decryptionKey.decrypt(
+        session.session_data.ephemeral,
+        session.session_data.mac,
+        session.session_data.ciphertext,
+    ));
 }
 
 /**
@@ -128,6 +156,8 @@ function MatrixClient(opts) {
 
     MatrixBaseApis.call(this, opts);
 
+    this.olmVersion = null; // Populated after initCrypto is done
+
     this.reEmitter = new ReEmitter(this);
 
     this.store = opts.store || new StubStore();
@@ -180,10 +210,6 @@ function MatrixClient(opts) {
 
     this._forceTURN = opts.forceTURN || false;
 
-    if (CRYPTO_ENABLED) {
-        this.olmVersion = Crypto.getOlmVersion();
-    }
-
     // List of which rooms have encryption enabled: separate from crypto because
     // we still want to know which rooms are encrypted even if crypto is disabled:
     // we don't want to start sending unencrypted events to them.
@@ -191,6 +217,8 @@ function MatrixClient(opts) {
 
     // The pushprocessor caches useful things, so keep one and re-use it
     this._pushProcessor = new PushProcessor(this);
+
+    this._serverSupportsLazyLoading = null;
 }
 utils.inherits(MatrixClient, EventEmitter);
 utils.extend(MatrixClient.prototype, MatrixBaseApis.prototype);
@@ -371,6 +399,13 @@ MatrixClient.prototype.setNotifTimelineSet = function(notifTimelineSet) {
  * successfully initialised.
  */
 MatrixClient.prototype.initCrypto = async function() {
+    if (!isCryptoAvailable()) {
+        throw new Error(
+            `End-to-end encryption not supported in this js-sdk build: did ` +
+                `you remember to load the olm library?`,
+        );
+    }
+
     if (this._crypto) {
         console.warn("Attempt to re-initialise e2e encryption on MatrixClient");
         return;
@@ -388,13 +423,6 @@ MatrixClient.prototype.initCrypto = async function() {
     // initialise the list of encrypted rooms (whether or not crypto is enabled)
     await this._roomList.init();
 
-    if (!CRYPTO_ENABLED) {
-        throw new Error(
-            `End-to-end encryption not supported in this js-sdk build: did ` +
-                `you remember to load the olm library?`,
-        );
-    }
-
     const userId = this.getUserId();
     if (userId === null) {
         throw new Error(
@@ -426,6 +454,9 @@ MatrixClient.prototype.initCrypto = async function() {
 
     await crypto.init();
 
+    this.olmVersion = Crypto.getOlmVersion();
+
+
     // if crypto initialisation was successful, tell it to attach its event
     // handlers.
     crypto.registerEventHandlers(this);
@@ -529,7 +560,15 @@ MatrixClient.prototype.setDeviceVerified = function(userId, deviceId, verified)
     if (verified === undefined) {
         verified = true;
     }
-    return _setDeviceVerification(this, userId, deviceId, verified, null);
+    const prom = _setDeviceVerification(this, userId, deviceId, verified, null);
+
+    // if one of the user's own devices is being marked as verified / unverified,
+    // check the key backup status, since whether or not we use this depends on
+    // whether it has a signature from a verified device
+    if (userId == this.credentials.userId) {
+        this._crypto.checkKeyBackup();
+    }
+    return prom;
 };
 
 /**
@@ -688,6 +727,21 @@ MatrixClient.prototype.isRoomEncrypted = function(roomId) {
     return this._roomList.isRoomEncrypted(roomId);
 };
 
+/**
+ * Forces the current outbound group session to be discarded such
+ * that another one will be created next time an event is sent.
+ *
+ * @param {string} roomId The ID of the room to discard the session for
+ *
+ * This should not normally be necessary.
+ */
+MatrixClient.prototype.forceDiscardSession = function(roomId) {
+    if (!this._crypto) {
+        throw new Error("End-to-End encryption disabled");
+    }
+    this._crypto.forceDiscardSession(roomId);
+};
+
 /**
  * Get a list containing all of the room keys
  *
@@ -718,6 +772,333 @@ MatrixClient.prototype.importRoomKeys = function(keys) {
     return this._crypto.importRoomKeys(keys);
 };
 
+/**
+ * Get information about the current key backup.
+ * @returns {Promise} Information object from API or null
+ */
+MatrixClient.prototype.getKeyBackupVersion = function() {
+    return this._http.authedRequest(
+        undefined, "GET", "/room_keys/version",
+    ).then((res) => {
+        if (res.algorithm !== olmlib.MEGOLM_BACKUP_ALGORITHM) {
+            const err = "Unknown backup algorithm: " + res.algorithm;
+            return Promise.reject(err);
+        } else if (!(typeof res.auth_data === "object")
+                   || !res.auth_data.public_key) {
+            const err = "Invalid backup data returned";
+            return Promise.reject(err);
+        } else {
+            return res;
+        }
+    }).catch((e) => {
+        if (e.errcode === 'M_NOT_FOUND') {
+            return null;
+        } else {
+            throw e;
+        }
+    });
+};
+
+/**
+ * @param {object} info key backup info dict from getKeyBackupVersion()
+ * @return {object} {
+ *     usable: [bool], // is the backup trusted, true iff there is a sig that is valid & from a trusted device
+ *     sigs: [
+ *         valid: [bool],
+ *         device: [DeviceInfo],
+ *     ]
+ * }
+ */
+MatrixClient.prototype.isKeyBackupTrusted = function(info) {
+    return this._crypto.isKeyBackupTrusted(info);
+};
+
+/**
+ * @returns {bool} true if the client is configured to back up keys to
+ *     the server, otherwise false.
+ */
+MatrixClient.prototype.getKeyBackupEnabled = function() {
+    if (this._crypto === null) {
+        throw new Error("End-to-end encryption disabled");
+    }
+    return Boolean(this._crypto.backupKey);
+};
+
+/**
+ * Enable backing up of keys, using data previously returned from
+ * getKeyBackupVersion.
+ *
+ * @param {object} info Backup information object as returned by getKeyBackupVersion
+ */
+MatrixClient.prototype.enableKeyBackup = function(info) {
+    if (this._crypto === null) {
+        throw new Error("End-to-end encryption disabled");
+    }
+
+    this._crypto.backupInfo = info;
+    if (this._crypto.backupKey) this._crypto.backupKey.free();
+    this._crypto.backupKey = new global.Olm.PkEncryption();
+    this._crypto.backupKey.set_recipient_key(info.auth_data.public_key);
+
+    this.emit('crypto.keyBackupStatus', true);
+};
+
+/**
+ * Disable backing up of keys.
+ */
+MatrixClient.prototype.disableKeyBackup = function() {
+    if (this._crypto === null) {
+        throw new Error("End-to-end encryption disabled");
+    }
+
+    this._crypto.backupInfo = null;
+    if (this._crypto.backupKey) this._crypto.backupKey.free();
+    this._crypto.backupKey = null;
+
+    this.emit('crypto.keyBackupStatus', false);
+};
+
+/**
+ * Set up the data required to create a new backup version.  The backup version
+ * will not be created and enabled until createKeyBackupVersion is called.
+ *
+ * @param {string} password Passphrase string that can be entered by the user
+ *     when restoring the backup as an alternative to entering the recovery key.
+ *     Optional.
+ *
+ * @returns {Promise<object>} Object that can be passed to createKeyBackupVersion and
+ *     additionally has a 'recovery_key' member with the user-facing recovery key string.
+ */
+MatrixClient.prototype.prepareKeyBackupVersion = async function(password) {
+    if (this._crypto === null) {
+        throw new Error("End-to-end encryption disabled");
+    }
+
+    const decryption = new global.Olm.PkDecryption();
+    try {
+        let publicKey;
+        const authData = {};
+        if (password) {
+            const keyInfo = await keyForNewBackup(password);
+            publicKey = decryption.init_with_private_key(keyInfo.key);
+            authData.private_key_salt = keyInfo.salt;
+            authData.private_key_iterations = keyInfo.iterations;
+        } else {
+            publicKey = decryption.generate_key();
+        }
+
+        authData.public_key = publicKey;
+
+        return {
+            algorithm: olmlib.MEGOLM_BACKUP_ALGORITHM,
+            auth_data: authData,
+            recovery_key: encodeRecoveryKey(decryption.get_private_key()),
+        };
+    } finally {
+        decryption.free();
+    }
+};
+
+/**
+ * Create a new key backup version and enable it, using the information return
+ * from prepareKeyBackupVersion.
+ *
+ * @param {object} info Info object from prepareKeyBackupVersion
+ * @returns {Promise<object>} Object with 'version' param indicating the version created
+ */
+MatrixClient.prototype.createKeyBackupVersion = function(info) {
+    if (this._crypto === null) {
+        throw new Error("End-to-end encryption disabled");
+    }
+
+    const data = {
+        algorithm: info.algorithm,
+        auth_data: info.auth_data,
+    };
+    return this._crypto._signObject(data.auth_data).then(() => {
+        return this._http.authedRequest(
+            undefined, "POST", "/room_keys/version", undefined, data,
+        );
+    }).then((res) => {
+        this.enableKeyBackup({
+            algorithm: info.algorithm,
+            auth_data: info.auth_data,
+            version: res.version,
+        });
+        return res;
+    });
+};
+
+MatrixClient.prototype.deleteKeyBackupVersion = function(version) {
+    if (this._crypto === null) {
+        throw new Error("End-to-end encryption disabled");
+    }
+
+    // If we're currently backing up to this backup... stop.
+    // (We start using it automatically in createKeyBackupVersion
+    // so this is symmetrical).
+    if (this._crypto.backupInfo && this._crypto.backupInfo.version === version) {
+        this.disableKeyBackup();
+    }
+
+    const path = utils.encodeUri("/room_keys/version/$version", {
+        $version: version,
+    });
+
+    return this._http.authedRequest(
+        undefined, "DELETE", path, undefined, undefined,
+    );
+};
+
+MatrixClient.prototype._makeKeyBackupPath = function(roomId, sessionId, version) {
+    let path;
+    if (sessionId !== undefined) {
+        path = utils.encodeUri("/room_keys/keys/$roomId/$sessionId", {
+            $roomId: roomId,
+            $sessionId: sessionId,
+        });
+    } else if (roomId !== undefined) {
+        path = utils.encodeUri("/room_keys/keys/$roomId", {
+            $roomId: roomId,
+        });
+    } else {
+        path = "/room_keys/keys";
+    }
+    const queryData = version === undefined ? undefined : { version: version };
+    return {
+        path: path,
+        queryData: queryData,
+    };
+};
+
+/**
+ * Back up session keys to the homeserver.
+ * @param {string} roomId ID of the room that the keys are for Optional.
+ * @param {string} sessionId ID of the session that the keys are for Optional.
+ * @param {integer} version backup version Optional.
+ * @param {object} data Object keys to send
+ * @return {module:client.Promise} a promise that will resolve when the keys
+ * are uploaded
+ */
+MatrixClient.prototype.sendKeyBackup = function(roomId, sessionId, version, data) {
+    if (this._crypto === null) {
+        throw new Error("End-to-end encryption disabled");
+    }
+
+    const path = this._makeKeyBackupPath(roomId, sessionId, version);
+    return this._http.authedRequest(
+        undefined, "PUT", path.path, path.queryData, data,
+    );
+};
+
+MatrixClient.prototype.backupAllGroupSessions = function(version) {
+    if (this._crypto === null) {
+        throw new Error("End-to-end encryption disabled");
+    }
+
+    return this._crypto.backupAllGroupSessions(version);
+};
+
+MatrixClient.prototype.isValidRecoveryKey = function(recoveryKey) {
+    try {
+        decodeRecoveryKey(recoveryKey);
+        return true;
+    } catch (e) {
+        return false;
+    }
+};
+
+MatrixClient.prototype.restoreKeyBackupWithPassword = async function(
+    password, targetRoomId, targetSessionId, version,
+) {
+    const backupInfo = await this.getKeyBackupVersion();
+
+    const privKey = await keyForExistingBackup(backupInfo, password);
+    return this._restoreKeyBackup(
+        privKey, targetRoomId, targetSessionId, version,
+    );
+};
+
+MatrixClient.prototype.restoreKeyBackupWithRecoveryKey = function(
+    recoveryKey, targetRoomId, targetSessionId, version,
+) {
+    const privKey = decodeRecoveryKey(recoveryKey);
+    return this._restoreKeyBackup(
+        privKey, targetRoomId, targetSessionId, version,
+    );
+};
+
+MatrixClient.prototype._restoreKeyBackup = function(
+    privKey, targetRoomId, targetSessionId, version,
+) {
+    if (this._crypto === null) {
+        throw new Error("End-to-end encryption disabled");
+    }
+    let totalKeyCount = 0;
+    let keys = [];
+
+    const path = this._makeKeyBackupPath(targetRoomId, targetSessionId, version);
+
+    const decryption = new global.Olm.PkDecryption();
+    try {
+        decryption.init_with_private_key(privKey);
+    } catch(e) {
+        decryption.free();
+        throw e;
+    }
+
+    return this._http.authedRequest(
+        undefined, "GET", path.path, path.queryData,
+    ).then((res) => {
+        if (res.rooms) {
+            for (const [roomId, roomData] of Object.entries(res.rooms)) {
+                if (!roomData.sessions) continue;
+
+                totalKeyCount += Object.keys(roomData.sessions).length;
+                const roomKeys = keysFromRecoverySession(
+                    roomData.sessions, decryption, roomId, roomKeys,
+                );
+                for (const k of roomKeys) {
+                    k.room_id = roomId;
+                    keys.push(k);
+                }
+            }
+        } else if (res.sessions) {
+            totalKeyCount = Object.keys(res.sessions).length;
+            keys = keysFromRecoverySession(
+                res.sessions, decryption, targetRoomId, keys,
+            );
+        } else {
+            totalKeyCount = 1;
+            try {
+                const key = keyFromRecoverySession(res, decryption);
+                key.room_id = targetRoomId;
+                key.session_id = targetSessionId;
+                keys.push(key);
+            } catch (e) {
+                console.log("Failed to decrypt session from backup");
+            }
+        }
+
+        return this.importRoomKeys(keys);
+    }).then(() => {
+        return {total: totalKeyCount, imported: keys.length};
+    }).finally(() => {
+        decryption.free();
+    });
+};
+
+MatrixClient.prototype.deleteKeysFromBackup = function(roomId, sessionId, version) {
+    if (this._crypto === null) {
+        throw new Error("End-to-end encryption disabled");
+    }
+
+    const path = this._makeKeyBackupPath(roomId, sessionId, version);
+    return this._http.authedRequest(
+        undefined, "DELETE", path.path, path.queryData,
+    );
+};
+
 // Group ops
 // =========
 // Operations on groups that come down the sync stream (ie. ones the
@@ -742,6 +1123,17 @@ MatrixClient.prototype.getGroups = function() {
     return this.store.getGroups();
 };
 
+/**
+ * Get the config for the media repository.
+ * @param {module:client.callback} callback Optional.
+ * @return {module:client.Promise} Resolves with an object containing the config.
+ */
+MatrixClient.prototype.getMediaConfig = function(callback) {
+    return this._http.authedRequestWithPrefix(
+        callback, "GET", "/config", undefined, undefined, httpApi.PREFIX_MEDIA_R0,
+    );
+};
+
 // Room ops
 // ========
 
@@ -765,6 +1157,37 @@ MatrixClient.prototype.getRooms = function() {
     return this.store.getRooms();
 };
 
+/**
+ * Retrieve all rooms that should be displayed to the user
+ * This is essentially getRooms() with some rooms filtered out, eg. old versions
+ * of rooms that have been replaced or (in future) other rooms that have been
+ * marked at the protocol level as not to be displayed to the user.
+ * @return {Room[]} A list of rooms, or an empty list if there is no data store.
+ */
+MatrixClient.prototype.getVisibleRooms = function() {
+    const allRooms = this.store.getRooms();
+
+    const replacedRooms = new Set();
+    for (const r of allRooms) {
+        const createEvent = r.currentState.getStateEvents('m.room.create', '');
+        // invites are included in this list and we don't know their create events yet
+        if (createEvent) {
+            const predecessor = createEvent.getContent()['predecessor'];
+            if (predecessor && predecessor['room_id']) {
+                replacedRooms.add(predecessor['room_id']);
+            }
+        }
+    }
+
+    return allRooms.filter((r) => {
+        const tombstone = r.currentState.getStateEvents('m.room.tombstone', '');
+        if (tombstone && replacedRooms.has(r.roomId)) {
+            return false;
+        }
+        return true;
+    });
+};
+
 /**
  * Retrieve a user.
  * @param {string} userId The user ID to retrieve.
@@ -857,6 +1280,8 @@ MatrixClient.prototype.isUserIgnored = function(userId) {
  * </strong> Default: true.
  * @param {boolean} opts.inviteSignUrl If the caller has a keypair 3pid invite,
  *                                     the signing URL is passed in this parameter.
+ * @param {string[]} opts.viaServers The server names to try and join through in
+ *                                   addition to those that are automatically chosen.
  * @param {module:client.callback} callback Optional.
  * @return {module:client.Promise} Resolves: Room object.
  * @return {module:http-api.MatrixError} Rejects: with an error response.
@@ -885,6 +1310,13 @@ MatrixClient.prototype.joinRoom = function(roomIdOrAlias, opts, callback) {
         );
     }
 
+    const queryString = {};
+    if (opts.viaServers) {
+        queryString["server_name"] = opts.viaServers;
+    }
+
+    const reqOpts = {qsStringifyOptions: {arrayFormat: 'repeat'}};
+
     const defer = Promise.defer();
 
     const self = this;
@@ -895,7 +1327,8 @@ MatrixClient.prototype.joinRoom = function(roomIdOrAlias, opts, callback) {
         }
 
         const path = utils.encodeUri("/join/$roomid", { $roomid: roomIdOrAlias});
-        return self._http.authedRequest(undefined, "POST", path, undefined, data);
+        return self._http.authedRequest(
+            undefined, "POST", path, queryString, data, reqOpts);
     }).then(function(res) {
         const roomId = res.room_id;
         const syncApi = new SyncApi(self, self._clientOpts);
@@ -1115,6 +1548,13 @@ MatrixClient.prototype.sendEvent = function(roomId, eventType, content, txnId,
         room.addPendingEvent(localEvent, txnId);
     }
 
+    // addPendingEvent can change the state to NOT_SENT if it believes
+    // that there's other events that have failed. We won't bother to
+    // try sending the event if the state has changed as such.
+    if (localEvent.status === EventStatus.NOT_SENT) {
+        return Promise.reject(new Error("Event blocked by other events not yet sent"));
+    }
+
     return _sendEvent(this, room, localEvent, callback);
 };
 
@@ -1934,14 +2374,6 @@ MatrixClient.prototype.scrollback = function(room, limit, callback) {
     // reduce the required number of events appropriately
     limit = limit - numAdded;
 
-    const path = utils.encodeUri(
-        "/rooms/$roomId/messages", {$roomId: room.roomId},
-    );
-    const params = {
-        from: room.oldState.paginationToken,
-        limit: limit,
-        dir: 'b',
-    };
     const defer = Promise.defer();
     info = {
         promise: defer.promise,
@@ -1951,9 +2383,17 @@ MatrixClient.prototype.scrollback = function(room, limit, callback) {
     // wait for a time before doing this request
     // (which may be 0 in order not to special case the code paths)
     Promise.delay(timeToWaitMs).then(function() {
-        return self._http.authedRequest(callback, "GET", path, params);
+        return self._createMessagesRequest(
+            room.roomId,
+            room.oldState.paginationToken,
+            limit,
+            'b');
     }).done(function(res) {
         const matrixEvents = utils.map(res.chunk, _PojoToMatrixEventMapper(self));
+        if (res.state) {
+            const stateEvents = utils.map(res.state, _PojoToMatrixEventMapper(self));
+            room.currentState.setUnknownStateEvents(stateEvents);
+        }
         room.addEventsToTimeline(matrixEvents, true, room.getLiveTimeline());
         room.oldState.paginationToken = res.end;
         if (res.chunk.length === 0) {
@@ -1972,73 +2412,6 @@ MatrixClient.prototype.scrollback = function(room, limit, callback) {
     return defer.promise;
 };
 
-/**
- * Take an EventContext, and back/forward-fill results.
- *
- * @param {module:models/event-context.EventContext} eventContext  context
- *    object to be updated
- * @param {Object}  opts
- * @param {boolean} opts.backwards  true to fill backwards, false to go forwards
- * @param {boolean} opts.limit      number of events to request
- *
- * @return {module:client.Promise} Resolves: updated EventContext object
- * @return {Error} Rejects: with an error response.
- */
-MatrixClient.prototype.paginateEventContext = function(eventContext, opts) {
-    // TODO: we should implement a backoff (as per scrollback()) to deal more
-    // nicely with HTTP errors.
-    opts = opts || {};
-    const backwards = opts.backwards || false;
-
-    const token = eventContext.getPaginateToken(backwards);
-    if (!token) {
-        // no more results.
-        return Promise.reject(new Error("No paginate token"));
-    }
-
-    const dir = backwards ? 'b' : 'f';
-    const pendingRequest = eventContext._paginateRequests[dir];
-
-    if (pendingRequest) {
-        // already a request in progress - return the existing promise
-        return pendingRequest;
-    }
-
-    const path = utils.encodeUri(
-        "/rooms/$roomId/messages", {$roomId: eventContext.getEvent().getRoomId()},
-    );
-    const params = {
-        from: token,
-        limit: ('limit' in opts) ? opts.limit : 30,
-        dir: dir,
-    };
-
-    const self = this;
-    const promise =
-        self._http.authedRequest(undefined, "GET", path, params,
-    ).then(function(res) {
-        let token = res.end;
-        if (res.chunk.length === 0) {
-            token = null;
-        } else {
-            const matrixEvents = utils.map(res.chunk, self.getEventMapper());
-            if (backwards) {
-                // eventContext expects the events in timeline order, but
-                // back-pagination returns them in reverse order.
-                matrixEvents.reverse();
-            }
-            eventContext.addEvents(matrixEvents, backwards);
-        }
-        eventContext.setPaginateToken(token, backwards);
-        return eventContext;
-    }).finally(function() {
-        eventContext._paginateRequests[dir] = null;
-    });
-    eventContext._paginateRequests[dir] = promise;
-
-    return promise;
-};
-
 /**
  * Get an EventTimeline for the given event
  *
@@ -2072,11 +2445,16 @@ MatrixClient.prototype.getEventTimeline = function(timelineSet, eventId) {
         },
     );
 
+    let params = undefined;
+    if (this._clientOpts.lazyLoadMembers) {
+        params = {filter: JSON.stringify(Filter.LAZY_LOADING_MESSAGES_FILTER)};
+    }
+
     // TODO: we should implement a backoff (as per scrollback()) to deal more
     // nicely with HTTP errors.
     const self = this;
     const promise =
-        self._http.authedRequest(undefined, "GET", path,
+        self._http.authedRequest(undefined, "GET", path, params,
     ).then(function(res) {
         if (!res.event) {
             throw new Error("'event' not in '/context' result - homeserver too old?");
@@ -2103,6 +2481,9 @@ MatrixClient.prototype.getEventTimeline = function(timelineSet, eventId) {
             timeline.initialiseState(utils.map(res.state,
                                                self.getEventMapper()));
             timeline.getState(EventTimeline.FORWARDS).paginationToken = res.end;
+        } else {
+            const stateEvents = utils.map(res.state, self.getEventMapper());
+            timeline.getState(EventTimeline.BACKWARDS).setUnknownStateEvents(stateEvents);
         }
         timelineSet.addEventsToTimeline(matrixEvents, true, timeline, res.start);
 
@@ -2117,6 +2498,49 @@ MatrixClient.prototype.getEventTimeline = function(timelineSet, eventId) {
     return promise;
 };
 
+/**
+ * Makes a request to /messages with the appropriate lazy loading filter set.
+ * XXX: if we do get rid of scrollback (as it's not used at the moment),
+ * we could inline this method again in paginateEventTimeline as that would
+ * then be the only call-site
+ * @param {string} roomId
+ * @param {string} fromToken
+ * @param {number} limit the maximum amount of events the retrieve
+ * @param {string} dir 'f' or 'b'
+ * @param {Filter} timelineFilter the timeline filter to pass
+ * @return {Promise}
+ */
+MatrixClient.prototype._createMessagesRequest =
+function(roomId, fromToken, limit, dir, timelineFilter = undefined) {
+    const path = utils.encodeUri(
+        "/rooms/$roomId/messages", {$roomId: roomId},
+    );
+    if (limit === undefined) {
+        limit = 30;
+    }
+    const params = {
+        from: fromToken,
+        limit: limit,
+        dir: dir,
+    };
+
+    let filter = null;
+    if (this._clientOpts.lazyLoadMembers) {
+        // create a shallow copy of LAZY_LOADING_MESSAGES_FILTER,
+        // so the timelineFilter doesn't get written into it below
+        filter = Object.assign({}, Filter.LAZY_LOADING_MESSAGES_FILTER);
+    }
+    if (timelineFilter) {
+        // XXX: it's horrific that /messages' filter parameter doesn't match
+        // /sync's one - see https://matrix.org/jira/browse/SPEC-451
+        filter = filter || {};
+        Object.assign(filter, timelineFilter.getRoomTimelineFilterComponent());
+    }
+    if (filter) {
+        params.filter = JSON.stringify(filter);
+    }
+    return this._http.authedRequest(undefined, "GET", path, params);
+};
 
 /**
  * Take an EventTimeline, and back/forward-fill results.
@@ -2211,25 +2635,18 @@ MatrixClient.prototype.paginateEventTimeline = function(eventTimeline, opts) {
             throw new Error("Unknown room " + eventTimeline.getRoomId());
         }
 
-        path = utils.encodeUri(
-            "/rooms/$roomId/messages", {$roomId: eventTimeline.getRoomId()},
-        );
-        params = {
-            from: token,
-            limit: ('limit' in opts) ? opts.limit : 30,
-            dir: dir,
-        };
-
-        const filter = eventTimeline.getFilter();
-        if (filter) {
-            // XXX: it's horrific that /messages' filter parameter doesn't match
-            // /sync's one - see https://matrix.org/jira/browse/SPEC-451
-            params.filter = JSON.stringify(filter.getRoomTimelineFilterComponent());
-        }
-
-        promise =
-            this._http.authedRequest(undefined, "GET", path, params,
-        ).then(function(res) {
+        promise = this._createMessagesRequest(
+            eventTimeline.getRoomId(),
+            token,
+            opts.limit,
+            dir,
+            eventTimeline.getFilter());
+        promise.then(function(res) {
+            if (res.state) {
+                const roomState = eventTimeline.getState(dir);
+                const stateEvents = utils.map(res.state, self.getEventMapper());
+                roomState.setUnknownStateEvents(stateEvents);
+            }
             const token = res.end;
             const matrixEvents = utils.map(res.chunk, self.getEventMapper());
             eventTimeline.getTimelineSet()
@@ -3034,8 +3451,11 @@ MatrixClient.prototype.getTurnServers = function() {
  *
  * @param {Boolean=} opts.disablePresence True to perform syncing without automatically
  * updating presence.
+ * @param {Boolean=} opts.lazyLoadMembers True to not load all membership events during
+ * initial sync but fetch them when needed by calling `loadOutOfBandMembers`
+ * This will override the filter option at this moment.
  */
-MatrixClient.prototype.startClient = function(opts) {
+MatrixClient.prototype.startClient = async function(opts) {
     if (this.clientRunning) {
         // client is already running.
         return;
@@ -3073,11 +3493,29 @@ MatrixClient.prototype.startClient = function(opts) {
         return this._canResetTimelineCallback(roomId);
     };
     this._clientOpts = opts;
-
     this._syncApi = new SyncApi(this, opts);
     this._syncApi.sync();
 };
 
+/**
+ * store client options with boolean/string/numeric values
+ * to know in the next session what flags the sync data was
+ * created with (e.g. lazy loading)
+ * @param {object} opts the complete set of client options
+ * @return {Promise} for store operation */
+MatrixClient.prototype._storeClientOptions = function() {
+    const primTypes = ["boolean", "string", "number"];
+    const serializableOpts = Object.entries(this._clientOpts)
+        .filter(([key, value]) => {
+            return primTypes.includes(typeof value);
+        })
+        .reduce((obj, [key, value]) => {
+            obj[key] = value;
+            return obj;
+        }, {});
+    return this.store.storeClientOptions(serializableOpts);
+};
+
 /**
  * High level helper method to stop the client from polling and allow a
  * clean shutdown.
@@ -3100,6 +3538,36 @@ MatrixClient.prototype.stopClient = function() {
     global.clearTimeout(this._checkTurnServersTimeoutID);
 };
 
+/*
+ * Query the server to see if it support members lazy loading
+ * @return {Promise<boolean>} true if server supports lazy loading
+ */
+MatrixClient.prototype.doesServerSupportLazyLoading = async function() {
+    if (this._serverSupportsLazyLoading === null) {
+        const response = await this._http.request(
+            undefined, // callback
+            "GET", "/_matrix/client/versions",
+            undefined, // queryParams
+            undefined, // data
+            {
+                prefix: '',
+            },
+        );
+        const unstableFeatures = response["unstable_features"];
+        this._serverSupportsLazyLoading =
+            unstableFeatures && unstableFeatures["m.lazy_load_members"];
+    }
+    return this._serverSupportsLazyLoading;
+};
+
+/*
+ * Get if lazy loading members is being used.
+ * @return {boolean} Whether or not members are lazy loaded by this client
+ */
+MatrixClient.prototype.hasLazyLoadMembersEnabled = function() {
+    return !!this._clientOpts.lazyLoadMembers;
+};
+
 /*
  * Set a function which is called when /sync returns a 'limited' response.
  * It is called with a room ID and returns a boolean. It should return 'true' if the SDK
@@ -3395,14 +3863,7 @@ MatrixClient.prototype.getEventMapper = function() {
  * @return {string} A new client secret
  */
 MatrixClient.prototype.generateClientSecret = function() {
-    let ret = "";
-    const chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
-
-    for (let i = 0; i < 32; i++) {
-        ret += chars.charAt(Math.floor(Math.random() * chars.length));
-    }
-
-    return ret;
+    return randomString(32);
 };
 
 /** */
@@ -3445,6 +3906,12 @@ module.exports.CRYPTO_ENABLED = CRYPTO_ENABLED;
  * a state of SYNCING. <i>This is the equivalent of "syncComplete" in the
  * previous API.</i></li>
  *
+ * <li>CATCHUP: The client has detected the connection to the server might be
+ * available again and will now try to do a sync again. As this sync might take
+ * a long time (depending how long ago was last synced, and general server
+ * performance) the client is put in this mode so the UI can reflect trying
+ * to catch up with the server after losing connection.</li>
+ *
  * <li>SYNCING : The client is currently polling for new events from the server.
  * This will be called <i>after</i> processing latest events from a sync.</li>
  *
@@ -3468,11 +3935,11 @@ module.exports.CRYPTO_ENABLED = CRYPTO_ENABLED;
  *                                          +---->STOPPED
  *                                          |
  *              +----->PREPARED -------> SYNCING <--+
- *              |        ^                |  ^      |
- *              |        |                |  |      |
- *              |        |                V  |      |
- *   null ------+        |  +--------RECONNECTING   |
- *              |        |  V                       |
+ *              |                        ^  |  ^    |
+ *              |      CATCHUP ----------+  |  |    |
+ *              |        ^                  V  |    |
+ *   null ------+        |  +------- RECONNECTING   |
+ *              |        V  V                       |
  *              +------->ERROR ---------------------+
  *
  * NB: 'null' will never be emitted by this event.
@@ -3522,7 +3989,7 @@ module.exports.CRYPTO_ENABLED = CRYPTO_ENABLED;
  *
  * @param {?Object} data Data about this transition.
  *
- * @param {MatrixError} data.err The matrix error if <code>state=ERROR</code>.
+ * @param {MatrixError} data.error The matrix error if <code>state=ERROR</code>.
  *
  * @param {String} data.oldSyncToken The 'since' token passed to /sync.
  *    <code>null</code> for the first successful sync since this client was
@@ -3640,6 +4107,24 @@ module.exports.CRYPTO_ENABLED = CRYPTO_ENABLED;
  * });
  */
 
+/**
+ * Fires whenever the status of e2e key backup changes, as returned by getKeyBackupEnabled()
+ * @event module:client~MatrixClient#"crypto.keyBackupStatus"
+ * @param {bool} enabled true if key backup has been enabled, otherwise false
+ * @example
+ * matrixClient.on("crypto.keyBackupStatus", function(enabled){
+ *   if (enabled) {
+ *     [...]
+ *   }
+ * });
+ */
+
+/**
+ * Fires when we want to suggest to the user that they restore their megolm keys
+ * from backup or by cross-signing the device.
+ *
+ * @event module:client~MatrixClient#"crypto.suggestKeyRestore"
+ */
 
 // EventEmitter JSDocs
 

src/crypto/DeviceList.js

@@ -24,6 +24,7 @@ limitations under the License.
 
 import Promise from 'bluebird';
 
+import logger from '../logger';
 import DeviceInfo from './deviceinfo';
 import olmlib from './olmlib';
 import IndexedDBCryptoStore from './store/indexeddb-crypto-store';
@@ -71,6 +72,9 @@ export default class DeviceList {
         // }
         this._devices = {};
 
+        // map of identity keys to the user who owns it
+        this._userByIdentityKey = {};
+
         // which users we are tracking device status for.
         // userId -> TRACKING_STATUS_*
         this._deviceTrackingStatus = {}; // loaded from storage in load()
@@ -110,7 +114,7 @@ export default class DeviceList {
             'readwrite', [IndexedDBCryptoStore.STORE_DEVICE_DATA], (txn) => {
                 this._cryptoStore.getEndToEndDeviceData(txn, (deviceData) => {
                     if (deviceData === null) {
-                        console.log("Migrating e2e device data...");
+                        logger.log("Migrating e2e device data...");
                         this._devices = this._sessionStore.getAllEndToEndDevices() || {};
                         this._deviceTrackingStatus = (
                             this._sessionStore.getEndToEndDeviceTrackingStatus() || {}
@@ -128,6 +132,16 @@ export default class DeviceList {
                             deviceData.trackingStatus : {};
                         this._syncToken = deviceData ? deviceData.syncToken : null;
                     }
+                    this._userByIdentityKey = {};
+                    for (const user of Object.keys(this._devices)) {
+                        const userDevices = this._devices[user];
+                        for (const device of Object.keys(userDevices)) {
+                            const idKey = userDevices[device].keys['curve25519:'+device];
+                            if (idKey !== undefined) {
+                                this._userByIdentityKey[idKey] = user;
+                            }
+                        }
+                    }
                 });
             },
         );
@@ -145,6 +159,12 @@ export default class DeviceList {
         }
     }
 
+    stop() {
+        if (this._saveTimer !== null) {
+            clearTimeout(this._saveTimer);
+        }
+    }
+
     /**
      * Save the device tracking state to storage, if any changes are
      * pending other than updating the sync token
@@ -190,7 +210,7 @@ export default class DeviceList {
             const resolveSavePromise = this._resolveSavePromise;
             this._savePromiseTime = targetTime;
             this._saveTimer = setTimeout(() => {
-                console.log('Saving device tracking data at token ' + this._syncToken);
+                logger.log('Saving device tracking data at token ' + this._syncToken);
                 // null out savePromise now (after the delay but before the write),
                 // otherwise we could return the existing promise when the save has
                 // actually already happened. Likewise for the dirty flag.
@@ -258,7 +278,7 @@ export default class DeviceList {
             if (this._keyDownloadsInProgressByUser[u]) {
                 // already a key download in progress/queued for this user; its results
                 // will be good enough for us.
-                console.log(
+                logger.log(
                     `downloadKeys: already have a download in progress for ` +
                     `${u}: awaiting its result`,
                 );
@@ -269,13 +289,13 @@ export default class DeviceList {
         });
 
         if (usersToDownload.length != 0) {
-            console.log("downloadKeys: downloading for", usersToDownload);
+            logger.log("downloadKeys: downloading for", usersToDownload);
             const downloadPromise = this._doKeyDownload(usersToDownload);
             promises.push(downloadPromise);
         }
 
         if (promises.length === 0) {
-            console.log("downloadKeys: already have all necessary keys");
+            logger.log("downloadKeys: already have all necessary keys");
         }
 
         return Promise.all(promises).then(() => {
@@ -357,13 +377,17 @@ export default class DeviceList {
     /**
      * Find a device by curve25519 identity key
      *
-     * @param {string} userId     owner of the device
      * @param {string} algorithm  encryption algorithm
      * @param {string} senderKey  curve25519 key to match
      *
      * @return {module:crypto/deviceinfo?}
      */
-    getDeviceByIdentityKey(userId, algorithm, senderKey) {
+    getDeviceByIdentityKey(algorithm, senderKey) {
+        const userId = this._userByIdentityKey[senderKey];
+        if (!userId) {
+            return null;
+        }
+
         if (
             algorithm !== olmlib.OLM_ALGORITHM &&
             algorithm !== olmlib.MEGOLM_ALGORITHM
@@ -408,7 +432,23 @@ export default class DeviceList {
      * @param {Object} devs New device info for user
      */
     storeDevicesForUser(u, devs) {
+        // remove previous devices from _userByIdentityKey
+        if (this._devices[u] !== undefined) {
+            for (const [deviceId, dev] of Object.entries(this._devices[u])) {
+                const identityKey = dev.keys['curve25519:'+deviceId];
+
+                delete this._userByIdentityKey[identityKey];
+            }
+        }
+
         this._devices[u] = devs;
+
+        // add new ones
+        for (const [deviceId, dev] of Object.entries(devs)) {
+            const identityKey = dev.keys['curve25519:'+deviceId];
+
+            this._userByIdentityKey[identityKey] = u;
+        }
         this._dirty = true;
     }
 
@@ -433,7 +473,7 @@ export default class DeviceList {
             throw new Error('userId must be a string; was '+userId);
         }
         if (!this._deviceTrackingStatus[userId]) {
-            console.log('Now tracking device list for ' + userId);
+            logger.log('Now tracking device list for ' + userId);
             this._deviceTrackingStatus[userId] = TRACKING_STATUS_PENDING_DOWNLOAD;
         }
         // we don't yet persist the tracking status, since there may be a lot
@@ -452,7 +492,7 @@ export default class DeviceList {
      */
     stopTrackingDeviceList(userId) {
         if (this._deviceTrackingStatus[userId]) {
-            console.log('No longer tracking device list for ' + userId);
+            logger.log('No longer tracking device list for ' + userId);
             this._deviceTrackingStatus[userId] = TRACKING_STATUS_NOT_TRACKED;
 
             // we don't yet persist the tracking status, since there may be a lot
@@ -487,7 +527,7 @@ export default class DeviceList {
      */
     invalidateUserDeviceList(userId) {
         if (this._deviceTrackingStatus[userId]) {
-            console.log("Marking device list outdated for", userId);
+            logger.log("Marking device list outdated for", userId);
             this._deviceTrackingStatus[userId] = TRACKING_STATUS_PENDING_DOWNLOAD;
 
             // we don't yet persist the tracking status, since there may be a lot
@@ -525,7 +565,23 @@ export default class DeviceList {
      * @param {Object} devices deviceId->{object} the new devices
      */
     _setRawStoredDevicesForUser(userId, devices) {
+        // remove old devices from _userByIdentityKey
+        if (this._devices[userId] !== undefined) {
+            for (const [deviceId, dev] of Object.entries(this._devices[userId])) {
+                const identityKey = dev.keys['curve25519:'+deviceId];
+
+                delete this._userByIdentityKey[identityKey];
+            }
+        }
+
         this._devices[userId] = devices;
+
+        // add new devices into _userByIdentityKey
+        for (const [deviceId, dev] of Object.entries(devices)) {
+            const identityKey = dev.keys['curve25519:'+deviceId];
+
+            this._userByIdentityKey[identityKey] = userId;
+        }
     }
 
     /**
@@ -550,7 +606,7 @@ export default class DeviceList {
         ).then(() => {
             finished(true);
         }, (e) => {
-            console.error(
+            logger.error(
                 'Error downloading keys for ' + users + ":", e,
             );
             finished(false);
@@ -573,7 +629,7 @@ export default class DeviceList {
                 // since we started this request. If that happens, we should
                 // ignore the completion of the first one.
                 if (this._keyDownloadsInProgressByUser[u] !== prom) {
-                    console.log('Another update in the queue for', u,
+                    logger.log('Another update in the queue for', u,
                                 '- not marking up-to-date');
                     return;
                 }
@@ -584,7 +640,7 @@ export default class DeviceList {
                         // we didn't get any new invalidations since this download started:
                         // this user's device list is now up to date.
                         this._deviceTrackingStatus[u] = TRACKING_STATUS_UP_TO_DATE;
-                        console.log("Device list for", u, "now up to date");
+                        logger.log("Device list for", u, "now up to date");
                     } else {
                         this._deviceTrackingStatus[u] = TRACKING_STATUS_PENDING_DOWNLOAD;
                     }
@@ -659,7 +715,7 @@ class DeviceListUpdateSerialiser {
 
         if (this._downloadInProgress) {
             // just queue up these users
-            console.log('Queued key download for', users);
+            logger.log('Queued key download for', users);
             return this._queuedQueryDeferred.promise;
         }
 
@@ -679,7 +735,7 @@ class DeviceListUpdateSerialiser {
         const deferred = this._queuedQueryDeferred;
         this._queuedQueryDeferred = null;
 
-        console.log('Starting key download for', downloadUsers);
+        logger.log('Starting key download for', downloadUsers);
         this._downloadInProgress = true;
 
         const opts = {};
@@ -706,7 +762,7 @@ class DeviceListUpdateSerialiser {
 
             return prom;
         }).done(() => {
-            console.log('Completed key download for ' + downloadUsers);
+            logger.log('Completed key download for ' + downloadUsers);
 
             this._downloadInProgress = false;
             deferred.resolve();
@@ -716,7 +772,7 @@ class DeviceListUpdateSerialiser {
                 this._doQueuedQueries();
             }
         }, (e) => {
-            console.warn('Error downloading keys for ' + downloadUsers + ':', e);
+            logger.warn('Error downloading keys for ' + downloadUsers + ':', e);
             this._downloadInProgress = false;
             deferred.reject(e);
         });
@@ -725,7 +781,7 @@ class DeviceListUpdateSerialiser {
     }
 
     async _processQueryResponseForUser(userId, response) {
-        console.log('got keys for ' + userId + ':', response);
+        logger.log('got keys for ' + userId + ':', response);
 
         // map from deviceid -> deviceinfo for this user
         const userStore = {};
@@ -763,7 +819,7 @@ async function _updateStoredDeviceKeysForUser(_olmDevice, userId, userStore,
         }
 
         if (!(deviceId in userResult)) {
-            console.log("Device " + userId + ":" + deviceId +
+            logger.log("Device " + userId + ":" + deviceId +
                 " has been removed");
             delete userStore[deviceId];
             updated = true;
@@ -780,12 +836,12 @@ async function _updateStoredDeviceKeysForUser(_olmDevice, userId, userStore,
         // check that the user_id and device_id in the response object are
         // correct
         if (deviceResult.user_id !== userId) {
-            console.warn("Mismatched user_id " + deviceResult.user_id +
+            logger.warn("Mismatched user_id " + deviceResult.user_id +
                " in keys from " + userId + ":" + deviceId);
             continue;
         }
         if (deviceResult.device_id !== deviceId) {
-            console.warn("Mismatched device_id " + deviceResult.device_id +
+            logger.warn("Mismatched device_id " + deviceResult.device_id +
                " in keys from " + userId + ":" + deviceId);
             continue;
         }
@@ -815,7 +871,7 @@ async function _storeDeviceKeys(_olmDevice, userStore, deviceResult) {
     const signKeyId = "ed25519:" + deviceId;
     const signKey = deviceResult.keys[signKeyId];
     if (!signKey) {
-        console.warn("Device " + userId + ":" + deviceId +
+        logger.warn("Device " + userId + ":" + deviceId +
             " has no ed25519 key");
         return false;
     }
@@ -825,7 +881,7 @@ async function _storeDeviceKeys(_olmDevice, userStore, deviceResult) {
     try {
         await olmlib.verifySignature(_olmDevice, deviceResult, userId, deviceId, signKey);
     } catch (e) {
-        console.warn("Unable to verify signature on device " +
+        logger.warn("Unable to verify signature on device " +
             userId + ":" + deviceId + ":" + e);
         return false;
     }
@@ -842,7 +898,7 @@ async function _storeDeviceKeys(_olmDevice, userStore, deviceResult) {
             // best off sticking with the original keys.
             //
             // Should we warn the user about it somehow?
-            console.warn("Ed25519 key for device " + userId + ":" +
+            logger.warn("Ed25519 key for device " + userId + ":" +
                deviceId + " has changed");
             return false;
         }

src/crypto/OlmDevice.js

@@ -15,19 +15,9 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
+import logger from '../logger';
 import IndexedDBCryptoStore from './store/indexeddb-crypto-store';
 
-/**
- * olm.js wrapper
- *
- * @module crypto/OlmDevice
- */
-const Olm = global.Olm;
-if (!Olm) {
-    throw new Error("global.Olm is not defined");
-}
-
-
 // The maximum size of an event is 65K, and we base64 the content, so this is a
 // reasonable approximation to the biggest plaintext we can encrypt.
 const MAX_PLAINTEXT_LENGTH = 65536 * 3 / 4;
@@ -127,7 +117,7 @@ OlmDevice.prototype.init = async function() {
     await this._migrateFromSessionStore();
 
     let e2eKeys;
-    const account = new Olm.Account();
+    const account = new global.Olm.Account();
     try {
         await _initialiseAccount(
             this._sessionStore, this._cryptoStore, this._pickleKey, account,
@@ -161,7 +151,7 @@ async function _initialiseAccount(sessionStore, cryptoStore, pickleKey, account)
  * @return {array} The version of Olm.
  */
 OlmDevice.getOlmVersion = function() {
-    return Olm.get_library_version();
+    return global.Olm.get_library_version();
 };
 
 OlmDevice.prototype._migrateFromSessionStore = async function() {
@@ -173,7 +163,7 @@ OlmDevice.prototype._migrateFromSessionStore = async function() {
                     // Migrate from sessionStore
                     pickledAccount = this._sessionStore.getEndToEndAccount();
                     if (pickledAccount !== null) {
-                        console.log("Migrating account from session store");
+                        logger.log("Migrating account from session store");
                         this._cryptoStore.storeAccount(txn, pickledAccount);
                     }
                 }
@@ -195,7 +185,7 @@ OlmDevice.prototype._migrateFromSessionStore = async function() {
                 // has run against the same localstorage and created some spurious sessions.
                 this._cryptoStore.countEndToEndSessions(txn, (count) => {
                     if (count) {
-                        console.log("Crypto store already has sessions: not migrating");
+                        logger.log("Crypto store already has sessions: not migrating");
                         return;
                     }
                     let numSessions = 0;
@@ -207,7 +197,7 @@ OlmDevice.prototype._migrateFromSessionStore = async function() {
                             );
                         }
                     }
-                    console.log(
+                    logger.log(
                         "Migrating " + numSessions + " sessions from session store",
                     );
                 });
@@ -236,14 +226,14 @@ OlmDevice.prototype._migrateFromSessionStore = async function() {
                             ), txn,
                         );
                     } catch (e) {
-                        console.warn(
+                        logger.warn(
                             "Failed to migrate session " + s.senderKey + "/" +
                             s.sessionId + ": " + e.stack || e,
                         );
                     }
                     ++numIbSessions;
                 }
-                console.log(
+                logger.log(
                     "Migrated " + numIbSessions +
                     " inbound group sessions from session store",
                 );
@@ -268,7 +258,7 @@ OlmDevice.prototype._migrateFromSessionStore = async function() {
  */
 OlmDevice.prototype._getAccount = function(txn, func) {
     this._cryptoStore.getAccount(txn, (pickledAccount) => {
-        const account = new Olm.Account();
+        const account = new global.Olm.Account();
         try {
             account.unpickle(this._pickleKey, pickledAccount);
             func(account);
@@ -305,8 +295,8 @@ OlmDevice.prototype._storeAccount = function(txn, account) {
  */
 OlmDevice.prototype._getSession = function(deviceKey, sessionId, txn, func) {
     this._cryptoStore.getEndToEndSession(
-        deviceKey, sessionId, txn, (pickledSession) => {
-            this._unpickleSession(pickledSession, func);
+        deviceKey, sessionId, txn, (sessionInfo) => {
+            this._unpickleSession(sessionInfo, func);
         },
     );
 };
@@ -316,15 +306,17 @@ OlmDevice.prototype._getSession = function(deviceKey, sessionId, txn, func) {
  * function with it. The session object is destroyed once the function
  * returns.
  *
- * @param {string} pickledSession
+ * @param {object} sessionInfo
  * @param {function} func
  * @private
  */
-OlmDevice.prototype._unpickleSession = function(pickledSession, func) {
-    const session = new Olm.Session();
+OlmDevice.prototype._unpickleSession = function(sessionInfo, func) {
+    const session = new global.Olm.Session();
     try {
-        session.unpickle(this._pickleKey, pickledSession);
-        func(session);
+        session.unpickle(this._pickleKey, sessionInfo.session);
+        const unpickledSessInfo = Object.assign({}, sessionInfo, {session});
+
+        func(unpickledSessInfo);
     } finally {
         session.free();
     }
@@ -334,14 +326,17 @@ OlmDevice.prototype._unpickleSession = function(pickledSession, func) {
  * store our OlmSession in the session store
  *
  * @param {string} deviceKey
- * @param {OlmSession} session
+ * @param {object} sessionInfo {session: OlmSession, lastReceivedMessageTs: int}
  * @param {*} txn Opaque transaction object from cryptoStore.doTxn()
  * @private
  */
-OlmDevice.prototype._saveSession = function(deviceKey, session, txn) {
-    const pickledSession = session.pickle(this._pickleKey);
+OlmDevice.prototype._saveSession = function(deviceKey, sessionInfo, txn) {
+    const sessionId = sessionInfo.session.session_id();
+    const pickledSessionInfo = Object.assign(sessionInfo, {
+        session: sessionInfo.session.pickle(this._pickleKey),
+    });
     this._cryptoStore.storeEndToEndSession(
-        deviceKey, session.session_id(), pickledSession, txn,
+        deviceKey, sessionId, pickledSessionInfo, txn,
     );
 };
 
@@ -354,7 +349,7 @@ OlmDevice.prototype._saveSession = function(deviceKey, session, txn) {
  * @private
  */
 OlmDevice.prototype._getUtility = function(func) {
-    const utility = new Olm.Utility();
+    const utility = new global.Olm.Utility();
     try {
         return func(utility);
     } finally {
@@ -466,12 +461,19 @@ OlmDevice.prototype.createOutboundSession = async function(
         ],
         (txn) => {
             this._getAccount(txn, (account) => {
-                const session = new Olm.Session();
+                const session = new global.Olm.Session();
                 try {
                     session.create_outbound(account, theirIdentityKey, theirOneTimeKey);
                     newSessionId = session.session_id();
                     this._storeAccount(txn, account);
-                    this._saveSession(theirIdentityKey, session, txn);
+                    const sessionInfo = {
+                        session,
+                        // Pretend we've received a message at this point, otherwise
+                        // if we try to send a message to the device, it won't use
+                        // this session
+                        lastReceivedMessageTs: Date.now(),
+                    };
+                    this._saveSession(theirIdentityKey, sessionInfo, txn);
                 } finally {
                     session.free();
                 }
@@ -510,7 +512,7 @@ OlmDevice.prototype.createInboundSession = async function(
         ],
         (txn) => {
             this._getAccount(txn, (account) => {
-                const session = new Olm.Session();
+                const session = new global.Olm.Session();
                 try {
                     session.create_inbound_from(
                         account, theirDeviceIdentityKey, ciphertext,
@@ -520,7 +522,13 @@ OlmDevice.prototype.createInboundSession = async function(
 
                     const payloadString = session.decrypt(messageType, ciphertext);
 
-                    this._saveSession(theirDeviceIdentityKey, session, txn);
+                    const sessionInfo = {
+                        session,
+                        // this counts as a received message: set last received message time
+                        // to now
+                        lastReceivedMessageTs: Date.now(),
+                    };
+                    this._saveSession(theirDeviceIdentityKey, sessionInfo, txn);
 
                     result = {
                         payload: payloadString,
@@ -568,13 +576,30 @@ OlmDevice.prototype.getSessionIdsForDevice = async function(theirDeviceIdentityK
  * @return {Promise<?string>}  session id, or null if no established session
  */
 OlmDevice.prototype.getSessionIdForDevice = async function(theirDeviceIdentityKey) {
-    const sessionIds = await this.getSessionIdsForDevice(theirDeviceIdentityKey);
-    if (sessionIds.length === 0) {
+    const sessionInfos = await this.getSessionInfoForDevice(theirDeviceIdentityKey);
+    if (sessionInfos.length === 0) {
         return null;
     }
-    // Use the session with the lowest ID.
-    sessionIds.sort();
-    return sessionIds[0];
+    // Use the session that has most recently received a message
+    let idxOfBest = 0;
+    for (let i = 1; i < sessionInfos.length; i++) {
+        const thisSessInfo = sessionInfos[i];
+        const thisLastReceived = thisSessInfo.lastReceivedMessageTs === undefined ?
+            0 : thisSessInfo.lastReceivedMessageTs;
+
+        const bestSessInfo = sessionInfos[idxOfBest];
+        const bestLastReceived = bestSessInfo.lastReceivedMessageTs === undefined ?
+            0 : bestSessInfo.lastReceivedMessageTs;
+        if (
+            thisLastReceived > bestLastReceived || (
+                thisLastReceived === bestLastReceived &&
+                thisSessInfo.sessionId < bestSessInfo.sessionId
+            )
+        ) {
+            idxOfBest = i;
+        }
+    }
+    return sessionInfos[idxOfBest].sessionId;
 };
 
 /**
@@ -597,9 +622,10 @@ OlmDevice.prototype.getSessionInfoForDevice = async function(deviceIdentityKey)
             this._cryptoStore.getEndToEndSessions(deviceIdentityKey, txn, (sessions) => {
                 const sessionIds = Object.keys(sessions).sort();
                 for (const sessionId of sessionIds) {
-                    this._unpickleSession(sessions[sessionId], (session) => {
+                    this._unpickleSession(sessions[sessionId], (sessInfo) => {
                         info.push({
-                            hasReceivedMessage: session.has_received_message(),
+                            lastReceivedMessageTs: sessInfo.lastReceivedMessageTs,
+                            hasReceivedMessage: sessInfo.session.has_received_message(),
                             sessionId: sessionId,
                         });
                     });
@@ -630,9 +656,9 @@ OlmDevice.prototype.encryptMessage = async function(
     await this._cryptoStore.doTxn(
         'readwrite', [IndexedDBCryptoStore.STORE_SESSIONS],
         (txn) => {
-            this._getSession(theirDeviceIdentityKey, sessionId, txn, (session) => {
-                res = session.encrypt(payloadString);
-                this._saveSession(theirDeviceIdentityKey, session, txn);
+            this._getSession(theirDeviceIdentityKey, sessionId, txn, (sessionInfo) => {
+                res = sessionInfo.session.encrypt(payloadString);
+                this._saveSession(theirDeviceIdentityKey, sessionInfo, txn);
             });
         },
     );
@@ -657,9 +683,10 @@ OlmDevice.prototype.decryptMessage = async function(
     await this._cryptoStore.doTxn(
         'readwrite', [IndexedDBCryptoStore.STORE_SESSIONS],
         (txn) => {
-            this._getSession(theirDeviceIdentityKey, sessionId, txn, (session) => {
-                payloadString = session.decrypt(messageType, ciphertext);
-                this._saveSession(theirDeviceIdentityKey, session, txn);
+            this._getSession(theirDeviceIdentityKey, sessionId, txn, (sessionInfo) => {
+                payloadString = sessionInfo.session.decrypt(messageType, ciphertext);
+                sessionInfo.lastReceivedMessageTs = Date.now();
+                this._saveSession(theirDeviceIdentityKey, sessionInfo, txn);
             });
         },
     );
@@ -689,8 +716,8 @@ OlmDevice.prototype.matchesSession = async function(
     await this._cryptoStore.doTxn(
         'readonly', [IndexedDBCryptoStore.STORE_SESSIONS],
         (txn) => {
-            this._getSession(theirDeviceIdentityKey, sessionId, txn, (session) => {
-                matches = session.matches_inbound(ciphertext);
+            this._getSession(theirDeviceIdentityKey, sessionId, txn, (sessionInfo) => {
+                matches = sessionInfo.session.matches_inbound(ciphertext);
             });
         },
     );
@@ -724,11 +751,11 @@ OlmDevice.prototype._saveOutboundGroupSession = function(session) {
  */
 OlmDevice.prototype._getOutboundGroupSession = function(sessionId, func) {
     const pickled = this._outboundGroupSessionStore[sessionId];
-    if (pickled === null) {
+    if (pickled === undefined) {
         throw new Error("Unknown outbound group session " + sessionId);
     }
 
-    const session = new Olm.OutboundGroupSession();
+    const session = new global.Olm.OutboundGroupSession();
     try {
         session.unpickle(this._pickleKey, pickled);
         return func(session);
@@ -744,7 +771,7 @@ OlmDevice.prototype._getOutboundGroupSession = function(sessionId, func) {
  * @return {string} sessionId for the outbound session.
  */
 OlmDevice.prototype.createOutboundGroupSession = function() {
-    const session = new Olm.OutboundGroupSession();
+    const session = new global.Olm.OutboundGroupSession();
     try {
         session.create();
         this._saveOutboundGroupSession(session);
@@ -816,7 +843,7 @@ OlmDevice.prototype.getOutboundGroupSessionKey = function(sessionId) {
  * @return {*} result of func
  */
 OlmDevice.prototype._unpickleInboundGroupSession = function(sessionData, func) {
-    const session = new Olm.InboundGroupSession();
+    const session = new global.Olm.InboundGroupSession();
     try {
         session.unpickle(this._pickleKey, sessionData.session);
         return func(session);
@@ -889,7 +916,7 @@ OlmDevice.prototype.addInboundGroupSession = async function(
                 roomId, senderKey, sessionId, txn,
                 (existingSession, existingSessionData) => {
                     if (existingSession) {
-                        console.log(
+                        logger.log(
                             "Update for megolm session " + senderKey + "/" + sessionId,
                         );
                         // for now we just ignore updates. TODO: implement something here
@@ -897,7 +924,7 @@ OlmDevice.prototype.addInboundGroupSession = async function(
                     }
 
                     // new session.
-                    const session = new Olm.InboundGroupSession();
+                    const session = new global.Olm.InboundGroupSession();
                     try {
                         if (exportFormat) {
                             session.import_session(sessionKey);
@@ -1034,7 +1061,7 @@ OlmDevice.prototype.hasInboundSessionKeys = async function(roomId, senderKey, se
                     }
 
                     if (roomId !== sessionData.room_id) {
-                        console.warn(
+                        logger.warn(
                             `requested keys for inbound group session ${senderKey}|` +
                             `${sessionId}, with incorrect room_id ` +
                             `(expected ${sessionData.room_id}, ` +
@@ -1058,6 +1085,8 @@ OlmDevice.prototype.hasInboundSessionKeys = async function(roomId, senderKey, se
  * @param {string} roomId    room in which the message was received
  * @param {string} senderKey base64-encoded curve25519 key of the sender
  * @param {string} sessionId session identifier
+ * @param {integer} chainIndex The chain index at which to export the session.
+ *     If omitted, export at the first index we know about.
  *
  * @returns {Promise<{chain_index: number, key: string,
  *        forwarding_curve25519_key_chain: Array<string>,
@@ -1065,9 +1094,12 @@ OlmDevice.prototype.hasInboundSessionKeys = async function(roomId, senderKey, se
  *    }>}
  *    details of the session key. The key is a base64-encoded megolm key in
  *    export format.
+ *
+ * @throws Error If the given chain index could not be obtained from the known
+ *     index (ie. the given chain index is before the first we have).
  */
 OlmDevice.prototype.getInboundGroupSessionKey = async function(
-    roomId, senderKey, sessionId,
+    roomId, senderKey, sessionId, chainIndex,
 ) {
     let result;
     await this._cryptoStore.doTxn(
@@ -1078,14 +1110,19 @@ OlmDevice.prototype.getInboundGroupSessionKey = async function(
                         result = null;
                         return;
                     }
-                    const messageIndex = session.first_known_index();
+
+                    if (chainIndex === undefined) {
+                        chainIndex = session.first_known_index();
+                    }
+
+                    const exportedSession = session.export_session(chainIndex);
 
                     const claimedKeys = sessionData.keysClaimed || {};
                     const senderEd25519Key = claimedKeys.ed25519 || null;
 
                     result = {
-                        "chain_index": messageIndex,
-                        "key": session.export_session(messageIndex),
+                        "chain_index": chainIndex,
+                        "key": exportedSession,
                         "forwarding_curve25519_key_chain":
                             sessionData.forwardingCurve25519KeyChain || [],
                         "sender_claimed_ed25519_key": senderEd25519Key,
@@ -1119,6 +1156,7 @@ OlmDevice.prototype.exportInboundGroupSession = function(
             "session_id": sessionId,
             "session_key": session.export_session(messageIndex),
             "forwarding_curve25519_key_chain": session.forwardingCurve25519KeyChain || [],
+            "first_known_index": session.first_known_index(),
         };
     });
 };

src/crypto/OutgoingRoomKeyRequestManager.js

@@ -16,6 +16,7 @@ limitations under the License.
 
 import Promise from 'bluebird';
 
+import logger from '../logger';
 import utils from '../utils';
 
 /**
@@ -108,7 +109,7 @@ export default class OutgoingRoomKeyRequestManager {
      * Called when the client is stopped. Stops any running background processes.
      */
     stop() {
-        console.log('stopping OutgoingRoomKeyRequestManager');
+        logger.log('stopping OutgoingRoomKeyRequestManager');
         // stop the timer on the next run
         this._clientRunning = false;
     }
@@ -173,7 +174,7 @@ export default class OutgoingRoomKeyRequestManager {
                     // may have seen it, so we still need to send a cancellation
                     // in that case :/
 
-                    console.log(
+                    logger.log(
                         'deleting unnecessary room key request for ' +
                         stringifyRequestBody(requestBody),
                     );
@@ -201,7 +202,7 @@ export default class OutgoingRoomKeyRequestManager {
                             // the request cancelled. There is no point in
                             // sending another cancellation since the other tab
                             // will do it.
-                            console.log(
+                            logger.log(
                                 'Tried to cancel room key request for ' +
                                 stringifyRequestBody(requestBody) +
                                 ' but it was already cancelled in another tab',
@@ -222,7 +223,7 @@ export default class OutgoingRoomKeyRequestManager {
                             updatedReq,
                             andResend,
                         ).catch((e) => {
-                            console.error(
+                            logger.error(
                                 "Error sending room key request cancellation;"
                                 + " will retry later.", e,
                             );
@@ -243,6 +244,21 @@ export default class OutgoingRoomKeyRequestManager {
         });
     }
 
+    /**
+     * Look for room key requests by target device and state
+     *
+     * @param {string} userId Target user ID
+     * @param {string} deviceId Target device ID
+     *
+     * @return {Promise} resolves to a list of all the
+     *    {@link module:crypto/store/base~OutgoingRoomKeyRequest}
+     */
+    getOutgoingSentRoomKeyRequest(userId, deviceId) {
+        return this._cryptoStore.getOutgoingRoomKeyRequestsByTarget(
+            userId, deviceId, [ROOM_KEY_REQUEST_STATES.SENT],
+        );
+    }
+
     // start the background timer to send queued requests, if the timer isn't
     // already running
     _startTimer() {
@@ -261,7 +277,7 @@ export default class OutgoingRoomKeyRequestManager {
             }).catch((e) => {
                 // this should only happen if there is an indexeddb error,
                 // in which case we're a bit stuffed anyway.
-                console.warn(
+                logger.warn(
                     `error in OutgoingRoomKeyRequestManager: ${e}`,
                 );
             }).done();
@@ -282,7 +298,7 @@ export default class OutgoingRoomKeyRequestManager {
             return Promise.resolve();
         }
 
-        console.log("Looking for queued outgoing room key requests");
+        logger.log("Looking for queued outgoing room key requests");
 
         return this._cryptoStore.getOutgoingRoomKeyRequestByState([
             ROOM_KEY_REQUEST_STATES.CANCELLATION_PENDING,
@@ -290,7 +306,7 @@ export default class OutgoingRoomKeyRequestManager {
             ROOM_KEY_REQUEST_STATES.UNSENT,
         ]).then((req) => {
             if (!req) {
-                console.log("No more outgoing room key requests");
+                logger.log("No more outgoing room key requests");
                 this._sendOutgoingRoomKeyRequestsTimer = null;
                 return;
             }
@@ -312,7 +328,7 @@ export default class OutgoingRoomKeyRequestManager {
                 // go around the loop again
                 return this._sendOutgoingRoomKeyRequests();
             }).catch((e) => {
-                console.error("Error sending room key request; will retry later.", e);
+                logger.error("Error sending room key request; will retry later.", e);
                 this._sendOutgoingRoomKeyRequestsTimer = null;
                 this._startTimer();
             }).done();
@@ -321,7 +337,7 @@ export default class OutgoingRoomKeyRequestManager {
 
     // given a RoomKeyRequest, send it and update the request record
     _sendOutgoingRoomKeyRequest(req) {
-        console.log(
+        logger.log(
             `Requesting keys for ${stringifyRequestBody(req.requestBody)}` +
             ` from ${stringifyRecipientList(req.recipients)}` +
             `(id ${req.requestId})`,
@@ -347,7 +363,7 @@ export default class OutgoingRoomKeyRequestManager {
     // Given a RoomKeyRequest, cancel it and delete the request record unless
     // andResend is set, in which case transition to UNSENT.
     _sendOutgoingRoomKeyRequestCancellation(req, andResend) {
-        console.log(
+        logger.log(
             `Sending cancellation for key request for ` +
             `${stringifyRequestBody(req.requestBody)} to ` +
             `${stringifyRecipientList(req.recipients)} ` +

src/crypto/RoomList.js

@@ -71,6 +71,9 @@ export default class RoomList {
     }
 
     async setRoomEncryption(roomId, roomInfo) {
+        // important that this happens before calling into the store
+        // as it prevents the Crypto::setRoomEncryption from calling
+        // this twice for consecutive m.room.encryption events
         this._roomEncryption[roomId] = roomInfo;
         await this._cryptoStore.doTxn(
             'readwrite', [IndexedDBCryptoStore.STORE_ROOMS], (txn) => {

src/crypto/algorithms/megolm.js

@@ -1,5 +1,6 @@
 /*
 Copyright 2015, 2016 OpenMarket Ltd
+Copyright 2018 New Vector Ltd
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -23,6 +24,7 @@ limitations under the License.
 
 import Promise from 'bluebird';
 
+const logger = require("../../logger");
 const utils = require("../../utils");
 const olmlib = require("../olmlib");
 const base = require("./base");
@@ -64,7 +66,7 @@ OutboundSessionInfo.prototype.needsRotation = function(
     if (this.useCount >= rotationPeriodMsgs ||
         sessionLifetime >= rotationPeriodMs
        ) {
-        console.log(
+        logger.log(
             "Rotating megolm session after " + this.useCount +
                 " messages, " + sessionLifetime + "ms",
         );
@@ -102,7 +104,7 @@ OutboundSessionInfo.prototype.sharedWithTooManyDevices = function(
         }
 
         if (!devicesInRoom.hasOwnProperty(userId)) {
-            console.log("Starting new session because we shared with " + userId);
+            logger.log("Starting new session because we shared with " + userId);
             return true;
         }
 
@@ -112,7 +114,7 @@ OutboundSessionInfo.prototype.sharedWithTooManyDevices = function(
             }
 
             if (!devicesInRoom[userId].hasOwnProperty(deviceId)) {
-                console.log(
+                logger.log(
                     "Starting new session because we shared with " +
                         userId + ":" + deviceId,
                 );
@@ -142,6 +144,11 @@ function MegolmEncryption(params) {
     // room).
     this._setupPromise = Promise.resolve();
 
+    // Map of outbound sessions by sessions ID. Used if we need a particular
+    // session (the session we're currently using to send is always obtained
+    // using _setupPromise).
+    this._outboundSessions = {};
+
     // default rotation periods
     this._sessionRotationPeriodMsgs = 100;
     this._sessionRotationPeriodMs = 7 * 24 * 3600 * 1000;
@@ -181,7 +188,7 @@ MegolmEncryption.prototype._ensureOutboundSession = function(devicesInRoom) {
         if (session && session.needsRotation(self._sessionRotationPeriodMsgs,
                                              self._sessionRotationPeriodMs)
            ) {
-            console.log("Starting new megolm session because we need to rotate.");
+            logger.log("Starting new megolm session because we need to rotate.");
             session = null;
         }
 
@@ -191,8 +198,9 @@ MegolmEncryption.prototype._ensureOutboundSession = function(devicesInRoom) {
         }
 
         if (!session) {
-            console.log(`Starting new megolm session for room ${self._roomId}`);
+            logger.log(`Starting new megolm session for room ${self._roomId}`);
             session = await self._prepareNewSession();
+            self._outboundSessions[session.sessionId] = session;
         }
 
         // now check if we need to share with any devices
@@ -262,6 +270,18 @@ MegolmEncryption.prototype._prepareNewSession = async function() {
         key.key, {ed25519: this._olmDevice.deviceEd25519Key},
     );
 
+    if (this._crypto.backupInfo) {
+        // don't wait for it to complete
+        this._crypto.backupGroupSession(
+            this._roomId, this._olmDevice.deviceCurve25519Key, [],
+            sessionId, key.key,
+        ).catch((e) => {
+            // This throws if the upload failed, but this is fine
+            // since it will have written it to the db and will retry.
+            console.log("Failed to back up group session", e);
+        });
+    }
+
     return new OutboundSessionInfo(sessionId);
 };
 
@@ -318,7 +338,7 @@ MegolmEncryption.prototype._splitUserDeviceMap = function(
                 continue;
             }
 
-            console.log(
+            logger.log(
                 "share keys with device " + userId + ":" + deviceId,
             );
 
@@ -407,8 +427,98 @@ MegolmEncryption.prototype._encryptAndSendKeysToDevices = function(
 };
 
 /**
- * @private
+ * Re-shares a megolm session key with devices if the key has already been
+ * sent to them.
  *
+ * @param {string} senderKey The key of the originating device for the session
+ * @param {string} sessionId ID of the outbound session to share
+ * @param {string} userId ID of the user who owns the target device
+ * @param {module:crypto/deviceinfo} device The target device
+ */
+MegolmEncryption.prototype.reshareKeyWithDevice = async function(
+    senderKey, sessionId, userId, device,
+) {
+    const obSessionInfo = this._outboundSessions[sessionId];
+    if (!obSessionInfo) {
+        logger.debug("Session ID " + sessionId + " not found: not re-sharing keys");
+        return;
+    }
+
+    // The chain index of the key we previously sent this device
+    if (obSessionInfo.sharedWithDevices[userId] === undefined) {
+        logger.debug("Session ID " + sessionId + " never shared with user " + userId);
+        return;
+    }
+    const sentChainIndex = obSessionInfo.sharedWithDevices[userId][device.deviceId];
+    if (sentChainIndex === undefined) {
+        logger.debug(
+            "Session ID " + sessionId + " never shared with device " +
+            userId + ":" + device.deviceId,
+        );
+        return;
+    }
+
+    // get the key from the inbound session: the outbound one will already
+    // have been ratcheted to the next chain index.
+    const key = await this._olmDevice.getInboundGroupSessionKey(
+        this._roomId, senderKey, sessionId, sentChainIndex,
+    );
+
+    if (!key) {
+        logger.warn(
+            "No outbound session key found for " + sessionId + ": not re-sharing keys",
+        );
+        return;
+    }
+
+    await olmlib.ensureOlmSessionsForDevices(
+        this._olmDevice, this._baseApis, {
+            [userId]: {
+                [device.deviceId]: device,
+            },
+        },
+    );
+
+    const payload = {
+        type: "m.forwarded_room_key",
+        content: {
+            algorithm: olmlib.MEGOLM_ALGORITHM,
+            room_id: this._roomId,
+            session_id: sessionId,
+            session_key: key.key,
+            chain_index: key.chain_index,
+            sender_key: senderKey,
+            sender_claimed_ed25519_key: key.sender_claimed_ed25519_key,
+            forwarding_curve25519_key_chain: key.forwarding_curve25519_key_chain,
+        },
+    };
+
+    const encryptedContent = {
+        algorithm: olmlib.OLM_ALGORITHM,
+        sender_key: this._olmDevice.deviceCurve25519Key,
+        ciphertext: {},
+    };
+    await olmlib.encryptMessageForDevice(
+        encryptedContent.ciphertext,
+        this._userId,
+        this._deviceId,
+        this._olmDevice,
+        userId,
+        device,
+        payload,
+    ),
+
+    await this._baseApis.sendToDevice("m.room.encrypted", {
+        [userId]: {
+            [device.deviceId]: encryptedContent,
+        },
+    });
+    logger.debug(
+        `Re-shared key for session ${sessionId}  with ${userId}:${device.deviceId}`,
+    );
+};
+
+/**
  * @param {module:crypto/algorithms/megolm.OutboundSessionInfo} session
  *
  * @param {object<string, module:crypto/deviceinfo[]>} devicesByUser
@@ -440,10 +550,10 @@ MegolmEncryption.prototype._shareKeyWithDevices = async function(session, device
             await this._encryptAndSendKeysToDevices(
                 session, key.chain_index, userDeviceMaps[i], payload,
             );
-            console.log(`Completed megolm keyshare in ${this._roomId} `
+            logger.log(`Completed megolm keyshare in ${this._roomId} `
                 + `(slice ${i + 1}/${userDeviceMaps.length})`);
         } catch (e) {
-            console.log(`megolm keyshare in ${this._roomId} `
+            logger.log(`megolm keyshare in ${this._roomId} `
                 + `(slice ${i + 1}/${userDeviceMaps.length}) failed`);
 
             throw e;
@@ -462,7 +572,7 @@ MegolmEncryption.prototype._shareKeyWithDevices = async function(session, device
  */
 MegolmEncryption.prototype.encryptMessage = function(room, eventType, content) {
     const self = this;
-    console.log(`Starting to encrypt event for ${this._roomId}`);
+    logger.log(`Starting to encrypt event for ${this._roomId}`);
 
     return this._getDevicesInRoom(room).then(function(devicesInRoom) {
         // check if any of these devices are not yet known to the user.
@@ -488,6 +598,8 @@ MegolmEncryption.prototype.encryptMessage = function(room, eventType, content) {
             session_id: session.sessionId,
              // Include our device ID so that recipients can send us a
              // m.new_device message if they don't have our session key.
+             // XXX: Do we still need this now that m.new_device messages
+             // no longer exist since #483?
             device_id: self._deviceId,
         };
 
@@ -496,6 +608,16 @@ MegolmEncryption.prototype.encryptMessage = function(room, eventType, content) {
     });
 };
 
+/**
+ * Forces the current outbound group session to be discarded such
+ * that another one will be created next time an event is sent.
+ *
+ * This should not normally be necessary.
+ */
+MegolmEncryption.prototype.forceDiscardSession = function() {
+    this._setupPromise = this._setupPromise.then(() => null);
+};
+
 /**
  * Checks the devices we're about to send to and see if any are entirely
  * unknown to the user.  If so, warn the user, and mark them as known to
@@ -535,8 +657,9 @@ MegolmEncryption.prototype._checkForUnknownDevices = function(devicesInRoom) {
  * @return {module:client.Promise} Promise which resolves to a map
  *     from userId to deviceId to deviceInfo
  */
-MegolmEncryption.prototype._getDevicesInRoom = function(room) {
-    const roomMembers = utils.map(room.getEncryptionTargetMembers(), function(u) {
+MegolmEncryption.prototype._getDevicesInRoom = async function(room) {
+    const members = await room.getEncryptionTargetMembers();
+    const roomMembers = utils.map(members, function(u) {
         return u.userId;
     });
 
@@ -549,35 +672,31 @@ MegolmEncryption.prototype._getDevicesInRoom = function(room) {
     // We are happy to use a cached version here: we assume that if we already
     // have a list of the user's devices, then we already share an e2e room
     // with them, which means that they will have announced any new devices via
-    // an m.new_device.
-    //
-    // XXX: what if the cache is stale, and the user left the room we had in
-    // common and then added new devices before joining this one? --Matthew
-    //
-    // yup, see https://github.com/vector-im/riot-web/issues/2305 --richvdh
-    return this._crypto.downloadKeys(roomMembers, false).then((devices) => {
-        // remove any blocked devices
-        for (const userId in devices) {
-            if (!devices.hasOwnProperty(userId)) {
+    // device_lists in their /sync response.  This cache should then be maintained
+    // using all the device_lists changes and left fields.
+    // See https://github.com/vector-im/riot-web/issues/2305 for details.
+    const devices = await this._crypto.downloadKeys(roomMembers, false);
+    // remove any blocked devices
+    for (const userId in devices) {
+        if (!devices.hasOwnProperty(userId)) {
+            continue;
+        }
+
+        const userDevices = devices[userId];
+        for (const deviceId in userDevices) {
+            if (!userDevices.hasOwnProperty(deviceId)) {
                 continue;
             }
 
-            const userDevices = devices[userId];
-            for (const deviceId in userDevices) {
-                if (!userDevices.hasOwnProperty(deviceId)) {
-                    continue;
-                }
-
-                if (userDevices[deviceId].isBlocked() ||
-                    (userDevices[deviceId].isUnverified() && isBlacklisting)
-                   ) {
-                    delete userDevices[deviceId];
-                }
+            if (userDevices[deviceId].isBlocked() ||
+                (userDevices[deviceId].isUnverified() && isBlacklisting)
+               ) {
+                delete userDevices[deviceId];
             }
         }
+    }
 
-        return devices;
-    });
+    return devices;
 };
 
 /**
@@ -772,12 +891,12 @@ MegolmDecryption.prototype.onRoomKeyEvent = function(event) {
         !sessionId ||
         !content.session_key
        ) {
-        console.error("key event is missing fields");
+        logger.error("key event is missing fields");
         return;
     }
 
     if (!senderKey) {
-        console.error("key event has no sender key (not encrypted?)");
+        logger.error("key event has no sender key (not encrypted?)");
         return;
     }
 
@@ -794,13 +913,13 @@ MegolmDecryption.prototype.onRoomKeyEvent = function(event) {
 
         senderKey = content.sender_key;
         if (!senderKey) {
-            console.error("forwarded_room_key event is missing sender_key field");
+            logger.error("forwarded_room_key event is missing sender_key field");
             return;
         }
 
         const ed25519Key = content.sender_claimed_ed25519_key;
         if (!ed25519Key) {
-            console.error(
+            logger.error(
                 `forwarded_room_key_event is missing sender_claimed_ed25519_key field`,
             );
             return;
@@ -813,8 +932,8 @@ MegolmDecryption.prototype.onRoomKeyEvent = function(event) {
         keysClaimed = event.getKeysClaimed();
     }
 
-    console.log(`Adding key for megolm session ${senderKey}|${sessionId}`);
-    this._olmDevice.addInboundGroupSession(
+    logger.log(`Adding key for megolm session ${senderKey}|${sessionId}`);
+    return this._olmDevice.addInboundGroupSession(
         content.room_id, senderKey, forwardingKeyChain, sessionId,
         content.session_key, keysClaimed,
         exportFormat,
@@ -829,8 +948,21 @@ MegolmDecryption.prototype.onRoomKeyEvent = function(event) {
 
         // have another go at decrypting events sent with this session.
         this._retryDecryption(senderKey, sessionId);
+    }).then(() => {
+        if (this._crypto.backupInfo) {
+            // don't wait for the keys to be backed up for the server
+            this._crypto.backupGroupSession(
+                content.room_id, senderKey, forwardingKeyChain,
+                content.session_id, content.session_key, keysClaimed,
+                exportFormat,
+            ).catch((e) => {
+                // This throws if the upload failed, but this is fine
+                // since it will have written it to the db and will retry.
+                console.log("Failed to back up group session", e);
+            });
+        }
     }).catch((e) => {
-        console.error(`Error handling m.room_key_event: ${e}`);
+        logger.error(`Error handling m.room_key_event: ${e}`);
     });
 };
 
@@ -872,7 +1004,7 @@ MegolmDecryption.prototype.shareKeysWithDevice = function(keyRequest) {
             return null;
         }
 
-        console.log(
+        logger.log(
             "sharing keys for session " + body.sender_key + "|"
             + body.session_id + " with device "
             + userId + ":" + deviceId,
@@ -946,6 +1078,22 @@ MegolmDecryption.prototype.importRoomKey = function(session) {
         session.sender_claimed_keys,
         true,
     ).then(() => {
+        if (this._crypto.backupInfo) {
+            // don't wait for it to complete
+            this._crypto.backupGroupSession(
+                session.room_id,
+                session.sender_key,
+                session.forwarding_curve25519_key_chain,
+                session.session_id,
+                session.session_key,
+                session.sender_claimed_keys,
+                true,
+            ).catch((e) => {
+                // This throws if the upload failed, but this is fine
+                // since it will have written it to the db and will retry.
+                console.log("Failed to back up group session", e);
+            });
+        }
         // have another go at decrypting events sent with this session.
         this._retryDecryption(session.sender_key, session.session_id);
     });

src/crypto/algorithms/olm.js

@@ -22,6 +22,7 @@ limitations under the License.
  */
 import Promise from 'bluebird';
 
+const logger = require("../../logger");
 const utils = require("../../utils");
 const olmlib = require("../olmlib");
 const DeviceInfo = require("../deviceinfo");
@@ -83,60 +84,62 @@ OlmEncryption.prototype._ensureSession = function(roomMembers) {
  *
  * @return {module:client.Promise} Promise which resolves to the new event body
  */
-OlmEncryption.prototype.encryptMessage = function(room, eventType, content) {
+OlmEncryption.prototype.encryptMessage = async function(room, eventType, content) {
     // pick the list of recipients based on the membership list.
     //
     // TODO: there is a race condition here! What if a new user turns up
     // just as you are sending a secret message?
 
-    const users = utils.map(room.getEncryptionTargetMembers(), function(u) {
+    const members = await room.getEncryptionTargetMembers();
+
+    const users = utils.map(members, function(u) {
         return u.userId;
     });
 
     const self = this;
-    return this._ensureSession(users).then(function() {
-        const payloadFields = {
-            room_id: room.roomId,
-            type: eventType,
-            content: content,
-        };
+    await this._ensureSession(users);
 
-        const encryptedContent = {
-            algorithm: olmlib.OLM_ALGORITHM,
-            sender_key: self._olmDevice.deviceCurve25519Key,
-            ciphertext: {},
-        };
+    const payloadFields = {
+        room_id: room.roomId,
+        type: eventType,
+        content: content,
+    };
 
-        const promises = [];
+    const encryptedContent = {
+        algorithm: olmlib.OLM_ALGORITHM,
+        sender_key: self._olmDevice.deviceCurve25519Key,
+        ciphertext: {},
+    };
 
-        for (let i = 0; i < users.length; ++i) {
-            const userId = users[i];
-            const devices = self._crypto.getStoredDevicesForUser(userId);
+    const promises = [];
 
-            for (let j = 0; j < devices.length; ++j) {
-                const deviceInfo = devices[j];
-                const key = deviceInfo.getIdentityKey();
-                if (key == self._olmDevice.deviceCurve25519Key) {
-                    // don't bother sending to ourself
-                    continue;
-                }
-                if (deviceInfo.verified == DeviceVerification.BLOCKED) {
-                    // don't bother setting up sessions with blocked users
-                    continue;
-                }
+    for (let i = 0; i < users.length; ++i) {
+        const userId = users[i];
+        const devices = self._crypto.getStoredDevicesForUser(userId);
 
-                promises.push(
-                    olmlib.encryptMessageForDevice(
-                        encryptedContent.ciphertext,
-                        self._userId, self._deviceId, self._olmDevice,
-                        userId, deviceInfo, payloadFields,
-                    ),
-                );
+        for (let j = 0; j < devices.length; ++j) {
+            const deviceInfo = devices[j];
+            const key = deviceInfo.getIdentityKey();
+            if (key == self._olmDevice.deviceCurve25519Key) {
+                // don't bother sending to ourself
+                continue;
+            }
+            if (deviceInfo.verified == DeviceVerification.BLOCKED) {
+                // don't bother setting up sessions with blocked users
+                continue;
             }
-        }
 
-        return Promise.all(promises).return(encryptedContent);
-    });
+            promises.push(
+                olmlib.encryptMessageForDevice(
+                    encryptedContent.ciphertext,
+                    self._userId, self._deviceId, self._olmDevice,
+                    userId, deviceInfo, payloadFields,
+                ),
+            );
+        }
+    }
+
+    return await Promise.all(promises).return(encryptedContent);
 };
 
 /**
@@ -271,7 +274,7 @@ OlmDecryption.prototype._decryptMessage = async function(
             const payload = await this._olmDevice.decryptMessage(
                 theirDeviceIdentityKey, sessionId, message.type, message.body,
             );
-            console.log(
+            logger.log(
                 "Decrypted Olm message from " + theirDeviceIdentityKey +
                     " with session " + sessionId,
             );
@@ -326,7 +329,7 @@ OlmDecryption.prototype._decryptMessage = async function(
         );
     }
 
-    console.log(
+    logger.log(
         "created new inbound Olm session ID " +
             res.session_id + " with " + theirDeviceIdentityKey,
     );

src/crypto/backup_password.js

@@ -0,0 +1,81 @@
+/*
+Copyright 2018 New Vector Ltd
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+import { randomString } from '../randomstring';
+
+const DEFAULT_ITERATIONS = 500000;
+
+export async function keyForExistingBackup(backupData, password) {
+    if (!global.Olm) {
+        throw new Error("Olm is not available");
+    }
+
+    const authData = backupData.auth_data;
+
+    if (!authData.private_key_salt || !authData.private_key_iterations) {
+        throw new Error(
+            "Salt and/or iterations not found: " +
+            "this backup cannot be restored with a passphrase",
+        );
+    }
+
+    return await deriveKey(
+        password, backupData.auth_data.private_key_salt,
+        backupData.auth_data.private_key_iterations,
+    );
+}
+
+export async function keyForNewBackup(password) {
+    if (!global.Olm) {
+        throw new Error("Olm is not available");
+    }
+
+    const salt = randomString(32);
+
+    const key = await deriveKey(password, salt, DEFAULT_ITERATIONS);
+
+    return { key, salt, iterations: DEFAULT_ITERATIONS };
+}
+
+async function deriveKey(password, salt, iterations) {
+    const subtleCrypto = global.crypto.subtle;
+    const TextEncoder = global.TextEncoder;
+    if (!subtleCrypto || !TextEncoder) {
+        // TODO: Implement this for node
+        throw new Error("Password-based backup is not avaiable on this platform");
+    }
+
+    const key = await subtleCrypto.importKey(
+        'raw',
+        new TextEncoder().encode(password),
+        {name: 'PBKDF2'},
+        false,
+        ['deriveBits'],
+    );
+
+    const keybits = await subtleCrypto.deriveBits(
+        {
+            name: 'PBKDF2',
+            salt: new TextEncoder().encode(salt),
+            iterations: iterations,
+            hash: 'SHA-512',
+        },
+        key,
+        global.Olm.PRIVATE_KEY_LENGTH * 8,
+    );
+
+    return new Uint8Array(keybits);
+}

src/crypto/index.js

@@ -25,6 +25,7 @@ const anotherjson = require('another-json');
 import Promise from 'bluebird';
 import {EventEmitter} from 'events';
 
+const logger = require("../logger");
 const utils = require("../utils");
 const OlmDevice = require("./OlmDevice");
 const olmlib = require("./olmlib");
@@ -36,6 +37,12 @@ const DeviceList = require('./DeviceList').default;
 import OutgoingRoomKeyRequestManager from './OutgoingRoomKeyRequestManager';
 import IndexedDBCryptoStore from './store/indexeddb-crypto-store';
 
+export function isCryptoAvailable() {
+    return Boolean(global.Olm);
+}
+
+const MIN_FORCE_SESSION_INTERVAL_MS = 60 * 60 * 1000;
+
 /**
  * Cryptography bits
  *
@@ -62,7 +69,7 @@ import IndexedDBCryptoStore from './store/indexeddb-crypto-store';
  *
  * @param {RoomList} roomList An initialised RoomList object
  */
-function Crypto(baseApis, sessionStore, userId, deviceId,
+export default function Crypto(baseApis, sessionStore, userId, deviceId,
                 clientStore, cryptoStore, roomList) {
     this._baseApis = baseApis;
     this._sessionStore = sessionStore;
@@ -72,6 +79,14 @@ function Crypto(baseApis, sessionStore, userId, deviceId,
     this._cryptoStore = cryptoStore;
     this._roomList = roomList;
 
+    // track whether this device's megolm keys are being backed up incrementally
+    // to the server or not.
+    // XXX: this should probably have a single source of truth from OlmAccount
+    this.backupInfo = null; // The info dict from /room_keys/version
+    this.backupKey = null; // The encryption key object
+    this._checkedForBackup = false; // Have we checked the server for a backup we can use?
+    this._sendingBackups = false; // Are we currently sending backups?
+
     this._olmDevice = new OlmDevice(sessionStore, cryptoStore);
     this._deviceList = new DeviceList(
         baseApis, cryptoStore, sessionStore, this._olmDevice,
@@ -106,6 +121,24 @@ function Crypto(baseApis, sessionStore, userId, deviceId,
     this._receivedRoomKeyRequestCancellations = [];
     // true if we are currently processing received room key requests
     this._processingRoomKeyRequests = false;
+    // controls whether device tracking is delayed
+    // until calling encryptEvent or trackRoomDevices,
+    // or done immediately upon enabling room encryption.
+    this._lazyLoadMembers = false;
+    // in case _lazyLoadMembers is true,
+    // track if an initial tracking of all the room members
+    // has happened for a given room. This is delayed
+    // to avoid loading room members as long as possible.
+    this._roomDeviceTrackingState = {};
+
+    // The timestamp of the last time we forced establishment
+    // of a new session for each device, in milliseconds.
+    // {
+    //     userId: {
+    //         deviceId: 1234567890000,
+    //     },
+    // }
+    this._lastNewSessionForced = {};
 }
 utils.inherits(Crypto, EventEmitter);
 
@@ -115,6 +148,8 @@ utils.inherits(Crypto, EventEmitter);
  * Returns a promise which resolves once the crypto module is ready for use.
  */
 Crypto.prototype.init = async function() {
+    await global.Olm.init();
+
     const sessionStoreHasAccount = Boolean(this._sessionStore.getEndToEndAccount());
     let cryptoStoreHasAccount;
     await this._cryptoStore.doTxn(
@@ -165,6 +200,121 @@ Crypto.prototype.init = async function() {
         );
         this._deviceList.saveIfDirty();
     }
+
+    this._checkAndStartKeyBackup();
+};
+
+/**
+ * Check the server for an active key backup and
+ * if one is present and has a valid signature from
+ * one of the user's verified devices, start backing up
+ * to it.
+ */
+Crypto.prototype._checkAndStartKeyBackup = async function() {
+    console.log("Checking key backup status...");
+    let backupInfo;
+    try {
+        backupInfo = await this._baseApis.getKeyBackupVersion();
+    } catch (e) {
+        console.log("Error checking for active key backup", e);
+        if (e.httpStatus / 100 === 4) {
+            // well that's told us. we won't try again.
+            this._checkedForBackup = true;
+        }
+        return;
+    }
+    this._checkedForBackup = true;
+
+    const trustInfo = await this.isKeyBackupTrusted(backupInfo);
+
+    if (trustInfo.usable && !this.backupInfo) {
+        console.log("Found usable key backup: enabling key backups");
+        this._baseApis.enableKeyBackup(backupInfo);
+    } else if (!trustInfo.usable && this.backupInfo) {
+        console.log("No usable key backup: disabling key backup");
+        this._baseApis.disableKeyBackup();
+    } else if (!trustInfo.usable && !this.backupInfo) {
+        console.log("No usable key backup: not enabling key backup");
+    }
+};
+
+/**
+ * Forces a re-check of the key backup and enables/disables it
+ * as appropriate
+ *
+ * @param {object} backupInfo Backup info from /room_keys/version endpoint
+ */
+Crypto.prototype.checkKeyBackup = async function(backupInfo) {
+    this._checkedForBackup = false;
+    await this._checkAndStartKeyBackup();
+};
+
+/**
+ * @param {object} backupInfo key backup info dict from /room_keys/version
+ * @return {object} {
+ *     usable: [bool], // is the backup trusted, true iff there is a sig that is valid & from a trusted device
+ *     sigs: [
+ *         valid: [bool],
+ *         device: [DeviceInfo],
+ *     ]
+ * }
+ */
+Crypto.prototype.isKeyBackupTrusted = async function(backupInfo) {
+    const ret = {
+        usable: false,
+        sigs: [],
+    };
+
+    if (
+        !backupInfo ||
+        !backupInfo.algorithm ||
+        !backupInfo.auth_data ||
+        !backupInfo.auth_data.public_key ||
+        !backupInfo.auth_data.signatures
+    ) {
+        console.log("Key backup is absent or missing required data");
+        return ret;
+    }
+
+    const mySigs = backupInfo.auth_data.signatures[this._userId];
+    if (!mySigs || mySigs.length === 0) {
+        console.log("Ignoring key backup because it lacks any signatures from this user");
+        return ret;
+    }
+
+    for (const keyId of Object.keys(mySigs)) {
+        const device = this._deviceList.getStoredDevice(
+            this._userId, keyId.split(':')[1], // XXX: is this how we're supposed to get the device ID?
+        );
+        if (!device) {
+            console.log("Ignoring signature from unknown key " + keyId);
+            continue;
+        }
+        const sigInfo = { device };
+        try {
+            await olmlib.verifySignature(
+                this._olmDevice,
+                backupInfo.auth_data,
+                this._userId,
+                device.deviceId,
+                device.getFingerprint(),
+            );
+            sigInfo.valid = true;
+        } catch (e) {
+            console.log("Bad signature from device " + device.deviceId, e);
+            sigInfo.valid = false;
+        }
+        ret.sigs.push(sigInfo);
+    }
+
+    ret.usable = ret.sigs.some((s) => s.valid && s.device.isVerified());
+    return ret;
+};
+
+/**
+ */
+Crypto.prototype.enableLazyLoading = function() {
+    this._lazyLoadMembers = true;
 };
 
 /**
@@ -181,7 +331,7 @@ Crypto.prototype.registerEventHandlers = function(eventEmitter) {
         try {
             crypto._onRoomMembership(event, member, oldMembership);
         } catch (e) {
-             console.error("Error handling membership change:", e);
+             logger.error("Error handling membership change:", e);
         }
     });
 
@@ -199,6 +349,7 @@ Crypto.prototype.start = function() {
 /** Stop background processes related to crypto */
 Crypto.prototype.stop = function() {
     this._outgoingRoomKeyRequestManager.stop();
+    this._deviceList.stop();
 };
 
 /**
@@ -366,7 +517,7 @@ function _maybeUploadOneTimeKeys(crypto) {
         // create any more keys.
         return uploadLoop(keyCount);
     }).catch((e) => {
-        console.error("Error uploading one-time keys", e.stack || e);
+        logger.error("Error uploading one-time keys", e.stack || e);
     }).finally(() => {
         // reset _oneTimeKeyCount to prevent start uploading based on old data.
         // it will be set again on the next /sync-response
@@ -573,7 +724,7 @@ Crypto.prototype.getEventSenderDeviceInfo = function(event) {
     // identity key of the device which set up the Megolm session.
 
     const device = this._deviceList.getDeviceByIdentityKey(
-        event.getSender(), algorithm, senderKey,
+        algorithm, senderKey,
     );
 
     if (device === null) {
@@ -591,13 +742,13 @@ Crypto.prototype.getEventSenderDeviceInfo = function(event) {
 
     const claimedKey = event.getClaimedEd25519Key();
     if (!claimedKey) {
-        console.warn("Event " + event.getId() + " claims no ed25519 key: " +
+        logger.warn("Event " + event.getId() + " claims no ed25519 key: " +
                      "cannot verify sending device");
         return null;
     }
 
     if (claimedKey !== device.getFingerprint()) {
-        console.warn(
+        logger.warn(
             "Event " + event.getId() + " claims ed25519 key " + claimedKey +
                 "but sender device has key " + device.getFingerprint());
         return null;
@@ -606,6 +757,23 @@ Crypto.prototype.getEventSenderDeviceInfo = function(event) {
     return device;
 };
 
+/**
+ * Forces the current outbound group session to be discarded such
+ * that another one will be created next time an event is sent.
+ *
+ * @param {string} roomId The ID of the room to discard the session for
+ *
+ * This should not normally be necessary.
+ */
+Crypto.prototype.forceDiscardSession = function(roomId) {
+    const alg = this._roomEncryptors[roomId];
+    if (alg === undefined) throw new Error("Room not encrypted");
+    if (alg.forceDiscardSession === undefined) {
+        throw new Error("Room encryption algorithm doesn't support session discarding");
+    }
+    alg.forceDiscardSession();
+};
+
 /**
  * Configure a room to use encryption (ie, save a flag in the sessionstore).
  *
@@ -614,25 +782,49 @@ Crypto.prototype.getEventSenderDeviceInfo = function(event) {
  * @param {object} config The encryption config for the room.
  *
  * @param {boolean=} inhibitDeviceQuery true to suppress device list query for
- *   users in the room (for now)
+ *   users in the room (for now). In case lazy loading is enabled,
+ *   the device query is always inhibited as the members are not tracked.
  */
 Crypto.prototype.setRoomEncryption = async function(roomId, config, inhibitDeviceQuery) {
-    // if we already have encryption in this room, we should ignore this event
-    // (for now at least. maybe we should alert the user somehow?)
+    // if state is being replayed from storage, we might already have a configuration
+    // for this room as they are persisted as well.
+    // We just need to make sure the algorithm is initialized in this case.
+    // However, if the new config is different,
+    // we should bail out as room encryption can't be changed once set.
     const existingConfig = this._roomList.getRoomEncryption(roomId);
-    if (existingConfig && JSON.stringify(existingConfig) != JSON.stringify(config)) {
-        console.error("Ignoring m.room.encryption event which requests " +
-                      "a change of config in " + roomId);
+    if (existingConfig) {
+        if (JSON.stringify(existingConfig) != JSON.stringify(config)) {
+            logger.error("Ignoring m.room.encryption event which requests " +
+                          "a change of config in " + roomId);
+            return;
+        }
+    }
+    // if we already have encryption in this room, we should ignore this event,
+    // as it would reset the encryption algorithm.
+    // This is at least expected to be called twice, as sync calls onCryptoEvent
+    // for both the timeline and state sections in the /sync response,
+    // the encryption event would appear in both.
+    // If it's called more than twice though,
+    // it signals a bug on client or server.
+    const existingAlg = this._roomEncryptors[roomId];
+    if (existingAlg) {
         return;
     }
 
+    // _roomList.getRoomEncryption will not race with _roomList.setRoomEncryption
+    // because it first stores in memory. We should await the promise only
+    // after all the in-memory state (_roomEncryptors and _roomList) has been updated
+    // to avoid races when calling this method multiple times. Hence keep a hold of the promise.
+    let storeConfigPromise = null;
+    if(!existingConfig) {
+        storeConfigPromise = this._roomList.setRoomEncryption(roomId, config);
+    }
+
     const AlgClass = algorithms.ENCRYPTION_CLASSES[config.algorithm];
     if (!AlgClass) {
         throw new Error("Unable to encrypt with " + config.algorithm);
     }
 
-    await this._roomList.setRoomEncryption(roomId, config);
-
     const alg = new AlgClass({
         userId: this._userId,
         deviceId: this._deviceId,
@@ -644,24 +836,59 @@ Crypto.prototype.setRoomEncryption = async function(roomId, config, inhibitDevic
     });
     this._roomEncryptors[roomId] = alg;
 
-    // make sure we are tracking the device lists for all users in this room.
-    console.log("Enabling encryption in " + roomId + "; " +
-                "starting to track device lists for all users therein");
-    const room = this._clientStore.getRoom(roomId);
-    if (!room) {
-        throw new Error(`Unable to enable encryption in unknown room ${roomId}`);
+    if (storeConfigPromise) {
+        await storeConfigPromise;
     }
 
-    const members = room.getEncryptionTargetMembers();
-    members.forEach((m) => {
-        this._deviceList.startTrackingDeviceList(m.userId);
-    });
-    if (!inhibitDeviceQuery) {
-        this._deviceList.refreshOutdatedDeviceLists();
+    if (!this._lazyLoadMembers) {
+        logger.log("Enabling encryption in " + roomId + "; " +
+            "starting to track device lists for all users therein");
+
+        await this.trackRoomDevices(roomId);
+        // TODO: this flag is only not used from MatrixClient::setRoomEncryption
+        // which is never used (inside riot at least)
+        // but didn't want to remove it as it technically would
+        // be a breaking change.
+        if(!this.inhibitDeviceQuery) {
+            this._deviceList.refreshOutdatedDeviceLists();
+        }
+    } else {
+        logger.log("Enabling encryption in " + roomId);
     }
 };
 
 
+/**
+ * Make sure we are tracking the device lists for all users in this room.
+ *
+ * @param {string} roomId The room ID to start tracking devices in.
+ * @returns {Promise} when all devices for the room have been fetched and marked to track
+ */
+Crypto.prototype.trackRoomDevices = function(roomId) {
+    const trackMembers = async () => {
+        // not an encrypted room
+        if (!this._roomEncryptors[roomId]) {
+            return;
+        }
+        const room = this._clientStore.getRoom(roomId);
+        if (!room) {
+            throw new Error(`Unable to start tracking devices in unknown room ${roomId}`);
+        }
+        logger.log(`Starting to track devices for room ${roomId} ...`);
+        const members = await room.getEncryptionTargetMembers();
+        members.forEach((m) => {
+            this._deviceList.startTrackingDeviceList(m.userId);
+        });
+    };
+
+    let promise = this._roomDeviceTrackingState[roomId];
+    if (!promise) {
+        promise = trackMembers();
+        this._roomDeviceTrackingState[roomId] = promise;
+    }
+    return promise;
+};
+
 /**
  * @typedef {Object} module:crypto~OlmSessionResult
  * @property {module:crypto/deviceinfo} device  device info
@@ -724,6 +951,7 @@ Crypto.prototype.exportRoomKeys = async function() {
                 const sess = this._olmDevice.exportInboundGroupSession(
                     s.senderKey, s.sessionId, s.sessionData,
                 );
+                delete sess.first_known_index;
                 sess.algorithm = olmlib.MEGOLM_ALGORITHM;
                 exportedSessions.push(sess);
             });
@@ -743,7 +971,7 @@ Crypto.prototype.importRoomKeys = function(keys) {
     return Promise.map(
         keys, (key) => {
             if (!key.room_id || !key.algorithm) {
-                console.warn("ignoring room key entry with missing fields", key);
+                logger.warn("ignoring room key entry with missing fields", key);
                 return null;
             }
 
@@ -753,6 +981,133 @@ Crypto.prototype.importRoomKeys = function(keys) {
     );
 };
 
+Crypto.prototype._maybeSendKeyBackup = async function(delay, retry) {
+    if (retry === undefined) retry = true;
+
+    if (!this._sendingBackups) {
+        this._sendingBackups = true;
+        try {
+            if (delay === undefined) {
+                // by default, wait between 0 and 10 seconds, to avoid backup
+                // requests from different clients hitting the server all at
+                // the same time when a new key is sent
+                delay = Math.random() * 10000;
+            }
+            await Promise.delay(delay);
+            let numFailures = 0; // number of consecutive failures
+            while (1) {
+                if (!this.backupKey) {
+                    return;
+                }
+                // FIXME: figure out what limit is reasonable
+                const sessions = await this._cryptoStore.getSessionsNeedingBackup(10);
+                if (!sessions.length) {
+                    return;
+                }
+                const data = {};
+                for (const session of sessions) {
+                    const roomId = session.sessionData.room_id;
+                    if (data[roomId] === undefined) {
+                        data[roomId] = {sessions: {}};
+                    }
+
+                    const sessionData = await this._olmDevice.exportInboundGroupSession(
+                        session.senderKey, session.sessionId, session.sessionData,
+                    );
+                    sessionData.algorithm = olmlib.MEGOLM_ALGORITHM;
+                    delete sessionData.session_id;
+                    delete sessionData.room_id;
+                    const firstKnownIndex = sessionData.first_known_index;
+                    delete sessionData.first_known_index;
+                    const encrypted = this.backupKey.encrypt(JSON.stringify(sessionData));
+
+                    const forwardedCount =
+                          (sessionData.forwarding_curve25519_key_chain || []).length;
+
+                    const device = this._deviceList.getDeviceByIdentityKey(
+                        olmlib.MEGOLM_ALGORITHM, session.senderKey,
+                    );
+
+                    data[roomId]['sessions'][session.sessionId] = {
+                        first_message_index: firstKnownIndex,
+                        forwarded_count: forwardedCount,
+                        is_verified: !!(device && device.isVerified()),
+                        session_data: encrypted,
+                    };
+                }
+
+                try {
+                    await this._baseApis.sendKeyBackup(
+                        undefined, undefined, this.backupInfo.version,
+                        {rooms: data},
+                    );
+                    numFailures = 0;
+                    await this._cryptoStore.unmarkSessionsNeedingBackup(sessions);
+                } catch (err) {
+                    numFailures++;
+                    console.log("send failed", err);
+                    if (err.httpStatus === 400
+                        || err.httpStatus === 403
+                        || err.httpStatus === 401
+                        || !retry) {
+                        // retrying probably won't help much, so we should give up
+                        // FIXME: disable backups?
+                        throw err;
+                    }
+                }
+                if (numFailures) {
+                    // exponential backoff if we have failures
+                    await new Promise((resolve, reject) => {
+                        setTimeout(
+                            resolve,
+                            1000 * Math.pow(2, Math.min(numFailures - 1, 4)),
+                        );
+                    });
+                }
+            }
+        } finally {
+            this._sendingBackups = false;
+        }
+    }
+};
+
+Crypto.prototype.backupGroupSession = async function(
+    roomId, senderKey, forwardingCurve25519KeyChain,
+    sessionId, sessionKey, keysClaimed,
+    exportFormat,
+) {
+    if (!this.backupInfo) {
+        throw new Error("Key backups are not enabled");
+    }
+
+    await this._cryptoStore.markSessionsNeedingBackup([{
+        senderKey: senderKey,
+        sessionId: sessionId,
+    }]);
+
+    await this._maybeSendKeyBackup();
+};
+
+Crypto.prototype.backupAllGroupSessions = async function(version) {
+    await this._cryptoStore.doTxn(
+        'readwrite',
+        [
+            IndexedDBCryptoStore.STORE_INBOUND_GROUP_SESSIONS,
+            IndexedDBCryptoStore.STORE_BACKUP,
+        ],
+        (txn) => {
+            this._cryptoStore.getAllEndToEndInboundGroupSessions(txn, (session) => {
+                if (session !== null) {
+                    this._cryptoStore.markSessionsNeedingBackup([session], txn);
+                }
+            });
+        },
+    );
+
+    await this._maybeSendKeyBackup(0, false);
+};
+
+/* eslint-disable valid-jsdoc */    //https://github.com/eslint/eslint/issues/7307
 /**
  * Encrypt an event according to the configuration of the room.
  *
@@ -763,7 +1118,8 @@ Crypto.prototype.importRoomKeys = function(keys) {
  * @return {module:client.Promise?} Promise which resolves when the event has been
  *     encrypted, or null if nothing was needed
  */
-Crypto.prototype.encryptEvent = function(event, room) {
+/* eslint-enable valid-jsdoc */
+Crypto.prototype.encryptEvent = async function(event, room) {
     if (!room) {
         throw new Error("Cannot send encrypted messages in unknown rooms");
     }
@@ -781,6 +1137,12 @@ Crypto.prototype.encryptEvent = function(event, room) {
         );
     }
 
+    if (!this._roomDeviceTrackingState[roomId]) {
+        this.trackRoomDevices(roomId);
+    }
+    // wait for all the room devices to be loaded
+    await this._roomDeviceTrackingState[roomId];
+
     let content = event.getContent();
     // If event has an m.relates_to then we need
     // to put this on the wrapping event instead
@@ -791,20 +1153,19 @@ Crypto.prototype.encryptEvent = function(event, room) {
         delete content['m.relates_to'];
     }
 
-    return alg.encryptMessage(
-        room, event.getType(), content,
-    ).then((encryptedContent) => {
-        if (mRelatesTo) {
-            encryptedContent['m.relates_to'] = mRelatesTo;
-        }
+    const encryptedContent = await alg.encryptMessage(
+        room, event.getType(), content);
 
-        event.makeEncrypted(
-            "m.room.encrypted",
-            encryptedContent,
-            this._olmDevice.deviceCurve25519Key,
-            this._olmDevice.deviceEd25519Key,
-        );
-    });
+    if (mRelatesTo) {
+        encryptedContent['m.relates_to'] = mRelatesTo;
+    }
+
+    event.makeEncrypted(
+        "m.room.encrypted",
+        encryptedContent,
+        this._olmDevice.deviceCurve25519Key,
+        this._olmDevice.deviceEd25519Key,
+    );
 };
 
 /**
@@ -852,7 +1213,7 @@ Crypto.prototype.handleDeviceListChanges = async function(syncData, syncDeviceLi
     // If we didn't make this assumption, we'd have to use the /keys/changes API
     // to get key changes between the sync token in the device list and the 'old'
     // sync token used here to make sure we didn't miss any.
-    this._evalDeviceListChanges(syncDeviceLists);
+    await this._evalDeviceListChanges(syncDeviceLists);
 };
 
 /**
@@ -866,7 +1227,7 @@ Crypto.prototype.requestRoomKey = function(requestBody, recipients) {
         requestBody, recipients,
     ).catch((e) => {
         // this normally means we couldn't talk to the store
-        console.error(
+        logger.error(
             'Error requesting key for event', e,
         );
     }).done();
@@ -883,7 +1244,7 @@ Crypto.prototype.requestRoomKey = function(requestBody, recipients) {
 Crypto.prototype.cancelRoomKeyRequest = function(requestBody, andResend) {
     this._outgoingRoomKeyRequestManager.cancelRoomKeyRequest(requestBody, andResend)
     .catch((e) => {
-        console.warn("Error clearing pending room key requests", e);
+        logger.warn("Error clearing pending room key requests", e);
     }).done();
 };
 
@@ -901,7 +1262,7 @@ Crypto.prototype.onCryptoEvent = async function(event) {
         // finished processing the sync, in onSyncCompleted.
         await this.setRoomEncryption(roomId, content, true);
     } catch (e) {
-        console.error("Error configuring encryption in room " + roomId +
+        logger.error("Error configuring encryption in room " + roomId +
                       ":", e);
     }
 };
@@ -917,8 +1278,9 @@ Crypto.prototype.onSyncWillProcess = async function(syncData) {
         // scratch, so mark everything as untracked. onCryptoEvent will
         // be called for all e2e rooms during the processing of the sync,
         // at which point we'll start tracking all the users of that room.
-        console.log("Initial sync performed - resetting device tracking state");
+        logger.log("Initial sync performed - resetting device tracking state");
         this._deviceList.stopTrackingAllDeviceLists();
+        this._roomDeviceTrackingState = {};
     }
 };
 
@@ -964,11 +1326,12 @@ Crypto.prototype._evalDeviceListChanges = async function(deviceLists) {
         });
     }
 
-    if (deviceLists.left && Array.isArray(deviceLists.left)) {
+    if (deviceLists.left && Array.isArray(deviceLists.left) &&
+        deviceLists.left.length) {
         // Check we really don't share any rooms with these users
         // any more: the server isn't required to give us the
         // exact correct set.
-        const e2eUserIds = new Set(this._getE2eUsers());
+        const e2eUserIds = new Set(await this._getTrackedE2eUsers());
 
         deviceLists.left.forEach((u) => {
             if (!e2eUserIds.has(u)) {
@@ -980,13 +1343,14 @@ Crypto.prototype._evalDeviceListChanges = async function(deviceLists) {
 
 /**
  * Get a list of all the IDs of users we share an e2e room with
+ * for which we are tracking devices already
  *
  * @returns {string[]} List of user IDs
  */
-Crypto.prototype._getE2eUsers = function() {
+Crypto.prototype._getTrackedE2eUsers = async function() {
     const e2eUserIds = [];
-    for (const room of this._getE2eRooms()) {
-        const members = room.getEncryptionTargetMembers();
+    for (const room of this._getTrackedE2eRooms()) {
+        const members = await room.getEncryptionTargetMembers();
         for (const member of members) {
             e2eUserIds.push(member.userId);
         }
@@ -995,27 +1359,25 @@ Crypto.prototype._getE2eUsers = function() {
 };
 
 /**
- * Get a list of the e2e-enabled rooms we are members of
+ * Get a list of the e2e-enabled rooms we are members of,
+ * and for which we are already tracking the devices
  *
  * @returns {module:models.Room[]}
  */
-Crypto.prototype._getE2eRooms = function() {
+Crypto.prototype._getTrackedE2eRooms = function() {
     return this._clientStore.getRooms().filter((room) => {
         // check for rooms with encryption enabled
         const alg = this._roomEncryptors[room.roomId];
         if (!alg) {
             return false;
         }
-
-        // ignore any rooms which we have left
-        const me = room.getMember(this._userId);
-        if (!me || (
-            me.membership !== "join" && me.membership !== "invite"
-        )) {
+        if (!this._roomDeviceTrackingState[room.roomId]) {
             return false;
         }
 
-        return true;
+        // ignore any rooms which we have left
+        const myMembership = room.getMyMembership();
+        return myMembership === "join" || myMembership === "invite";
     });
 };
 
@@ -1027,6 +1389,8 @@ Crypto.prototype._onToDeviceEvent = function(event) {
             this._onRoomKeyEvent(event);
         } else if (event.getType() == "m.room_key_request") {
             this._onRoomKeyRequestEvent(event);
+        } else if (event.getContent().msgtype === "m.bad.encrypted") {
+            this._onToDeviceBadEncrypted(event);
         } else if (event.isBeingDecrypted()) {
             // once the event has been decrypted, try again
             event.once('Event.decrypted', (ev) => {
@@ -1034,7 +1398,7 @@ Crypto.prototype._onToDeviceEvent = function(event) {
             });
         }
     } catch (e) {
-        console.error("Error handling toDeviceEvent:", e);
+        logger.error("Error handling toDeviceEvent:", e);
     }
 };
 
@@ -1048,14 +1412,109 @@ Crypto.prototype._onRoomKeyEvent = function(event) {
     const content = event.getContent();
 
     if (!content.room_id || !content.algorithm) {
-        console.error("key event is missing fields");
+        logger.error("key event is missing fields");
         return;
     }
 
+    if (!this._checkedForBackup) {
+        // don't bother awaiting on this - the important thing is that we retry if we
+        // haven't managed to check before
+        this._checkAndStartKeyBackup();
+    }
+
     const alg = this._getRoomDecryptor(content.room_id, content.algorithm);
     alg.onRoomKeyEvent(event);
 };
 
+/**
+ * Handle a toDevice event that couldn't be decrypted
+ *
+ * @private
+ * @param {module:models/event.MatrixEvent} event undecryptable event
+ */
+Crypto.prototype._onToDeviceBadEncrypted = async function(event) {
+    const content = event.getWireContent();
+    const sender = event.getSender();
+    const algorithm = content.algorithm;
+    const deviceKey = content.sender_key;
+
+    if (sender === undefined || deviceKey === undefined || deviceKey === undefined) {
+        return;
+    }
+
+    // check when we last forced a new session with this device: if we've already done so
+    // recently, don't do it again.
+    this._lastNewSessionForced[sender] = this._lastNewSessionForced[sender] || {};
+    const lastNewSessionForced = this._lastNewSessionForced[sender][deviceKey] || 0;
+    if (lastNewSessionForced + MIN_FORCE_SESSION_INTERVAL_MS > Date.now()) {
+        logger.debug(
+            "New session already forced with device " + sender + ":" + deviceKey +
+            " at " + lastNewSessionForced + ": not forcing another",
+        );
+        return;
+    }
+
+    // establish a new olm session with this device since we're failing to decrypt messages
+    // on a current session.
+    // Note that an undecryptable message from another device could easily be spoofed -
+    // is there anything we can do to mitigate this?
+    const device = this._deviceList.getDeviceByIdentityKey(algorithm, deviceKey);
+    if (!device) {
+        logger.info(
+            "Couldn't find device for identity key " + deviceKey +
+            ": not re-establishing session",
+        );
+        return;
+    }
+    const devicesByUser = {};
+    devicesByUser[sender] = [device];
+    await olmlib.ensureOlmSessionsForDevices(
+        this._olmDevice, this._baseApis, devicesByUser, true,
+    );
+
+    this._lastNewSessionForced[sender][deviceKey] = Date.now();
+
+    // Now send a blank message on that session so the other side knows about it.
+    // (The keyshare request is sent in the clear so that won't do)
+    // We send this first such that, as long as the toDevice messages arrive in the
+    // same order we sent them, the other end will get this first, set up the new session,
+    // then get the keyshare request and send the key over this new session (because it
+    // is the session it has most recently received a message on).
+    const encryptedContent = {
+        algorithm: olmlib.OLM_ALGORITHM,
+        sender_key: this._olmDevice.deviceCurve25519Key,
+        ciphertext: {},
+    };
+    await olmlib.encryptMessageForDevice(
+        encryptedContent.ciphertext,
+        this._userId,
+        this._deviceId,
+        this._olmDevice,
+        sender,
+        device,
+        {type: "m.dummy"},
+    );
+
+    await this._baseApis.sendToDevice("m.room.encrypted", {
+        [sender]: {
+            [device.deviceId]: encryptedContent,
+        },
+    });
+
+
+    // Most of the time this probably won't be necessary since we'll have queued up a key request when
+    // we failed to decrypt the message and will be waiting a bit for the key to arrive before sending
+    // it. This won't always be the case though so we need to re-send any that have already been sent
+    // to avoid races.
+    const requestsToResend =
+        await this._outgoingRoomKeyRequestManager.getOutgoingSentRoomKeyRequest(
+            sender, device.deviceId,
+        );
+    for (const keyReq of requestsToResend) {
+        this.cancelRoomKeyRequest(keyReq.requestBody, true);
+    }
+};
+
 /**
  * Handle a change in the membership state of a member of a room
  *
@@ -1080,15 +1539,20 @@ Crypto.prototype._onRoomMembership = function(event, member, oldMembership) {
         // not encrypting in this room
         return;
     }
-
-    if (member.membership == 'join') {
-        console.log('Join event for ' + member.userId + ' in ' + roomId);
-        // make sure we are tracking the deviceList for this user
-        this._deviceList.startTrackingDeviceList(member.userId);
-    } else if (member.membership == 'invite' &&
-             this._clientStore.getRoom(roomId).shouldEncryptForInvitedMembers()) {
-        console.log('Invite event for ' + member.userId + ' in ' + roomId);
-        this._deviceList.startTrackingDeviceList(member.userId);
+    // only mark users in this room as tracked if we already started tracking in this room
+    // this way we don't start device queries after sync on behalf of this room which we won't use
+    // the result of anyway, as we'll need to do a query again once all the members are fetched
+    // by calling _trackRoomDevices
+    if (this._roomDeviceTrackingState[roomId]) {
+        if (member.membership == 'join') {
+            logger.log('Join event for ' + member.userId + ' in ' + roomId);
+            // make sure we are tracking the deviceList for this user
+            this._deviceList.startTrackingDeviceList(member.userId);
+        } else if (member.membership == 'invite' &&
+                 this._clientStore.getRoom(roomId).shouldEncryptForInvitedMembers()) {
+            logger.log('Invite event for ' + member.userId + ' in ' + roomId);
+            this._deviceList.startTrackingDeviceList(member.userId);
+        }
     }
 
     alg.onRoomMembership(event, member, oldMembership);
@@ -1153,7 +1617,7 @@ Crypto.prototype._processReceivedRoomKeyRequests = async function() {
                 this._processReceivedRoomKeyRequestCancellation(cancellation),
         );
     } catch (e) {
-        console.error(`Error processing room key requsts: ${e}`);
+        logger.error(`Error processing room key requsts: ${e}`);
     } finally {
         this._processingRoomKeyRequests = false;
     }
@@ -1172,13 +1636,31 @@ Crypto.prototype._processReceivedRoomKeyRequest = async function(req) {
     const roomId = body.room_id;
     const alg = body.algorithm;
 
-    console.log(`m.room_key_request from ${userId}:${deviceId}` +
+    logger.log(`m.room_key_request from ${userId}:${deviceId}` +
                 ` for ${roomId} / ${body.session_id} (id ${req.requestId})`);
 
     if (userId !== this._userId) {
-        // TODO: determine if we sent this device the keys already: in
-        // which case we can do so again.
-        console.log("Ignoring room key request from other user for now");
+        if (!this._roomEncryptors[roomId]) {
+            logger.debug(`room key request for unencrypted room ${roomId}`);
+            return;
+        }
+        const encryptor = this._roomEncryptors[roomId];
+        const device = this._deviceList.getStoredDevice(userId, deviceId);
+        if (!device) {
+            logger.debug(`Ignoring keyshare for unknown device ${userId}:${deviceId}`);
+            return;
+        }
+
+        try {
+            await encryptor.reshareKeyWithDevice(
+                body.sender_key, body.session_id, userId, device,
+            );
+        } catch (e) {
+            logger.warn(
+                "Failed to re-share keys for session " + body.session_id +
+                " with device " + userId + ":" + device.deviceId, e,
+            );
+        }
         return;
     }
 
@@ -1188,18 +1670,18 @@ Crypto.prototype._processReceivedRoomKeyRequest = async function(req) {
     // if we don't have a decryptor for this room/alg, we don't have
     // the keys for the requested events, and can drop the requests.
     if (!this._roomDecryptors[roomId]) {
-        console.log(`room key request for unencrypted room ${roomId}`);
+        logger.log(`room key request for unencrypted room ${roomId}`);
         return;
     }
 
     const decryptor = this._roomDecryptors[roomId][alg];
     if (!decryptor) {
-        console.log(`room key request for unknown alg ${alg} in room ${roomId}`);
+        logger.log(`room key request for unknown alg ${alg} in room ${roomId}`);
         return;
     }
 
     if (!await decryptor.hasKeysForKeyRequest(req)) {
-        console.log(
+        logger.log(
             `room key request for unknown session ${roomId} / ` +
                 body.session_id,
         );
@@ -1213,7 +1695,7 @@ Crypto.prototype._processReceivedRoomKeyRequest = async function(req) {
     // if the device is is verified already, share the keys
     const device = this._deviceList.getStoredDevice(userId, deviceId);
     if (device && device.isVerified()) {
-        console.log('device is already verified: sharing keys');
+        logger.log('device is already verified: sharing keys');
         req.share();
         return;
     }
@@ -1230,7 +1712,7 @@ Crypto.prototype._processReceivedRoomKeyRequest = async function(req) {
 Crypto.prototype._processReceivedRoomKeyRequestCancellation = async function(
     cancellation,
 ) {
-    console.log(
+    logger.log(
         `m.room_key_request cancellation for ${cancellation.userId}:` +
             `${cancellation.deviceId} (id ${cancellation.requestId})`,
     );
@@ -1415,6 +1897,3 @@ class IncomingRoomKeyRequestCancellation {
  * @event module:client~MatrixClient#"crypto.warning"
  * @param {string} type One of the strings listed above
  */
-
-/** */
-module.exports = Crypto;

src/crypto/olmlib.js

@@ -23,6 +23,7 @@ limitations under the License.
 import Promise from 'bluebird';
 const anotherjson = require('another-json');
 
+const logger = require("../logger");
 const utils = require("../utils");
 
 /**
@@ -35,6 +36,11 @@ module.exports.OLM_ALGORITHM = "m.olm.v1.curve25519-aes-sha2";
  */
 module.exports.MEGOLM_ALGORITHM = "m.megolm.v1.aes-sha2";
 
+/**
+ * matrix algorithm tag for megolm backups
+ */
+module.exports.MEGOLM_BACKUP_ALGORITHM = "m.megolm_backup.v1.curve25519-aes-sha2";
+
 
 /**
  * Encrypt an event payload for an Olm device
@@ -65,7 +71,7 @@ module.exports.encryptMessageForDevice = async function(
         return;
     }
 
-    console.log(
+    logger.log(
         "Using sessionid " + sessionId + " for device " +
             recipientUserId + ":" + recipientDevice.deviceId,
     );
@@ -115,14 +121,17 @@ module.exports.encryptMessageForDevice = async function(
  * @param {module:base-apis~MatrixBaseApis} baseApis
  *
  * @param {object<string, module:crypto/deviceinfo[]>} devicesByUser
- *    map from userid to list of devices
+ *    map from userid to list of devices to ensure sessions for
+ *
+ * @param {bolean} force If true, establish a new session even if one already exists.
+ *     Optional.
  *
  * @return {module:client.Promise} resolves once the sessions are complete, to
  *    an Object mapping from userId to deviceId to
  *    {@link module:crypto~OlmSessionResult}
  */
 module.exports.ensureOlmSessionsForDevices = async function(
-    olmDevice, baseApis, devicesByUser,
+    olmDevice, baseApis, devicesByUser, force,
 ) {
     const devicesWithoutSession = [
         // [userId, deviceId], ...
@@ -140,7 +149,7 @@ module.exports.ensureOlmSessionsForDevices = async function(
             const deviceId = deviceInfo.deviceId;
             const key = deviceInfo.getIdentityKey();
             const sessionId = await olmDevice.getSessionIdForDevice(key);
-            if (sessionId === null) {
+            if (sessionId === null || force) {
                 devicesWithoutSession.push([userId, deviceId]);
             }
             result[userId][deviceId] = {
@@ -176,7 +185,7 @@ module.exports.ensureOlmSessionsForDevices = async function(
         for (let j = 0; j < devices.length; j++) {
             const deviceInfo = devices[j];
             const deviceId = deviceInfo.deviceId;
-            if (result[userId][deviceId].sessionId) {
+            if (result[userId][deviceId].sessionId && !force) {
                 // we already have a result for this device
                 continue;
             }
@@ -190,7 +199,7 @@ module.exports.ensureOlmSessionsForDevices = async function(
             }
 
             if (!oneTimeKey) {
-                console.warn(
+                logger.warn(
                     "No one-time keys (alg=" + oneTimeKeyAlgorithm +
                         ") for device " + userId + ":" + deviceId,
                 );
@@ -219,7 +228,7 @@ async function _verifyKeyAndStartSession(olmDevice, oneTimeKey, userId, deviceIn
             deviceInfo.getFingerprint(),
         );
     } catch (e) {
-        console.error(
+        logger.error(
             "Unable to verify signature on one-time key for device " +
                 userId + ":" + deviceId + ":", e,
         );
@@ -233,12 +242,12 @@ async function _verifyKeyAndStartSession(olmDevice, oneTimeKey, userId, deviceIn
         );
     } catch (e) {
         // possibly a bad key
-        console.error("Error starting session with device " +
+        logger.error("Error starting session with device " +
                       userId + ":" + deviceId + ": " + e);
         return null;
     }
 
-    console.log("Started new sessionid " + sid +
+    logger.log("Started new sessionid " + sid +
                 " for device " + userId + ":" + deviceId);
     return sid;
 }

src/crypto/recoverykey.js

@@ -0,0 +1,66 @@
+/*
+Copyright 2018 New Vector Ltd
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+import bs58 from 'bs58';
+
+// picked arbitrarily but to try & avoid clashing with any bitcoin ones
+// (which are also base58 encoded, but bitcoin's involve a lot more hashing)
+const OLM_RECOVERY_KEY_PREFIX = [0x8B, 0x01];
+
+export function encodeRecoveryKey(key) {
+    const buf = new Uint8Array(OLM_RECOVERY_KEY_PREFIX.length + key.length + 1);
+    buf.set(OLM_RECOVERY_KEY_PREFIX, 0);
+    buf.set(key, OLM_RECOVERY_KEY_PREFIX.length);
+
+    let parity = 0;
+    for (let i = 0; i < buf.length - 1; ++i) {
+        parity ^= buf[i];
+    }
+    buf[buf.length - 1] = parity;
+    const base58key = bs58.encode(buf);
+
+    return base58key.match(/.{1,4}/g).join(" ");
+}
+
+export function decodeRecoveryKey(recoverykey) {
+    const result = bs58.decode(recoverykey.replace(/ /g, ''));
+
+    let parity = 0;
+    for (const b of result) {
+        parity ^= b;
+    }
+    if (parity !== 0) {
+        throw new Error("Incorrect parity");
+    }
+
+    for (let i = 0; i < OLM_RECOVERY_KEY_PREFIX.length; ++i) {
+        if (result[i] !== OLM_RECOVERY_KEY_PREFIX[i]) {
+            throw new Error("Incorrect prefix");
+        }
+    }
+
+    if (
+        result.length !==
+        OLM_RECOVERY_KEY_PREFIX.length + global.Olm.PRIVATE_KEY_LENGTH + 1
+    ) {
+        throw new Error("Incorrect length");
+    }
+
+    return result.slice(
+        OLM_RECOVERY_KEY_PREFIX.length,
+        OLM_RECOVERY_KEY_PREFIX.length + global.Olm.PRIVATE_KEY_LENGTH,
+    );
+}

src/crypto/store/indexeddb-crypto-store-backend.js

@@ -16,9 +16,11 @@ limitations under the License.
 */
 
 import Promise from 'bluebird';
+
+import logger from '../../logger';
 import utils from '../../utils';
 
-export const VERSION = 6;
+export const VERSION = 7;
 
 /**
  * Implementation of a CryptoStore which is backed by an existing
@@ -38,7 +40,7 @@ export class Backend {
         // attempts to delete the database will block (and subsequent
         // attempts to re-create it will also block).
         db.onversionchange = (ev) => {
-            console.log(`versionchange for indexeddb ${this._dbName}: closing`);
+            logger.log(`versionchange for indexeddb ${this._dbName}: closing`);
             db.close();
         };
     }
@@ -64,7 +66,7 @@ export class Backend {
         this._getOutgoingRoomKeyRequest(txn, requestBody, (existing) => {
             if (existing) {
                 // this entry matches the request - return it.
-                console.log(
+                logger.log(
                     `already have key request outstanding for ` +
                         `${requestBody.room_id} / ${requestBody.session_id}: ` +
                         `not sending another`,
@@ -75,7 +77,7 @@ export class Backend {
 
             // we got to the end of the list without finding a match
             // - add the new request.
-            console.log(
+            logger.log(
                 `enqueueing key request for ${requestBody.room_id} / ` +
                     requestBody.session_id,
             );
@@ -204,6 +206,42 @@ export class Backend {
         return promiseifyTxn(txn).then(() => result);
     }
 
+    getOutgoingRoomKeyRequestsByTarget(userId, deviceId, wantedStates) {
+        let stateIndex = 0;
+        const results = [];
+
+        function onsuccess(ev) {
+            const cursor = ev.target.result;
+            if (cursor) {
+                const keyReq = cursor.value;
+                if (keyReq.recipients.includes({userId, deviceId})) {
+                    results.push(keyReq);
+                }
+                cursor.continue();
+            } else {
+                // try the next state in the list
+                stateIndex++;
+                if (stateIndex >= wantedStates.length) {
+                    // no matches
+                    return;
+                }
+
+                const wantedState = wantedStates[stateIndex];
+                const cursorReq = ev.target.source.openCursor(wantedState);
+                cursorReq.onsuccess = onsuccess;
+            }
+        }
+
+        const txn = this._db.transaction("outgoingRoomKeyRequests", "readonly");
+        const store = txn.objectStore("outgoingRoomKeyRequests");
+
+        const wantedState = wantedStates[stateIndex];
+        const cursorReq = store.index("state").openCursor(wantedState);
+        cursorReq.onsuccess = onsuccess;
+
+        return promiseifyTxn(txn).then(() => results);
+    }
+
     /**
      * Look for an existing room key request by id and state, and update it if
      * found
@@ -226,7 +264,7 @@ export class Backend {
             }
             const data = cursor.value;
             if (data.state != expectedState) {
-                console.warn(
+                logger.warn(
                     `Cannot update room key request from ${expectedState} ` +
                     `as it was already updated to ${data.state}`,
                 );
@@ -264,7 +302,7 @@ export class Backend {
             }
             const data = cursor.value;
             if (data.state != expectedState) {
-                console.warn(
+                logger.warn(
                     `Cannot delete room key request in state ${data.state} `
                         + `(expected ${expectedState})`,
                 );
@@ -312,7 +350,10 @@ export class Backend {
         getReq.onsuccess = function() {
             const cursor = getReq.result;
             if (cursor) {
-                results[cursor.value.sessionId] = cursor.value.session;
+                results[cursor.value.sessionId] = {
+                    session: cursor.value.session,
+                    lastReceivedMessageTs: cursor.value.lastReceivedMessageTs,
+                };
                 cursor.continue();
             } else {
                 try {
@@ -330,7 +371,10 @@ export class Backend {
         getReq.onsuccess = function() {
             try {
                 if (getReq.result) {
-                    func(getReq.result.session);
+                    func({
+                        session: getReq.result.session,
+                        lastReceivedMessageTs: getReq.result.lastReceivedMessageTs,
+                    });
                 } else {
                     func(null);
                 }
@@ -340,9 +384,14 @@ export class Backend {
         };
     }
 
-    storeEndToEndSession(deviceKey, sessionId, session, txn) {
+    storeEndToEndSession(deviceKey, sessionId, sessionInfo, txn) {
         const objectStore = txn.objectStore("sessions");
-        objectStore.put({deviceKey, sessionId, session});
+        objectStore.put({
+            deviceKey,
+            sessionId,
+            session: sessionInfo.session,
+            lastReceivedMessageTs: sessionInfo.lastReceivedMessageTs,
+        });
     }
 
     // Inbound group sessions
@@ -400,7 +449,7 @@ export class Backend {
                 ev.stopPropagation();
                 // ...and this stops it from aborting the transaction
                 ev.preventDefault();
-                console.log(
+                logger.log(
                     "Ignoring duplicate inbound group session: " +
                     senderCurve25519Key + " / " + sessionId,
                 );
@@ -460,6 +509,71 @@ export class Backend {
         };
     }
 
+    // session backups
+
+    getSessionsNeedingBackup(limit) {
+        return new Promise((resolve, reject) => {
+            const sessions = [];
+
+            const txn = this._db.transaction(
+                ["sessions_needing_backup", "inbound_group_sessions"],
+                "readonly",
+            );
+            txn.onerror = reject;
+            txn.oncomplete = function() {
+                resolve(sessions);
+            };
+            const objectStore = txn.objectStore("sessions_needing_backup");
+            const sessionStore = txn.objectStore("inbound_group_sessions");
+            const getReq = objectStore.openCursor();
+            getReq.onsuccess = function() {
+                const cursor = getReq.result;
+                if (cursor) {
+                    const sessionGetReq = sessionStore.get(cursor.key);
+                    sessionGetReq.onsuccess = function() {
+                        sessions.push({
+                            senderKey: sessionGetReq.result.senderCurve25519Key,
+                            sessionId: sessionGetReq.result.sessionId,
+                            sessionData: sessionGetReq.result.session,
+                        });
+                    };
+                    if (!limit || sessions.length < limit) {
+                        cursor.continue();
+                    }
+                }
+            };
+        });
+    }
+
+    unmarkSessionsNeedingBackup(sessions) {
+        const txn = this._db.transaction("sessions_needing_backup", "readwrite");
+        const objectStore = txn.objectStore("sessions_needing_backup");
+        return Promise.all(sessions.map((session) => {
+            return new Promise((resolve, reject) => {
+                const req = objectStore.delete([session.senderKey, session.sessionId]);
+                req.onsuccess = resolve;
+                req.onerror = reject;
+            });
+        }));
+    }
+
+    markSessionsNeedingBackup(sessions, txn) {
+        if (!txn) {
+            txn = this._db.transaction("sessions_needing_backup", "readwrite");
+        }
+        const objectStore = txn.objectStore("sessions_needing_backup");
+        return Promise.all(sessions.map((session) => {
+            return new Promise((resolve, reject) => {
+                const req = objectStore.put({
+                    senderCurve25519Key: session.senderKey,
+                    sessionId: session.sessionId,
+                });
+                req.onsuccess = resolve;
+                req.onerror = reject;
+            });
+        }));
+    }
+
     doTxn(mode, stores, func) {
         const txn = this._db.transaction(stores, mode);
         const promise = promiseifyTxn(txn);
@@ -471,7 +585,7 @@ export class Backend {
 }
 
 export function upgradeDatabase(db, oldVersion) {
-    console.log(
+    logger.log(
         `Upgrading IndexedDBCryptoStore from version ${oldVersion}`
             + ` to ${VERSION}`,
     );
@@ -498,6 +612,11 @@ export function upgradeDatabase(db, oldVersion) {
     if (oldVersion < 6) {
         db.createObjectStore("rooms");
     }
+    if (oldVersion < 7) {
+        db.createObjectStore("sessions_needing_backup", {
+            keyPath: ["senderCurve25519Key", "sessionId"],
+        });
+    }
     // Expand as needed.
 }
 

src/crypto/store/indexeddb-crypto-store.js

@@ -17,9 +17,11 @@ limitations under the License.
 
 import Promise from 'bluebird';
 
+import logger from '../../logger';
 import LocalStorageCryptoStore from './localStorage-crypto-store';
 import MemoryCryptoStore from './memory-crypto-store';
 import * as IndexedDBCryptoStoreBackend from './indexeddb-crypto-store-backend';
+import {InvalidCryptoStoreError} from '../../errors';
 
 /**
  * Internal module. indexeddb storage for e2e.
@@ -64,7 +66,7 @@ export default class IndexedDBCryptoStore {
                 return;
             }
 
-            console.log(`connecting to indexeddb ${this._dbName}`);
+            logger.log(`connecting to indexeddb ${this._dbName}`);
 
             const req = this._indexedDB.open(
                 this._dbName, IndexedDBCryptoStoreBackend.VERSION,
@@ -77,7 +79,7 @@ export default class IndexedDBCryptoStore {
             };
 
             req.onblocked = () => {
-                console.log(
+                logger.log(
                     `can't yet open IndexedDBCryptoStore because it is open elsewhere`,
                 );
             };
@@ -89,20 +91,42 @@ export default class IndexedDBCryptoStore {
             req.onsuccess = (r) => {
                 const db = r.target.result;
 
-                console.log(`connected to indexeddb ${this._dbName}`);
+                logger.log(`connected to indexeddb ${this._dbName}`);
                 resolve(new IndexedDBCryptoStoreBackend.Backend(db));
             };
+        }).then((backend) => {
+            // Edge has IndexedDB but doesn't support compund keys which we use fairly extensively.
+            // Try a dummy query which will fail if the browser doesn't support compund keys, so
+            // we can fall back to a different backend.
+            return backend.doTxn(
+                'readonly',
+                [IndexedDBCryptoStore.STORE_INBOUND_GROUP_SESSIONS],
+                (txn) => {
+                    backend.getEndToEndInboundGroupSession('', '', txn, () => {});
+                }).then(() => {
+                    return backend;
+                },
+            );
         }).catch((e) => {
-            console.warn(
+            if (e.name === 'VersionError') {
+                logger.warn("Crypto DB is too new for us to use!", e);
+                // don't fall back to a different store: the user has crypto data
+                // in this db so we should use it or nothing at all.
+                throw new InvalidCryptoStoreError(InvalidCryptoStoreError.TOO_NEW);
+            }
+            logger.warn(
                 `unable to connect to indexeddb ${this._dbName}` +
                     `: falling back to localStorage store: ${e}`,
             );
-            return new LocalStorageCryptoStore(global.localStorage);
-        }).catch((e) => {
-            console.warn(
-                `unable to open localStorage: falling back to in-memory store: ${e}`,
-            );
-            return new MemoryCryptoStore();
+
+            try {
+                return new LocalStorageCryptoStore(global.localStorage);
+            } catch (e) {
+                logger.warn(
+                    `unable to open localStorage: falling back to in-memory store: ${e}`,
+                );
+                return new MemoryCryptoStore();
+            }
         });
 
         return this._backendPromise;
@@ -120,11 +144,11 @@ export default class IndexedDBCryptoStore {
                 return;
             }
 
-            console.log(`Removing indexeddb instance: ${this._dbName}`);
+            logger.log(`Removing indexeddb instance: ${this._dbName}`);
             const req = this._indexedDB.deleteDatabase(this._dbName);
 
             req.onblocked = () => {
-                console.log(
+                logger.log(
                     `can't yet delete IndexedDBCryptoStore because it is open elsewhere`,
                 );
             };
@@ -134,14 +158,14 @@ export default class IndexedDBCryptoStore {
             };
 
             req.onsuccess = () => {
-                console.log(`Removed indexeddb instance: ${this._dbName}`);
+                logger.log(`Removed indexeddb instance: ${this._dbName}`);
                 resolve();
             };
         }).catch((e) => {
             // in firefox, with indexedDB disabled, this fails with a
             // DOMError. We treat this as non-fatal, so that people can
             // still use the app.
-            console.warn(`unable to delete IndexedDBCryptoStore: ${e}`);
+            logger.warn(`unable to delete IndexedDBCryptoStore: ${e}`);
         });
     }
 
@@ -193,6 +217,24 @@ export default class IndexedDBCryptoStore {
         });
     }
 
+    /**
+     * Look for room key requests by target device and state
+     *
+     * @param {string} userId Target user ID
+     * @param {string} deviceId Target device ID
+     * @param {Array<Number>} wantedStates list of acceptable states
+     *
+     * @return {Promise} resolves to a list of all the
+     *    {@link module:crypto/store/base~OutgoingRoomKeyRequest}
+     */
+    getOutgoingRoomKeyRequestsByTarget(userId, deviceId, wantedStates) {
+        return this._connect().then((backend) => {
+            return backend.getOutgoingRoomKeyRequestsByTarget(
+                userId, deviceId, wantedStates,
+            );
+        });
+    }
+
     /**
      * Look for an existing room key request by id and state, and update it if
      * found
@@ -270,7 +312,10 @@ export default class IndexedDBCryptoStore {
      * @param {string} sessionId The ID of the session to retrieve
      * @param {*} txn An active transaction. See doTxn().
      * @param {function(object)} func Called with A map from sessionId
-     *     to Base64 end-to-end session.
+     *     to session information object with 'session' key being the
+     *     Base64 end-to-end session and lastReceivedMessageTs being the
+     *     timestamp in milliseconds at which the session last received
+     *     a message.
      */
     getEndToEndSession(deviceKey, sessionId, txn, func) {
         this._backendPromise.value().getEndToEndSession(deviceKey, sessionId, txn, func);
@@ -282,7 +327,10 @@ export default class IndexedDBCryptoStore {
      * @param {string} deviceKey The public key of the other device.
      * @param {*} txn An active transaction. See doTxn().
      * @param {function(object)} func Called with A map from sessionId
-     *     to Base64 end-to-end session.
+     *     to session information object with 'session' key being the
+     *     Base64 end-to-end session and lastReceivedMessageTs being the
+     *     timestamp in milliseconds at which the session last received
+     *     a message.
      */
     getEndToEndSessions(deviceKey, txn, func) {
         this._backendPromise.value().getEndToEndSessions(deviceKey, txn, func);
@@ -292,12 +340,12 @@ export default class IndexedDBCryptoStore {
      * Store a session between the logged-in user and another device
      * @param {string} deviceKey The public key of the other device.
      * @param {string} sessionId The ID for this end-to-end session.
-     * @param {string} session Base64 encoded end-to-end session.
+     * @param {string} sessionInfo Session information object
      * @param {*} txn An active transaction. See doTxn().
      */
-    storeEndToEndSession(deviceKey, sessionId, session, txn) {
+    storeEndToEndSession(deviceKey, sessionId, sessionInfo, txn) {
         this._backendPromise.value().storeEndToEndSession(
-            deviceKey, sessionId, session, txn,
+            deviceKey, sessionId, sessionInfo, txn,
         );
     }
 
@@ -407,6 +455,43 @@ export default class IndexedDBCryptoStore {
         this._backendPromise.value().getEndToEndRooms(txn, func);
     }
 
+    // session backups
+
+    /**
+     * Get the inbound group sessions that need to be backed up.
+     * @param {integer} limit The maximum number of sessions to retrieve.  0
+     * for no limit.
+     * @returns {Promise} resolves to an array of inbound group sessions
+     */
+    getSessionsNeedingBackup(limit) {
+        return this._connect().then((backend) => {
+            return backend.getSessionsNeedingBackup(limit);
+        });
+    }
+
+    /**
+     * Unmark sessions as needing to be backed up.
+     * @param {Array<object>} sessions The sessions that need to be backed up.
+     * @returns {Promise} resolves when the sessions are unmarked
+     */
+    unmarkSessionsNeedingBackup(sessions) {
+        return this._connect().then((backend) => {
+            return backend.unmarkSessionsNeedingBackup(sessions);
+        });
+    }
+
+    /**
+     * Mark sessions as needing to be backed up.
+     * @param {Array<object>} sessions The sessions that need to be backed up.
+     * @param {*} txn An active transaction. See doTxn(). (optional)
+     * @returns {Promise} resolves when the sessions are marked
+     */
+    markSessionsNeedingBackup(sessions, txn) {
+        return this._connect().then((backend) => {
+            return backend.markSessionsNeedingBackup(sessions, txn);
+        });
+    }
+
     /**
      * Perform a transaction on the crypto store. Any store methods
      * that require a transaction (txn) object to be passed in may
@@ -440,3 +525,4 @@ IndexedDBCryptoStore.STORE_SESSIONS = 'sessions';
 IndexedDBCryptoStore.STORE_INBOUND_GROUP_SESSIONS = 'inbound_group_sessions';
 IndexedDBCryptoStore.STORE_DEVICE_DATA = 'device_data';
 IndexedDBCryptoStore.STORE_ROOMS = 'rooms';
+IndexedDBCryptoStore.STORE_BACKUP = 'sessions_needing_backup';

src/crypto/store/localStorage-crypto-store.js

@@ -15,6 +15,8 @@ limitations under the License.
 */
 
 import Promise from 'bluebird';
+
+import logger from '../../logger';
 import MemoryCryptoStore from './memory-crypto-store.js';
 
 /**
@@ -32,6 +34,7 @@ const KEY_END_TO_END_ACCOUNT = E2E_PREFIX + "account";
 const KEY_DEVICE_DATA = E2E_PREFIX + "device_data";
 const KEY_INBOUND_SESSION_PREFIX = E2E_PREFIX + "inboundgroupsessions/";
 const KEY_ROOMS_PREFIX = E2E_PREFIX + "rooms/";
+const KEY_SESSIONS_NEEDING_BACKUP = E2E_PREFIX + "sessionsneedingbackup";
 
 function keyEndToEndSessions(deviceKey) {
     return E2E_PREFIX + "sessions/" + deviceKey;
@@ -65,7 +68,21 @@ export default class LocalStorageCryptoStore extends MemoryCryptoStore {
     }
 
     _getEndToEndSessions(deviceKey, txn, func) {
-        return getJsonItem(this.store, keyEndToEndSessions(deviceKey));
+        const sessions = getJsonItem(this.store, keyEndToEndSessions(deviceKey));
+        const fixedSessions = {};
+
+        // fix up any old sessions to be objects rather than just the base64 pickle
+        for (const [sid, val] of Object.entries(sessions || {})) {
+            if (typeof val === 'string') {
+                fixedSessions[sid] = {
+                    session: val,
+                };
+            } else {
+                fixedSessions[sid] = val;
+            }
+        }
+
+        return fixedSessions;
     }
 
     getEndToEndSession(deviceKey, sessionId, txn, func) {
@@ -77,9 +94,9 @@ export default class LocalStorageCryptoStore extends MemoryCryptoStore {
         func(this._getEndToEndSessions(deviceKey) || {});
     }
 
-    storeEndToEndSession(deviceKey, sessionId, session, txn) {
+    storeEndToEndSession(deviceKey, sessionId, sessionInfo, txn) {
         const sessions = this._getEndToEndSessions(deviceKey) || {};
-        sessions[sessionId] = session;
+        sessions[sessionId] = sessionInfo;
         setJsonItem(
             this.store, keyEndToEndSessions(deviceKey), sessions,
         );
@@ -165,6 +182,58 @@ export default class LocalStorageCryptoStore extends MemoryCryptoStore {
         func(result);
     }
 
+    getSessionsNeedingBackup(limit) {
+        const sessionsNeedingBackup
+              = getJsonItem(this.store, KEY_SESSIONS_NEEDING_BACKUP) || {};
+        const sessions = [];
+
+        for (const session in sessionsNeedingBackup) {
+            if (Object.prototype.hasOwnProperty.call(sessionsNeedingBackup, session)) {
+                // see getAllEndToEndInboundGroupSessions for the magic number explanations
+                const senderKey = session.substr(0, 43);
+                const sessionId = session.substr(44);
+                this.getEndToEndInboundGroupSession(
+                    senderKey, sessionId, null,
+                    (sessionData) => {
+                        sessions.push({
+                            senderKey: senderKey,
+                            sessionId: sessionId,
+                            sessionData: sessionData,
+                        });
+                    },
+                );
+                if (limit && session.length >= limit) {
+                    break;
+                }
+            }
+        }
+        return Promise.resolve(sessions);
+    }
+
+    unmarkSessionsNeedingBackup(sessions) {
+        const sessionsNeedingBackup
+              = getJsonItem(this.store, KEY_SESSIONS_NEEDING_BACKUP) || {};
+        for (const session of sessions) {
+            delete sessionsNeedingBackup[session.senderKey + '/' + session.sessionId];
+        }
+        setJsonItem(
+            this.store, KEY_SESSIONS_NEEDING_BACKUP, sessionsNeedingBackup,
+        );
+        return Promise.resolve();
+    }
+
+    markSessionsNeedingBackup(sessions) {
+        const sessionsNeedingBackup
+              = getJsonItem(this.store, KEY_SESSIONS_NEEDING_BACKUP) || {};
+        for (const session of sessions) {
+            sessionsNeedingBackup[session.senderKey + '/' + session.sessionId] = true;
+        }
+        setJsonItem(
+            this.store, KEY_SESSIONS_NEEDING_BACKUP, sessionsNeedingBackup,
+        );
+        return Promise.resolve();
+    }
+
     /**
      * Delete all data from this store.
      *
@@ -199,8 +268,8 @@ function getJsonItem(store, key) {
         // JSON.parse(null) === null, so this returns null.
         return JSON.parse(store.getItem(key));
     } catch (e) {
-        console.log("Error: Failed to get key %s: %s", key, e.stack || e);
-        console.log(e.stack);
+        logger.log("Error: Failed to get key %s: %s", key, e.stack || e);
+        logger.log(e.stack);
     }
     return null;
 }

src/crypto/store/memory-crypto-store.js

@@ -17,6 +17,7 @@ limitations under the License.
 
 import Promise from 'bluebird';
 
+import logger from '../../logger';
 import utils from '../../utils';
 
 /**
@@ -41,6 +42,8 @@ export default class MemoryCryptoStore {
         this._deviceData = null;
         // roomId -> Opaque roomInfo object
         this._rooms = {};
+        // Set of {senderCurve25519Key+'/'+sessionId}
+        this._sessionsNeedingBackup = {};
     }
 
     /**
@@ -71,7 +74,7 @@ export default class MemoryCryptoStore {
 
             if (existing) {
                 // this entry matches the request - return it.
-                console.log(
+                logger.log(
                     `already have key request outstanding for ` +
                     `${requestBody.room_id} / ${requestBody.session_id}: ` +
                     `not sending another`,
@@ -81,7 +84,7 @@ export default class MemoryCryptoStore {
 
             // we got to the end of the list without finding a match
             // - add the new request.
-            console.log(
+            logger.log(
                 `enqueueing key request for ${requestBody.room_id} / ` +
                 requestBody.session_id,
             );
@@ -144,6 +147,19 @@ export default class MemoryCryptoStore {
         return Promise.resolve(null);
     }
 
+    getOutgoingRoomKeyRequestsByTarget(userId, deviceId, wantedStates) {
+        const results = [];
+
+        for (const req of this._outgoingRoomKeyRequests) {
+            for (const state of wantedStates) {
+                if (req.state === state && req.recipients.includes({userId, deviceId})) {
+                    results.push(req);
+                }
+            }
+        }
+        return Promise.resolve(results);
+    }
+
     /**
      * Look for an existing room key request by id and state, and update it if
      * found
@@ -163,7 +179,7 @@ export default class MemoryCryptoStore {
             }
 
             if (req.state != expectedState) {
-                console.warn(
+                logger.warn(
                     `Cannot update room key request from ${expectedState} ` +
                     `as it was already updated to ${req.state}`,
                 );
@@ -194,7 +210,7 @@ export default class MemoryCryptoStore {
             }
 
             if (req.state != expectedState) {
-                console.warn(
+                logger.warn(
                     `Cannot delete room key request in state ${req.state} `
                     + `(expected ${expectedState})`,
                 );
@@ -233,13 +249,13 @@ export default class MemoryCryptoStore {
         func(this._sessions[deviceKey] || {});
     }
 
-    storeEndToEndSession(deviceKey, sessionId, session, txn) {
+    storeEndToEndSession(deviceKey, sessionId, sessionInfo, txn) {
         let deviceSessions = this._sessions[deviceKey];
         if (deviceSessions === undefined) {
             deviceSessions = {};
             this._sessions[deviceKey] = deviceSessions;
         }
-        deviceSessions[sessionId] = session;
+        deviceSessions[sessionId] = sessionInfo;
     }
 
     // Inbound Group Sessions
@@ -295,6 +311,41 @@ export default class MemoryCryptoStore {
         func(this._rooms);
     }
 
+    getSessionsNeedingBackup(limit) {
+        const sessions = [];
+        for (const session in this._sessionsNeedingBackup) {
+            if (this._inboundGroupSessions[session]) {
+                sessions.push({
+                    senderKey: session.substr(0, 43),
+                    sessionId: session.substr(44),
+                    sessionData: this._inboundGroupSessions[session],
+                });
+                if (limit && session.length >= limit) {
+                    break;
+                }
+            }
+        }
+        return Promise.resolve(sessions);
+    }
+
+    unmarkSessionsNeedingBackup(sessions) {
+        for (const session of sessions) {
+            const sessionKey = session.senderKey + '/' + session.sessionId;
+            delete this._sessionsNeedingBackup[sessionKey];
+        }
+        return Promise.resolve();
+    }
+
+    markSessionsNeedingBackup(sessions) {
+        for (const session of sessions) {
+            const sessionKey = session.senderKey + '/' + session.sessionId;
+            this._sessionsNeedingBackup[sessionKey] = true;
+        }
+        return Promise.resolve();
+    }
+
+    // Session key backups
+
     doTxn(mode, stores, func) {
         return Promise.resolve(func(null));
     }

src/errors.js

@@ -0,0 +1,46 @@
+// can't just do InvalidStoreError extends Error
+// because of http://babeljs.io/docs/usage/caveats/#classes
+export function InvalidStoreError(reason, value) {
+    const message = `Store is invalid because ${reason}, ` +
+        `please stop the client, delete all data and start the client again`;
+    const instance = Reflect.construct(Error, [message]);
+    Reflect.setPrototypeOf(instance, Reflect.getPrototypeOf(this));
+    instance.reason = reason;
+    instance.value = value;
+    return instance;
+}
+
+InvalidStoreError.TOGGLED_LAZY_LOADING = "TOGGLED_LAZY_LOADING";
+
+InvalidStoreError.prototype = Object.create(Error.prototype, {
+  constructor: {
+    value: Error,
+    enumerable: false,
+    writable: true,
+    configurable: true,
+  },
+});
+Reflect.setPrototypeOf(InvalidStoreError, Error);
+
+
+export function InvalidCryptoStoreError(reason) {
+    const message = `Crypto store is invalid because ${reason}, ` +
+        `please stop the client, delete all data and start the client again`;
+    const instance = Reflect.construct(Error, [message]);
+    Reflect.setPrototypeOf(instance, Reflect.getPrototypeOf(this));
+    instance.reason = reason;
+    instance.name = 'InvalidCryptoStoreError';
+    return instance;
+}
+
+InvalidCryptoStoreError.TOO_NEW = "TOO_NEW";
+
+InvalidCryptoStoreError.prototype = Object.create(Error.prototype, {
+  constructor: {
+    value: Error,
+    enumerable: false,
+    writable: true,
+    configurable: true,
+  },
+});
+Reflect.setPrototypeOf(InvalidCryptoStoreError, Error);

src/filter.js

@@ -51,6 +51,17 @@ function Filter(userId, filterId) {
     this.definition = {};
 }
 
+Filter.LAZY_LOADING_MESSAGES_FILTER = {
+    lazy_load_members: true,
+};
+
+Filter.LAZY_LOADING_SYNC_FILTER = {
+    room: {
+        state: Filter.LAZY_LOADING_MESSAGES_FILTER,
+    },
+};
+
+
 /**
  * Get the ID of this filter on your homeserver (if known)
  * @return {?Number} The filter ID

src/http-api.js

@@ -752,6 +752,8 @@ module.exports.MatrixHttpApi.prototype = {
                     method: method,
                     withCredentials: false,
                     qs: queryParams,
+                    qsStringifyOptions: opts.qsStringifyOptions,
+                    useQuerystring: true,
                     body: data,
                     json: false,
                     timeout: localTimeoutMs,

src/logger.js

@@ -0,0 +1,36 @@
+/*
+Copyright 2018 André Jaenisch
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+/**
+ * @module logger
+ */
+const log = require("loglevel");
+
+// This is to demonstrate, that you can use any namespace you want.
+// Namespaces allow you to turn on/off the logging for specific parts of the
+// application.
+// An idea would be to control this via an environment variable (on Node.js).
+// See https://www.npmjs.com/package/debug to see how this could be implemented
+// Part of #332 is introducing a logging library in the first place.
+const DEFAULT_NAME_SPACE = "matrix";
+const logger = log.getLogger(DEFAULT_NAME_SPACE);
+logger.setLevel(log.levels.DEBUG);
+
+/**
+ * Drop-in replacement for <code>console</code> using {@link https://www.npmjs.com/package/loglevel|loglevel}.
+ * Can be tailored down to specific use cases if needed.
+*/
+module.exports = logger;

src/matrix.js

@@ -34,6 +34,8 @@ module.exports.SyncAccumulator = require("./sync-accumulator");
 module.exports.MatrixHttpApi = require("./http-api").MatrixHttpApi;
 /** The {@link module:http-api.MatrixError|MatrixError} class. */
 module.exports.MatrixError = require("./http-api").MatrixError;
+/** The {@link module:errors.InvalidStoreError|InvalidStoreError} class. */
+module.exports.InvalidStoreError = require("./errors").InvalidStoreError;
 /** The {@link module:client.MatrixClient|MatrixClient} class. */
 module.exports.MatrixClient = require("./client").MatrixClient;
 /** The {@link module:models/room|Room} class. */

src/models/event-timeline-set.js

@@ -168,49 +168,19 @@ EventTimelineSet.prototype.resetLiveTimeline = function(
     // if timeline support is disabled, forget about the old timelines
     const resetAllTimelines = !this._timelineSupport || !forwardPaginationToken;
 
-    let newTimeline;
+    const oldTimeline = this._liveTimeline;
+    const newTimeline = resetAllTimelines ?
+        oldTimeline.forkLive(EventTimeline.FORWARDS) :
+        oldTimeline.fork(EventTimeline.FORWARDS);
+
     if (resetAllTimelines) {
-        newTimeline = new EventTimeline(this);
         this._timelines = [newTimeline];
         this._eventIdToTimeline = {};
     } else {
-        newTimeline = this.addTimeline();
+        this._timelines.push(newTimeline);
     }
 
-    const oldTimeline = this._liveTimeline;
-
-    // Collect the state events from the old timeline
-    const evMap = oldTimeline.getState(EventTimeline.FORWARDS).events;
-    const events = [];
-    for (const evtype in evMap) {
-        if (!evMap.hasOwnProperty(evtype)) {
-            continue;
-        }
-        for (const stateKey in evMap[evtype]) {
-            if (!evMap[evtype].hasOwnProperty(stateKey)) {
-                continue;
-            }
-            events.push(evMap[evtype][stateKey]);
-        }
-    }
-
-    // Use those events to initialise the state of the new live timeline
-    newTimeline.initialiseState(events);
-
-    const freshEndState = newTimeline._endState;
-    // Now clobber the end state of the new live timeline with that from the
-    // previous live timeline. It will be identical except that we'll keep
-    // using the same RoomMember objects for the 'live' set of members with any
-    // listeners still attached
-    newTimeline._endState = oldTimeline._endState;
-
-    // If we're not resetting all timelines, we need to fix up the old live timeline
-    if (!resetAllTimelines) {
-        // Firstly, we just stole the old timeline's end state, so it needs a new one.
-        // Just swap them around and give it the one we just generated for the
-        // new live timeline.
-        oldTimeline._endState = freshEndState;
-
+    if (forwardPaginationToken) {
         // Now set the forward pagination token on the old live timeline
         // so it can be forward-paginated.
         oldTimeline.setPaginationToken(

src/models/event-timeline.js

@@ -106,6 +106,50 @@ EventTimeline.prototype.initialiseState = function(stateEvents) {
     this._endState.setStateEvents(stateEvents);
 };
 
+/**
+ * Forks the (live) timeline, taking ownership of the existing directional state of this timeline.
+ * All attached listeners will keep receiving state updates from the new live timeline state.
+ * The end state of this timeline gets replaced with an independent copy of the current RoomState,
+ * and will need a new pagination token if it ever needs to paginate forwards.
+
+ * @param {string} direction   EventTimeline.BACKWARDS to get the state at the
+ *   start of the timeline; EventTimeline.FORWARDS to get the state at the end
+ *   of the timeline.
+ *
+ * @return {EventTimeline} the new timeline
+ */
+EventTimeline.prototype.forkLive = function(direction) {
+    const forkState = this.getState(direction);
+    const timeline = new EventTimeline(this._eventTimelineSet);
+    timeline._startState = forkState.clone();
+    // Now clobber the end state of the new live timeline with that from the
+    // previous live timeline. It will be identical except that we'll keep
+    // using the same RoomMember objects for the 'live' set of members with any
+    // listeners still attached
+    timeline._endState = forkState;
+    // Firstly, we just stole the current timeline's end state, so it needs a new one.
+    // Make an immutable copy of the state so back pagination will get the correct sentinels.
+    this._endState = forkState.clone();
+    return timeline;
+};
+
+/**
+ * Creates an independent timeline, inheriting the directional state from this timeline.
+ *
+ * @param {string} direction   EventTimeline.BACKWARDS to get the state at the
+ *   start of the timeline; EventTimeline.FORWARDS to get the state at the end
+ *   of the timeline.
+ *
+ * @return {EventTimeline} the new timeline
+ */
+EventTimeline.prototype.fork = function(direction) {
+    const forkState = this.getState(direction);
+    const timeline = new EventTimeline(this._eventTimelineSet);
+    timeline._startState = forkState.clone();
+    timeline._endState = forkState.clone();
+    return timeline;
+};
+
 /**
  * Get the ID of the room for this timeline
  * @return {string} room ID

src/models/room-member.js

@@ -58,10 +58,27 @@ function RoomMember(roomId, userId) {
     this.events = {
         member: null,
     };
+    this._isOutOfBand = false;
     this._updateModifiedTime();
 }
 utils.inherits(RoomMember, EventEmitter);
 
+/**
+ * Mark the member as coming from a channel that is not sync
+ */
+RoomMember.prototype.markOutOfBand = function() {
+    this._isOutOfBand = true;
+};
+
+/**
+ * @return {bool} does the member come from a channel that is not sync?
+ * This is used to store the member seperately
+ * from the sync state so it available across browser sessions.
+ */
+RoomMember.prototype.isOutOfBand = function() {
+    return this._isOutOfBand;
+};
+
 /**
  * Update this room member's membership event. May fire "RoomMember.name" if
  * this event updates this member's name.
@@ -75,13 +92,20 @@ RoomMember.prototype.setMembershipEvent = function(event, roomState) {
     if (event.getType() !== "m.room.member") {
         return;
     }
+
+    this._isOutOfBand = false;
+
     this.events.member = event;
 
     const oldMembership = this.membership;
     this.membership = event.getDirectionalContent().membership;
 
     const oldName = this.name;
-    this.name = calculateDisplayName(this, event, roomState);
+    this.name = calculateDisplayName(
+        this.userId,
+        event.getDirectionalContent().displayname,
+        roomState);
+
     this.rawDisplayName = event.getDirectionalContent().displayname || this.userId;
     if (oldMembership !== this.membership) {
         this._updateModifiedTime();
@@ -177,6 +201,44 @@ RoomMember.prototype.getLastModifiedTime = function() {
     return this._modified;
 };
 
+
+RoomMember.prototype.isKicked = function() {
+    return this.membership === "leave" &&
+        this.events.member.getSender() !== this.events.member.getStateKey();
+};
+
+/**
+ * If this member was invited with the is_direct flag set, return
+ * the user that invited this member
+ * @return {string} user id of the inviter
+ */
+RoomMember.prototype.getDMInviter = function() {
+    // when not available because that room state hasn't been loaded in,
+    // we don't really know, but more likely to not be a direct chat
+    if (this.events.member) {
+        // TODO: persist the is_direct flag on the member as more member events
+        //       come in caused by displayName changes.
+
+        // the is_direct flag is set on the invite member event.
+        // This is copied on the prev_content section of the join member event
+        // when the invite is accepted.
+
+        const memberEvent = this.events.member;
+        let memberContent = memberEvent.getContent();
+        let inviteSender = memberEvent.getSender();
+
+        if (memberContent.membership === "join") {
+            memberContent = memberEvent.getPrevContent();
+            inviteSender = memberEvent.getUnsigned().prev_sender;
+        }
+
+        if (memberContent.membership === "invite" && memberContent.is_direct) {
+            return inviteSender;
+        }
+    }
+};
+
+
 /**
  * Get the avatar URL for a room member.
  * @param {string} baseUrl The base homeserver URL See
@@ -200,10 +262,12 @@ RoomMember.prototype.getAvatarUrl =
     if (allowDefault === undefined) {
         allowDefault = true;
     }
-    if (!this.events.member && !allowDefault) {
+
+    const rawUrl = this.getMxcAvatarUrl();
+
+    if (!rawUrl && !allowDefault) {
         return null;
     }
-    const rawUrl = this.events.member ? this.events.member.getContent().avatar_url : null;
     const httpUrl = ContentRepo.getHttpUriForMxc(
         baseUrl, rawUrl, width, height, resizeMethod, allowDirectLinks,
     );
@@ -216,12 +280,21 @@ RoomMember.prototype.getAvatarUrl =
     }
     return null;
 };
+/**
+ * get the mxc avatar url, either from a state event, or from a lazily loaded member
+ * @return {string} the mxc avatar url
+ */
+RoomMember.prototype.getMxcAvatarUrl = function() {
+    if(this.events.member) {
+        return this.events.member.getDirectionalContent().avatar_url;
+    } else if(this.user) {
+        return this.user.avatarUrl;
+    }
+    return null;
+};
 
-function calculateDisplayName(member, event, roomState) {
-    const displayName = event.getDirectionalContent().displayname;
-    const selfUserId = member.userId;
-
-    if (!displayName) {
+function calculateDisplayName(selfUserId, displayName, roomState) {
+    if (!displayName || displayName === selfUserId) {
         return selfUserId;
     }
 

src/models/room-state.js

@@ -22,6 +22,11 @@ const EventEmitter = require("events").EventEmitter;
 const utils = require("../utils");
 const RoomMember = require("./room-member");
 
+// possible statuses for out-of-band member loading
+const OOB_STATUS_NOTSTARTED = 1;
+const OOB_STATUS_INPROGRESS = 2;
+const OOB_STATUS_FINISHED = 3;
+
 /**
  * Construct room state.
  *
@@ -46,13 +51,17 @@ const RoomMember = require("./room-member");
  * @constructor
  * @param {?string} roomId Optional. The ID of the room which has this state.
  * If none is specified it just tracks paginationTokens, useful for notifTimelineSet
+ * @param {?object} oobMemberFlags Optional. The state of loading out of bound members.
+ * As the timeline might get reset while they are loading, this state needs to be inherited
+ * and shared when the room state is cloned for the new timeline.
+ * This should only be passed from clone.
  * @prop {Object.<string, RoomMember>} members The room member dictionary, keyed
  * on the user's ID.
  * @prop {Object.<string, Object.<string, MatrixEvent>>} events The state
  * events dictionary, keyed on the event type and then the state_key value.
  * @prop {string} paginationToken The pagination token for this state.
  */
-function RoomState(roomId) {
+function RoomState(roomId, oobMemberFlags = undefined) {
     this.roomId = roomId;
     this.members = {
         // userId: RoomMember
@@ -70,6 +79,22 @@ function RoomState(roomId) {
     this._userIdsToDisplayNames = {};
     this._tokenToInvite = {}; // 3pid invite state_key to m.room.member invite
     this._joinedMemberCount = null; // cache of the number of joined members
+    // joined members count from summary api
+    // once set, we know the server supports the summary api
+    // and we should only trust that
+    // we could also only trust that before OOB members
+    // are loaded but doesn't seem worth the hassle atm
+    this._summaryJoinedMemberCount = null;
+    // same for invited member count
+    this._invitedMemberCount = null;
+    this._summaryInvitedMemberCount = null;
+
+    if (!oobMemberFlags) {
+        oobMemberFlags = {
+            status: OOB_STATUS_NOTSTARTED,
+        };
+    }
+    this._oobMemberFlags = oobMemberFlags;
 }
 utils.inherits(RoomState, EventEmitter);
 
@@ -79,14 +104,48 @@ utils.inherits(RoomState, EventEmitter);
  * @return {integer} The number of members in this room whose membership is 'join'
  */
 RoomState.prototype.getJoinedMemberCount = function() {
+    if (this._summaryJoinedMemberCount !== null) {
+        return this._summaryJoinedMemberCount;
+    }
     if (this._joinedMemberCount === null) {
-        this._joinedMemberCount = this.getMembers().filter((m) => {
-            return m.membership === 'join';
-        }).length;
+        this._joinedMemberCount = this.getMembers().reduce((count, m) => {
+            return m.membership === 'join' ? count + 1 : count;
+        }, 0);
     }
     return this._joinedMemberCount;
 };
 
+/**
+ * Set the joined member count explicitly (like from summary part of the sync response)
+ * @param {number} count the amount of joined members
+ */
+RoomState.prototype.setJoinedMemberCount = function(count) {
+    this._summaryJoinedMemberCount = count;
+};
+/**
+ * Returns the number of invited members in this room
+ * @return {integer} The number of members in this room whose membership is 'invite'
+ */
+RoomState.prototype.getInvitedMemberCount = function() {
+    if (this._summaryInvitedMemberCount !== null) {
+        return this._summaryInvitedMemberCount;
+    }
+    if (this._invitedMemberCount === null) {
+        this._invitedMemberCount = this.getMembers().reduce((count, m) => {
+            return m.membership === 'invite' ? count + 1 : count;
+        }, 0);
+    }
+    return this._invitedMemberCount;
+};
+
+/**
+ * Set the amount of invited members in this room
+ * @param {number} count the amount of invited members
+ */
+RoomState.prototype.setInvitedMemberCount = function(count) {
+    this._summaryInvitedMemberCount = count;
+};
+
 /**
  * Get all RoomMembers in this room.
  * @return {Array<RoomMember>} A list of RoomMembers.
@@ -119,12 +178,9 @@ RoomState.prototype.getSentinelMember = function(userId) {
 
     if (sentinel === undefined) {
         sentinel = new RoomMember(this.roomId, userId);
-        const membershipEvent = this.getStateEvents("m.room.member", userId);
-        if (!membershipEvent) return null;
-        sentinel.setMembershipEvent(membershipEvent, this);
-        const pwrLvlEvent = this.getStateEvents("m.room.power_levels", "");
-        if (pwrLvlEvent) {
-            sentinel.setPowerLevelEvent(pwrLvlEvent);
+        const member = this.members[userId];
+        if (member) {
+            sentinel.setMembershipEvent(member.events.member, this);
         }
         this._sentinels[userId] = sentinel;
     }
@@ -152,6 +208,67 @@ RoomState.prototype.getStateEvents = function(eventType, stateKey) {
     return event ? event : null;
 };
 
+/**
+ * Creates a copy of this room state so that mutations to either won't affect the other.
+ * @return {RoomState} the copy of the room state
+ */
+RoomState.prototype.clone = function() {
+    const copy = new RoomState(this.roomId, this._oobMemberFlags);
+
+    // Ugly hack: because setStateEvents will mark
+    // members as susperseding future out of bound members
+    // if loading is in progress (through _oobMemberFlags)
+    // since these are not new members, we're merely copying them
+    // set the status to not started
+    // after copying, we set back the status
+    const status = this._oobMemberFlags.status;
+    this._oobMemberFlags.status = OOB_STATUS_NOTSTARTED;
+
+    Object.values(this.events).forEach((eventsByStateKey) => {
+        const eventsForType = Object.values(eventsByStateKey);
+        copy.setStateEvents(eventsForType);
+    });
+
+    // Ugly hack: see above
+    this._oobMemberFlags.status = status;
+
+    if (this._summaryInvitedMemberCount !== null) {
+        copy.setInvitedMemberCount(this.getInvitedMemberCount());
+    }
+    if (this._summaryJoinedMemberCount !== null) {
+        copy.setJoinedMemberCount(this.getJoinedMemberCount());
+    }
+
+    // copy out of band flags if needed
+    if (this._oobMemberFlags.status == OOB_STATUS_FINISHED) {
+        // copy markOutOfBand flags
+        this.getMembers().forEach((member) => {
+            if (member.isOutOfBand()) {
+                const copyMember = copy.getMember(member.userId);
+                copyMember.markOutOfBand();
+            }
+        });
+    }
+
+    return copy;
+};
+
+/**
+ * Add previously unknown state events.
+ * When lazy loading members while back-paginating,
+ * the relevant room state for the timeline chunk at the end
+ * of the chunk can be set with this method.
+ * @param {MatrixEvent[]} events state events to prepend
+ */
+RoomState.prototype.setUnknownStateEvents = function(events) {
+    const unknownStateEvents = events.filter((event) => {
+        return this.events[event.getType()] === undefined ||
+            this.events[event.getType()][event.getStateKey()] === undefined;
+    });
+
+    this.setStateEvents(unknownStateEvents);
+};
+
 /**
  * Add an array of one or more state MatrixEvents, overwriting
  * any existing state with the same {type, stateKey} tuple. Will fire
@@ -175,10 +292,7 @@ RoomState.prototype.setStateEvents = function(stateEvents) {
             return;
         }
 
-        if (self.events[event.getType()] === undefined) {
-            self.events[event.getType()] = {};
-        }
-        self.events[event.getType()][event.getStateKey()] = event;
+        self._setStateEvent(event);
         if (event.getType() === "m.room.member") {
             _updateDisplayNameCache(
                 self, event.getStateKey(), event.getContent().displayname,
@@ -216,27 +330,10 @@ RoomState.prototype.setStateEvents = function(stateEvents) {
                     event.getPrevContent().displayname;
             }
 
-            let member = self.members[userId];
-            self._joinedMemberCount = null;
-
-            if (!member) {
-                member = new RoomMember(event.getRoomId(), userId);
-                // add member to members before emitting any events,
-                // as event handlers often lookup the member
-                self.members[userId] = member;
-                self.emit("RoomState.newMember", event, self, member);
-            }
-
+            const member = self._getOrCreateMember(userId, event);
             member.setMembershipEvent(event, self);
-            // this member may have a power level already, so set it.
-            const pwrLvlEvent = self.getStateEvents("m.room.power_levels", "");
-            if (pwrLvlEvent) {
-                member.setPowerLevelEvent(pwrLvlEvent);
-            }
-
-            // blow away the sentinel which is now outdated
-            delete self._sentinels[userId];
 
+            self._updateMember(member);
             self.emit("RoomState.members", event, self, member);
         } else if (event.getType() === "m.room.power_levels") {
             const members = utils.values(self.members);
@@ -251,6 +348,140 @@ RoomState.prototype.setStateEvents = function(stateEvents) {
     });
 };
 
+/**
+ * Looks up a member by the given userId, and if it doesn't exist,
+ * create it and emit the `RoomState.newMember` event.
+ * This method makes sure the member is added to the members dictionary
+ * before emitting, as this is done from setStateEvents and _setOutOfBandMember.
+ * @param {string} userId the id of the user to look up
+ * @param {MatrixEvent} event the membership event for the (new) member. Used to emit.
+ * @fires module:client~MatrixClient#event:"RoomState.newMember"
+ * @returns {RoomMember} the member, existing or newly created.
+ */
+RoomState.prototype._getOrCreateMember = function(userId, event) {
+    let member = this.members[userId];
+    if (!member) {
+        member = new RoomMember(this.roomId, userId);
+        // add member to members before emitting any events,
+        // as event handlers often lookup the member
+        this.members[userId] = member;
+        this.emit("RoomState.newMember", event, this, member);
+    }
+    return member;
+};
+
+RoomState.prototype._setStateEvent = function(event) {
+    if (this.events[event.getType()] === undefined) {
+        this.events[event.getType()] = {};
+    }
+    this.events[event.getType()][event.getStateKey()] = event;
+};
+
+RoomState.prototype._updateMember = function(member) {
+    // this member may have a power level already, so set it.
+    const pwrLvlEvent = this.getStateEvents("m.room.power_levels", "");
+    if (pwrLvlEvent) {
+        member.setPowerLevelEvent(pwrLvlEvent);
+    }
+
+    // blow away the sentinel which is now outdated
+    delete this._sentinels[member.userId];
+
+    this.members[member.userId] = member;
+    this._joinedMemberCount = null;
+    this._invitedMemberCount = null;
+};
+
+/**
+ * Get the out-of-band members loading state, whether loading is needed or not.
+ * Note that loading might be in progress and hence isn't needed.
+ * @return {bool} whether or not the members of this room need to be loaded
+ */
+RoomState.prototype.needsOutOfBandMembers = function() {
+    return this._oobMemberFlags.status === OOB_STATUS_NOTSTARTED;
+};
+
+/**
+ * Mark this room state as waiting for out-of-band members,
+ * ensuring it doesn't ask for them to be requested again
+ * through needsOutOfBandMembers
+ */
+RoomState.prototype.markOutOfBandMembersStarted = function() {
+    if (this._oobMemberFlags.status !== OOB_STATUS_NOTSTARTED) {
+        return;
+    }
+    this._oobMemberFlags.status = OOB_STATUS_INPROGRESS;
+};
+
+/**
+ * Mark this room state as having failed to fetch out-of-band members
+ */
+RoomState.prototype.markOutOfBandMembersFailed = function() {
+    if (this._oobMemberFlags.status !== OOB_STATUS_INPROGRESS) {
+        return;
+    }
+    this._oobMemberFlags.status = OOB_STATUS_NOTSTARTED;
+};
+
+/**
+ * Clears the loaded out-of-band members
+ */
+RoomState.prototype.clearOutOfBandMembers = function() {
+    let count = 0;
+    Object.keys(this.members).forEach((userId) => {
+        const member = this.members[userId];
+        if (member.isOutOfBand()) {
+            ++count;
+            delete this.members[userId];
+        }
+    });
+    console.log(`LL: RoomState removed ${count} members...`);
+    this._oobMemberFlags.status = OOB_STATUS_NOTSTARTED;
+};
+
+/**
+ * Sets the loaded out-of-band members.
+ * @param {MatrixEvent[]} stateEvents array of membership state events
+ */
+RoomState.prototype.setOutOfBandMembers = function(stateEvents) {
+    console.log(`LL: RoomState about to set ${stateEvents.length} OOB members ...`);
+    if (this._oobMemberFlags.status !== OOB_STATUS_INPROGRESS) {
+        return;
+    }
+    console.log(`LL: RoomState put in OOB_STATUS_FINISHED state ...`);
+    this._oobMemberFlags.status = OOB_STATUS_FINISHED;
+    stateEvents.forEach((e) => this._setOutOfBandMember(e));
+};
+
+/**
+ * Sets a single out of band member, used by both setOutOfBandMembers and clone
+ * @param {MatrixEvent} stateEvent membership state event
+ */
+RoomState.prototype._setOutOfBandMember = function(stateEvent) {
+    if (stateEvent.getType() !== 'm.room.member') {
+        return;
+    }
+    const userId = stateEvent.getStateKey();
+    const existingMember = this.getMember(userId);
+    // never replace members received as part of the sync
+    if (existingMember && !existingMember.isOutOfBand()) {
+        return;
+    }
+
+    const member = this._getOrCreateMember(userId, stateEvent);
+    member.setMembershipEvent(stateEvent, this);
+    // needed to know which members need to be stored seperately
+    // as they are not part of the sync accumulator
+    // this is cleared by setMembershipEvent so when it's updated through /sync
+    member.markOutOfBand();
+
+    _updateDisplayNameCache(this, member.userId, member.name);
+
+    this._setStateEvent(stateEvent);
+    this._updateMember(member);
+    this.emit("RoomState.members", stateEvent, this, member);
+};
+
 /**
  * Set the current typing event for this room.
  * @param {MatrixEvent} event The typing event
@@ -404,11 +635,6 @@ RoomState.prototype.maySendStateEvent = function(stateEventType, userId) {
  *                        according to the room's state.
  */
 RoomState.prototype._maySendEventOfType = function(eventType, userId, state) {
-    const member = this.getMember(userId);
-    if (!member || member.membership == 'leave') {
-        return false;
-    }
-
     const power_levels_event = this.getStateEvents('m.room.power_levels', '');
 
     let power_levels;
@@ -416,25 +642,34 @@ RoomState.prototype._maySendEventOfType = function(eventType, userId, state) {
 
     let state_default = 0;
     let events_default = 0;
+    let powerLevel = 0;
     if (power_levels_event) {
         power_levels = power_levels_event.getContent();
         events_levels = power_levels.events || {};
 
-        if (utils.isNumber(power_levels.state_default)) {
+        if (Number.isFinite(power_levels.state_default)) {
             state_default = power_levels.state_default;
         } else {
             state_default = 50;
         }
-        if (utils.isNumber(power_levels.events_default)) {
+
+        const userPowerLevel = power_levels.users && power_levels.users[userId];
+        if (Number.isFinite(userPowerLevel)) {
+            powerLevel = userPowerLevel;
+        } else if(Number.isFinite(power_levels.users_default)) {
+            powerLevel = power_levels.users_default;
+        }
+
+        if (Number.isFinite(power_levels.events_default)) {
             events_default = power_levels.events_default;
         }
     }
 
     let required_level = state ? state_default : events_default;
-    if (utils.isNumber(events_levels[eventType])) {
+    if (Number.isFinite(events_levels[eventType])) {
         required_level = events_levels[eventType];
     }
-    return member.powerLevel >= required_level;
+    return powerLevel >= required_level;
 };
 
 /**

src/models/room.js

@@ -1,5 +1,6 @@
 /*
 Copyright 2015, 2016 OpenMarket Ltd
+Copyright 2018 New Vector Ltd
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -21,6 +22,7 @@ const EventEmitter = require("events").EventEmitter;
 
 const EventStatus = require("./event").EventStatus;
 const RoomSummary = require("./room-summary");
+const RoomMember = require("./room-member");
 const MatrixEvent = require("./event").MatrixEvent;
 const utils = require("../utils");
 const ContentRepo = require("../content-repo");
@@ -29,6 +31,8 @@ const EventTimelineSet = require("./event-timeline-set");
 
 import ReEmitter from '../ReEmitter';
 
+const LATEST_ROOM_VERSION = '1';
+
 function synthesizeReceipt(userId, event, receiptType) {
     // console.log("synthesizing receipt for "+event.getId());
     // This is really ugly because JS has no way to express an object literal
@@ -68,6 +72,8 @@ function synthesizeReceipt(userId, event, receiptType) {
  * @constructor
  * @alias module:models/room
  * @param {string} roomId Required. The ID of this room.
+ * @param {MatrixClient} client Required. The client, used to lazy load members.
+ * @param {string} myUserId Required. The ID of the syncing user.
  * @param {Object=} opts Configuration options
  * @param {*} opts.storageToken Optional. The token which a data store can use
  * to remember the state of the room. What this means is dependent on the store
@@ -102,7 +108,7 @@ function synthesizeReceipt(userId, event, receiptType) {
  * @prop {*} storageToken A token which a data store can use to remember
  * the state of the room.
  */
-function Room(roomId, opts) {
+function Room(roomId, client, myUserId, opts) {
     opts = opts || {};
     opts.pendingEventOrdering = opts.pendingEventOrdering || "chronological";
 
@@ -115,6 +121,7 @@ function Room(roomId, opts) {
         );
     }
 
+    this.myUserId = myUserId;
     this.roomId = roomId;
     this.name = roomId;
     this.tags = {
@@ -171,7 +178,18 @@ function Room(roomId, opts) {
 
     // read by megolm; boolean value - null indicates "use global value"
     this._blacklistUnverifiedDevices = null;
+    this._selfMembership = null;
+    this._summaryHeroes = null;
+    // awaited by getEncryptionTargetMembers while room members are loading
+
+    this._client = client;
+    if (!this._opts.lazyLoadMembers) {
+        this._membersPromise = Promise.resolve();
+    } else {
+        this._membersPromise = null;
+    }
 }
+
 utils.inherits(Room, EventEmitter);
 
 /**
@@ -180,12 +198,36 @@ utils.inherits(Room, EventEmitter);
  */
 Room.prototype.getVersion = function() {
     const createEvent = this.currentState.getStateEvents("m.room.create", "");
-    if (!createEvent) return null;
+    if (!createEvent) {
+        console.warn("Room " + this.room_id + " does not have an m.room.create event");
+        return '1';
+    }
     const ver = createEvent.getContent()['room_version'];
     if (ver === undefined) return '1';
     return ver;
 };
 
+/**
+ * Determines whether this room needs to be upgraded to a new version
+ * @returns {string?} What version the room should be upgraded to, or null if
+ *     the room does not require upgrading at this time.
+ */
+Room.prototype.shouldUpgradeToVersion = function() {
+    // This almost certainly won't be the way this actually works - this
+    // is essentially a stub method.
+    if (this.getVersion() === LATEST_ROOM_VERSION) return null;
+    return LATEST_ROOM_VERSION;
+};
+
+/**
+ * Determines whether the given user is permitted to perform a room upgrade
+ * @param {String} userId The ID of the user to test against
+ * @returns {bool} True if the given user is permitted to upgrade the room
+ */
+Room.prototype.userMayUpgradeRoom = function(userId) {
+    return this.currentState.maySendStateEvent("m.room.tombstone", userId);
+};
+
 /**
  * Get the list of pending sent events for this room
  *
@@ -213,6 +255,232 @@ Room.prototype.getLiveTimeline = function() {
     return this.getUnfilteredTimelineSet().getLiveTimeline();
 };
 
+/**
+ * @param {string} myUserId the user id for the logged in member
+ * @return {string} the membership type (join | leave | invite) for the logged in user
+ */
+Room.prototype.getMyMembership = function() {
+    return this._selfMembership;
+};
+
+/**
+ * If this room is a DM we're invited to,
+ * try to find out who invited us
+ * @return {string} user id of the inviter
+ */
+Room.prototype.getDMInviter = function() {
+    if (this.myUserId) {
+        const me = this.getMember(this.myUserId);
+        if (me) {
+            return me.getDMInviter();
+        }
+    }
+    if (this._selfMembership === "invite") {
+        // fall back to summary information
+        const memberCount = this.getInvitedAndJoinedMemberCount();
+        if (memberCount == 2 && this._summaryHeroes.length) {
+            return this._summaryHeroes[0];
+        }
+    }
+};
+
+/**
+ * Assuming this room is a DM room, tries to guess with which user.
+ * @return {string} user id of the other member (could be syncing user)
+ */
+Room.prototype.guessDMUserId = function() {
+    const me = this.getMember(this.myUserId);
+    if (me) {
+        const inviterId = me.getDMInviter();
+        if (inviterId) {
+            return inviterId;
+        }
+    }
+    // remember, we're assuming this room is a DM,
+    // so returning the first member we find should be fine
+    const hasHeroes = Array.isArray(this._summaryHeroes) &&
+        this._summaryHeroes.length;
+    if (hasHeroes) {
+        return this._summaryHeroes[0];
+    }
+    const members = this.currentState.getMembers();
+    const anyMember = members.find((m) => m.userId !== this.myUserId);
+    if (anyMember) {
+        return anyMember.userId;
+    }
+    // it really seems like I'm the only user in the room
+    // so I probably created a room with just me in it
+    // and marked it as a DM. Ok then
+    return this.myUserId;
+};
+
+Room.prototype.getAvatarFallbackMember = function() {
+    const memberCount = this.getInvitedAndJoinedMemberCount();
+    if (memberCount > 2) {
+        return;
+    }
+    const hasHeroes = Array.isArray(this._summaryHeroes) &&
+        this._summaryHeroes.length;
+    if (hasHeroes) {
+        const availableMember = this._summaryHeroes.map((userId) => {
+            return this.getMember(userId);
+        }).find((member) => !!member);
+        if (availableMember) {
+            return availableMember;
+        }
+    }
+    const members = this.currentState.getMembers();
+    // could be different than memberCount
+    // as this includes left members
+    if (members.length <= 2) {
+        const availableMember = members.find((m) => {
+            return m.userId !== this.myUserId;
+        });
+        if (availableMember) {
+            return availableMember;
+        }
+    }
+    // if all else fails, try falling back to a user,
+    // and create a one-off member for it
+    if (hasHeroes) {
+        const availableUser = this._summaryHeroes.map((userId) => {
+            return this._client.getUser(userId);
+        }).find((user) => !!user);
+        if (availableUser) {
+            const member = new RoomMember(
+                this.roomId, availableUser.userId);
+            member.user = availableUser;
+            return member;
+        }
+    }
+};
+
+/**
+ * Sets the membership this room was received as during sync
+ * @param {string} membership join | leave | invite
+ */
+Room.prototype.updateMyMembership = function(membership) {
+    const prevMembership = this._selfMembership;
+    this._selfMembership = membership;
+    if (prevMembership !== membership) {
+        if (membership === "leave") {
+            this._cleanupAfterLeaving();
+        }
+        this.emit("Room.myMembership", this, membership, prevMembership);
+    }
+};
+
+Room.prototype._loadMembersFromServer = async function() {
+    const lastSyncToken = this._client.store.getSyncToken();
+    const queryString = utils.encodeParams({
+        not_membership: "leave",
+        at: lastSyncToken,
+    });
+    const path = utils.encodeUri("/rooms/$roomId/members?" + queryString,
+        {$roomId: this.roomId});
+    const http = this._client._http;
+    const response = await http.authedRequest(undefined, "GET", path);
+    return response.chunk;
+};
+
+
+Room.prototype._loadMembers = async function() {
+    // were the members loaded from the server?
+    let fromServer = false;
+    let rawMembersEvents =
+        await this._client.store.getOutOfBandMembers(this.roomId);
+    if (rawMembersEvents === null) {
+        fromServer = true;
+        rawMembersEvents = await this._loadMembersFromServer();
+        console.log(`LL: got ${rawMembersEvents.length} ` +
+            `members from server for room ${this.roomId}`);
+    }
+    const memberEvents = rawMembersEvents.map(this._client.getEventMapper());
+    return {memberEvents, fromServer};
+};
+
+/**
+ * Preloads the member list in case lazy loading
+ * of memberships is in use. Can be called multiple times,
+ * it will only preload once.
+ * @return {Promise} when preloading is done and
+ * accessing the members on the room will take
+ * all members in the room into account
+ */
+Room.prototype.loadMembersIfNeeded = function() {
+    if (this._membersPromise) {
+        return this._membersPromise;
+    }
+
+    // mark the state so that incoming messages while
+    // the request is in flight get marked as superseding
+    // the OOB members
+    this.currentState.markOutOfBandMembersStarted();
+
+    const inMemoryUpdate = this._loadMembers().then((result) => {
+        this.currentState.setOutOfBandMembers(result.memberEvents);
+        // now the members are loaded, start to track the e2e devices if needed
+        if (this._client.isRoomEncrypted(this.roomId)) {
+            this._client._crypto.trackRoomDevices(this.roomId);
+        }
+        return result.fromServer;
+    }).catch((err) => {
+        // allow retries on fail
+        this._membersPromise = null;
+        this.currentState.markOutOfBandMembersFailed();
+        throw err;
+    });
+    // update members in storage, but don't wait for it
+    inMemoryUpdate.then((fromServer) => {
+        if (fromServer) {
+            const oobMembers = this.currentState.getMembers()
+                .filter((m) => m.isOutOfBand())
+                .map((m) => m.events.member.event);
+            console.log(`LL: telling store to write ${oobMembers.length}`
+                + ` members for room ${this.roomId}`);
+            const store = this._client.store;
+            return store.setOutOfBandMembers(this.roomId, oobMembers)
+                // swallow any IDB error as we don't want to fail
+                // because of this
+                .catch((err) => {
+                    console.log("LL: storing OOB room members failed, oh well",
+                        err);
+                });
+        }
+    }).catch((err) => {
+        // as this is not awaited anywhere,
+        // at least show the error in the console
+        console.error(err);
+    });
+
+    this._membersPromise = inMemoryUpdate;
+
+    return this._membersPromise;
+};
+
+/**
+ * Removes the lazily loaded members from storage if needed
+ */
+Room.prototype.clearLoadedMembersIfNeeded = async function() {
+    if (this._opts.lazyLoadMembers && this._membersPromise) {
+        await this.loadMembersIfNeeded();
+        await this._client.store.clearOutOfBandMembers(this.roomId);
+        this.currentState.clearOutOfBandMembers();
+        this._membersPromise = null;
+    }
+};
+
+/**
+ * called when sync receives this room in the leave section
+ * to do cleanup after leaving a room. Possibly called multiple times.
+ */
+Room.prototype._cleanupAfterLeaving = function() {
+    this.clearLoadedMembersIfNeeded().catch((err) => {
+        console.error(`error after clearing loaded members from ` +
+            `room ${this.roomId} after leaving`);
+        console.dir(err);
+    });
+};
 
 /**
  * Reset the live timeline of all timelineSets, and start new ones.
@@ -318,6 +586,26 @@ Room.prototype.setUnreadNotificationCount = function(type, count) {
     this._notificationCounts[type] = count;
 };
 
+Room.prototype.setSummary = function(summary) {
+    const heroes = summary["m.heroes"];
+    const joinedCount = summary["m.joined_member_count"];
+    const invitedCount = summary["m.invited_member_count"];
+    if (Number.isInteger(joinedCount)) {
+        this.currentState.setJoinedMemberCount(joinedCount);
+    }
+    if (Number.isInteger(invitedCount)) {
+        this.currentState.setInvitedMemberCount(invitedCount);
+    }
+    if (Array.isArray(heroes)) {
+        // be cautious about trusting server values,
+        // and make sure heroes doesn't contain our own id
+        // just to be sure
+        this._summaryHeroes = heroes.filter((userId) => {
+            return userId !== this.myUserId;
+        });
+    }
+};
+
 /**
  * Whether to send encrypted messages to devices within this room.
  * @param {Boolean} value true to blacklist unverified devices, null
@@ -442,11 +730,7 @@ Room.prototype.addEventsToTimeline = function(events, toStartOfTimeline,
  * @return {RoomMember} The member or <code>null</code>.
  */
  Room.prototype.getMember = function(userId) {
-    const member = this.currentState.members[userId];
-    if (!member) {
-        return null;
-    }
-    return member;
+    return this.currentState.getMember(userId);
  };
 
 /**
@@ -457,6 +741,33 @@ Room.prototype.addEventsToTimeline = function(events, toStartOfTimeline,
     return this.getMembersWithMembership("join");
  };
 
+/**
+ * Returns the number of joined members in this room
+ * This method caches the result.
+ * This is a wrapper around the method of the same name in roomState, returning
+ * its result for the room's current state.
+ * @return {integer} The number of members in this room whose membership is 'join'
+ */
+Room.prototype.getJoinedMemberCount = function() {
+    return this.currentState.getJoinedMemberCount();
+};
+
+/**
+ * Returns the number of invited members in this room
+ * @return {integer} The number of members in this room whose membership is 'invite'
+ */
+Room.prototype.getInvitedMemberCount = function() {
+    return this.currentState.getInvitedMemberCount();
+};
+
+/**
+ * Returns the number of invited + joined members in this room
+ * @return {integer} The number of members in this room whose membership is 'invite' or 'join'
+ */
+Room.prototype.getInvitedAndJoinedMemberCount = function() {
+    return this.getInvitedMemberCount() + this.getJoinedMemberCount();
+};
+
 /**
  * Get a list of members with given membership state.
  * @param {string} membership The membership state.
@@ -470,10 +781,11 @@ Room.prototype.addEventsToTimeline = function(events, toStartOfTimeline,
 
  /**
   * Get a list of members we should be encrypting for in this room
-  * @return {RoomMember[]} A list of members who we should encrypt messages for
-  *                        in this room.
+  * @return {Promise<RoomMember[]>} A list of members who
+  * we should encrypt messages for in this room.
   */
- Room.prototype.getEncryptionTargetMembers = function() {
+ Room.prototype.getEncryptionTargetMembers = async function() {
+    await this.loadMembersIfNeeded();
     let members = this.getMembersWithMembership("join");
     if (this.shouldEncryptForInvitedMembers()) {
         members = members.concat(this.getMembersWithMembership("invite"));
@@ -687,6 +999,10 @@ Room.prototype.addPendingEvent = function(event, txnId) {
     this._txnToEvent[txnId] = event;
 
     if (this._opts.pendingEventOrdering == "detached") {
+        if (this._pendingEventList.some((e) => e.status === EventStatus.NOT_SENT)) {
+            console.warn("Setting event as NOT_SENT due to messages in the same state");
+            event.status = EventStatus.NOT_SENT;
+        }
         this._pendingEventList.push(event);
     } else {
         for (let i = 0; i < this._timelineSets.length; i++) {
@@ -943,15 +1259,14 @@ Room.prototype.removeEvent = function(eventId) {
  * Recalculate various aspects of the room, including the room name and
  * room summary. Call this any time the room's current state is modified.
  * May fire "Room.name" if the room name is updated.
- * @param {string} userId The client's user ID.
  * @fires module:client~MatrixClient#event:"Room.name"
  */
-Room.prototype.recalculate = function(userId) {
+Room.prototype.recalculate = function() {
     // set fake stripped state events if this is an invite room so logic remains
     // consistent elsewhere.
     const self = this;
     const membershipEvent = this.currentState.getStateEvents(
-        "m.room.member", userId,
+        "m.room.member", this.myUserId,
     );
     if (membershipEvent && membershipEvent.getContent().membership === "invite") {
         const strippedStateEvents = membershipEvent.event.invite_room_state || [];
@@ -967,14 +1282,14 @@ Room.prototype.recalculate = function(userId) {
                     content: strippedEvent.content,
                     event_id: "$fake" + Date.now(),
                     room_id: self.roomId,
-                    user_id: userId, // technically a lie
+                    user_id: self.myUserId, // technically a lie
                 })]);
             }
         });
     }
 
     const oldName = this.name;
-    this.name = calculateRoomName(this, userId);
+    this.name = calculateRoomName(this, this.myUserId);
     this.summary = new RoomSummary(this.roomId, {
         title: this.name,
     });
@@ -1186,6 +1501,17 @@ Room.prototype.getAccountData = function(type) {
     return this.accountData[type];
 };
 
+
+/**
+ * Returns wheter the syncing user has permission to send a message in the room
+ * @return {boolean} true if the user should be permitted to send
+ *                   message events into the room.
+ */
+Room.prototype.maySendMessage = function() {
+    return this.getMyMembership() === 'join' &&
+        this.currentState.maySendEvent('m.room.message', this.myUserId);
+};
+
 /**
  * This is an internal method. Calculates the name of the room from the current
  * room state.
@@ -1219,87 +1545,83 @@ function calculateRoomName(room, userId, ignoreRoomNameEvent) {
         return alias;
     }
 
-    // get members that are NOT ourselves and are actually in the room.
-    const otherMembers = utils.filter(room.currentState.getMembers(), function(m) {
-        return (
-            m.userId !== userId && m.membership !== "leave" && m.membership !== "ban"
-        );
-    });
-    const allMembers = utils.filter(room.currentState.getMembers(), function(m) {
-        return (m.membership !== "leave");
-    });
-    const myMemberEventArray = utils.filter(room.currentState.getMembers(), function(m) {
-        return (m.userId == userId);
-    });
-    const myMemberEvent = (
-        (myMemberEventArray.length && myMemberEventArray[0].events) ?
-            myMemberEventArray[0].events.member.event : undefined
-    );
+    const joinedMemberCount = room.currentState.getJoinedMemberCount();
+    const invitedMemberCount = room.currentState.getInvitedMemberCount();
+    // -1 because these numbers include the syncing user
+    const inviteJoinCount = joinedMemberCount + invitedMemberCount - 1;
 
-    // TODO: Localisation
-    if (myMemberEvent && myMemberEvent.content.membership == "invite") {
-        if (room.currentState.getMember(myMemberEvent.sender)) {
-            // extract who invited us to the room
-            return room.currentState.getMember(
-                myMemberEvent.sender,
-            ).name;
-        } else if (allMembers[0].events.member) {
-            // use the sender field from the invite event, although this only
-            // gets us the mxid
-            return myMemberEvent.sender;
-        } else {
-            return "Room Invite";
-        }
+    // get members that are NOT ourselves and are actually in the room.
+    let otherNames = null;
+    if (room._summaryHeroes) {
+        // if we have a summary, the member state events
+        // should be in the room state
+        otherNames = room._summaryHeroes.map((userId) => {
+            const member = room.getMember(userId);
+            return member ? member.name : userId;
+        });
+    } else {
+        let otherMembers = room.currentState.getMembers().filter((m) => {
+            return m.userId !== userId &&
+                (m.membership === "invite" || m.membership === "join");
+        });
+        // make sure members have stable order
+        otherMembers.sort((a, b) => a.userId.localeCompare(b.userId));
+        // only 5 first members, immitate _summaryHeroes
+        otherMembers = otherMembers.slice(0, 5);
+        otherNames = otherMembers.map((m) => m.name);
     }
 
+    if (inviteJoinCount) {
+        return memberNamesToRoomName(otherNames, inviteJoinCount);
+    }
 
-    if (otherMembers.length === 0) {
-        const leftMembers = utils.filter(room.currentState.getMembers(), function(m) {
-            return m.userId !== userId && m.membership === "leave";
-        });
-        if (allMembers.length === 1) {
-            // self-chat, peeked room with 1 participant,
-            // or inbound invite, or outbound 3PID invite.
-            if (allMembers[0].userId === userId) {
-                const thirdPartyInvites =
-                    room.currentState.getStateEvents("m.room.third_party_invite");
-                if (thirdPartyInvites && thirdPartyInvites.length > 0) {
-                    let name = "Inviting " +
-                               thirdPartyInvites[0].getContent().display_name;
-                    if (thirdPartyInvites.length > 1) {
-                        if (thirdPartyInvites.length == 2) {
-                            name += " and " +
-                                    thirdPartyInvites[1].getContent().display_name;
-                        } else {
-                            name += " and " +
-                                    thirdPartyInvites.length + " others";
-                        }
-                    }
-                    return name;
-                } else if (leftMembers.length === 1) {
-                    // if it was a chat with one person who's now left, it's still
-                    // notionally a chat with them
-                    return leftMembers[0].name;
-                } else {
-                    return "Empty room";
-                }
-            } else {
-                return allMembers[0].name;
-            }
-        } else {
-            // there really isn't anyone in this room...
-            return "Empty room";
+    const myMembership = room.getMyMembership();
+    // if I have created a room and invited people throuh
+    // 3rd party invites
+    if (myMembership == 'join') {
+        const thirdPartyInvites =
+            room.currentState.getStateEvents("m.room.third_party_invite");
+
+        if (thirdPartyInvites && thirdPartyInvites.length) {
+            const thirdPartyNames = thirdPartyInvites.map((i) => {
+                return i.getContent().display_name;
+            });
+
+            return `Inviting ${memberNamesToRoomName(thirdPartyNames)}`;
         }
-    } else if (otherMembers.length === 1) {
-        return otherMembers[0].name;
-    } else if (otherMembers.length === 2) {
-        return (
-            otherMembers[0].name + " and " + otherMembers[1].name
-        );
+    }
+    // let's try to figure out who was here before
+    let leftNames = otherNames;
+    // if we didn't have heroes, try finding them in the room state
+    if(!leftNames.length) {
+        leftNames = room.currentState.getMembers().filter((m) => {
+            return m.userId !== userId &&
+                m.membership !== "invite" &&
+                m.membership !== "join";
+        }).map((m) => m.name);
+    }
+    if(leftNames.length) {
+        return `Empty room (was ${memberNamesToRoomName(leftNames)})`;
     } else {
-        return (
-            otherMembers[0].name + " and " + (otherMembers.length - 1) + " others"
-        );
+        return "Empty room";
+    }
+}
+
+function memberNamesToRoomName(names, count = (names.length + 1)) {
+    const countWithoutMe = count - 1;
+    if (!names.length) {
+       return "Empty room";
+    } else if (names.length === 1 && countWithoutMe <= 1) {
+        return names[0];
+    } else if (names.length === 2 && countWithoutMe <= 2) {
+        return `${names[0]} and ${names[1]}`;
+    } else {
+        const plural = countWithoutMe > 1;
+        if (plural) {
+            return `${names[0]} and ${countWithoutMe} others`;
+        } else {
+            return `${names[0]} and 1 other`;
+        }
     }
 }
 

src/randomstring.js

@@ -0,0 +1,26 @@
+/*
+Copyright 2018 New Vector Ltd
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+export function randomString(len) {
+    let ret = "";
+    const chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
+
+    for (let i = 0; i < len; ++i) {
+        ret += chars.charAt(Math.floor(Math.random() * chars.length));
+    }
+
+    return ret;
+}

src/store/indexeddb-local-backend.js

@@ -19,7 +19,7 @@ import Promise from 'bluebird';
 import SyncAccumulator from "../sync-accumulator";
 import utils from "../utils";
 
-const VERSION = 1;
+const VERSION = 3;
 
 function createDatabase(db) {
     // Make user store, clobber based on user ID. (userId property of User objects)
@@ -33,6 +33,20 @@ function createDatabase(db) {
     db.createObjectStore("sync", { keyPath: ["clobber"] });
 }
 
+function upgradeSchemaV2(db) {
+    const oobMembersStore = db.createObjectStore(
+        "oob_membership_events", {
+            keyPath: ["room_id", "state_key"],
+        });
+    oobMembersStore.createIndex("room", "room_id");
+}
+
+function upgradeSchemaV3(db) {
+    db.createObjectStore("client_options",
+        { keyPath: ["clobber"]});
+}
+
+
 /**
  * Helper method to collect results from a Cursor and promiseify it.
  * @param {ObjectStore|Index} store The store to perform openCursor on.
@@ -63,28 +77,39 @@ function selectQuery(store, keyRange, resultMapper) {
     });
 }
 
-function promiseifyTxn(txn) {
+function txnAsPromise(txn) {
     return new Promise((resolve, reject) => {
         txn.oncomplete = function(event) {
             resolve(event);
         };
         txn.onerror = function(event) {
-            reject(event);
+            reject(event.target.error);
         };
     });
 }
 
-function promiseifyRequest(req) {
+function reqAsEventPromise(req) {
     return new Promise((resolve, reject) => {
         req.onsuccess = function(event) {
             resolve(event);
         };
         req.onerror = function(event) {
-            reject(event);
+            reject(event.target.error);
         };
     });
 }
 
+function reqAsPromise(req) {
+    return new Promise((resolve, reject) => {
+        req.onsuccess = () => resolve(req);
+        req.onerror = (err) => reject(err);
+    });
+}
+
+function reqAsCursorPromise(req) {
+    return reqAsEventPromise(req).then((event) => event.target.result);
+}
+
 /**
  * Does the actual reading from and writing to the indexeddb
  *
@@ -104,6 +129,7 @@ const LocalIndexedDBStoreBackend = function LocalIndexedDBStoreBackend(
     this.db = null;
     this._disconnected = true;
     this._syncAccumulator = new SyncAccumulator();
+    this._isNewlyCreated = false;
 };
 
 
@@ -134,8 +160,15 @@ LocalIndexedDBStoreBackend.prototype = {
                 `LocalIndexedDBStoreBackend.connect: upgrading from ${oldVersion}`,
             );
             if (oldVersion < 1) { // The database did not previously exist.
+                this._isNewlyCreated = true;
                 createDatabase(db);
             }
+            if (oldVersion < 2) {
+                upgradeSchemaV2(db);
+            }
+            if (oldVersion < 3) {
+                upgradeSchemaV3(db);
+            }
             // Expand as needed.
         };
 
@@ -148,7 +181,7 @@ LocalIndexedDBStoreBackend.prototype = {
         console.log(
             `LocalIndexedDBStoreBackend.connect: awaiting connection...`,
         );
-        return promiseifyRequest(req).then((ev) => {
+        return reqAsEventPromise(req).then((ev) => {
             console.log(
                 `LocalIndexedDBStoreBackend.connect: connected`,
             );
@@ -163,6 +196,10 @@ LocalIndexedDBStoreBackend.prototype = {
             return this._init();
         });
     },
+    /** @return {bool} whether or not the database was newly created in this session. */
+    isNewlyCreated: function() {
+        return Promise.resolve(this._isNewlyCreated);
+    },
 
     /**
      * Having connected, load initial data from the database and prepare for use
@@ -187,6 +224,124 @@ LocalIndexedDBStoreBackend.prototype = {
         });
     },
 
+    /**
+     * Returns the out-of-band membership events for this room that
+     * were previously loaded.
+     * @param {string} roomId
+     * @returns {Promise<event[]>} the events, potentially an empty array if OOB loading didn't yield any new members
+     * @returns {null} in case the members for this room haven't been stored yet
+     */
+    getOutOfBandMembers: function(roomId) {
+        return new Promise((resolve, reject) =>{
+            const tx = this.db.transaction(["oob_membership_events"], "readonly");
+            const store = tx.objectStore("oob_membership_events");
+            const roomIndex = store.index("room");
+            const range = IDBKeyRange.only(roomId);
+            const request = roomIndex.openCursor(range);
+
+            const membershipEvents = [];
+            // did we encounter the oob_written marker object
+            // amongst the results? That means OOB member
+            // loading already happened for this room
+            // but there were no members to persist as they
+            // were all known already
+            let oobWritten = false;
+
+            request.onsuccess = (event) => {
+                const cursor = event.target.result;
+                if (!cursor) {
+                    // Unknown room
+                    if (!membershipEvents.length && !oobWritten) {
+                        return resolve(null);
+                    }
+                    return resolve(membershipEvents);
+                }
+                const record = cursor.value;
+                if (record.oob_written) {
+                    oobWritten = true;
+                } else {
+                    membershipEvents.push(record);
+                }
+                cursor.continue();
+            };
+            request.onerror = (err) => {
+                reject(err);
+            };
+        }).then((events) => {
+            console.log(`LL: got ${events && events.length}` +
+                ` membershipEvents from storage for room ${roomId} ...`);
+            return events;
+        });
+    },
+
+    /**
+     * Stores the out-of-band membership events for this room. Note that
+     * it still makes sense to store an empty array as the OOB status for the room is
+     * marked as fetched, and getOutOfBandMembers will return an empty array instead of null
+     * @param {string} roomId
+     * @param {event[]} membershipEvents the membership events to store
+     */
+    setOutOfBandMembers: async function(roomId, membershipEvents) {
+        console.log(`LL: backend about to store ${membershipEvents.length}` +
+            ` members for ${roomId}`);
+        const tx = this.db.transaction(["oob_membership_events"], "readwrite");
+        const store = tx.objectStore("oob_membership_events");
+        membershipEvents.forEach((e) => {
+            store.put(e);
+        });
+        // aside from all the events, we also write a marker object to the store
+        // to mark the fact that OOB members have been written for this room.
+        // It's possible that 0 members need to be written as all where previously know
+        // but we still need to know whether to return null or [] from getOutOfBandMembers
+        // where null means out of band members haven't been stored yet for this room
+        const markerObject = {
+            room_id: roomId,
+            oob_written: true,
+            state_key: 0,
+        };
+        store.put(markerObject);
+        await txnAsPromise(tx);
+        console.log(`LL: backend done storing for ${roomId}!`);
+    },
+
+    clearOutOfBandMembers: async function(roomId) {
+        // the approach to delete all members for a room
+        // is to get the min and max state key from the index
+        // for that room, and then delete between those
+        // keys in the store.
+        // this should be way faster than deleting every member
+        // individually for a large room.
+        const readTx = this.db.transaction(
+            ["oob_membership_events"],
+            "readonly");
+        const store = readTx.objectStore("oob_membership_events");
+        const roomIndex = store.index("room");
+        const roomRange = IDBKeyRange.only(roomId);
+
+        const minStateKeyProm = reqAsCursorPromise(
+                roomIndex.openKeyCursor(roomRange, "next"),
+            ).then((cursor) => cursor && cursor.primaryKey[1]);
+        const maxStateKeyProm = reqAsCursorPromise(
+                roomIndex.openKeyCursor(roomRange, "prev"),
+            ).then((cursor) => cursor && cursor.primaryKey[1]);
+        const [minStateKey, maxStateKey] = await Promise.all(
+            [minStateKeyProm, maxStateKeyProm]);
+
+        const writeTx = this.db.transaction(
+            ["oob_membership_events"],
+            "readwrite");
+        const writeStore = writeTx.objectStore("oob_membership_events");
+        const membersKeyRange = IDBKeyRange.bound(
+            [roomId, minStateKey],
+            [roomId, maxStateKey],
+        );
+
+        console.log(`LL: Deleting all users + marker in storage for ` +
+            `room ${roomId}, with key range:`,
+            [roomId, minStateKey], [roomId, maxStateKey]);
+        await reqAsPromise(writeStore.delete(membersKeyRange));
+    },
+
     /**
      * Clear the entire database. This should be used when logging out of a client
      * to prevent mixing data between accounts.
@@ -284,7 +439,7 @@ LocalIndexedDBStoreBackend.prototype = {
                 roomsData: roomsData,
                 groupsData: groupsData,
             }); // put == UPSERT
-            return promiseifyTxn(txn);
+            return txnAsPromise(txn);
         });
     },
 
@@ -301,7 +456,7 @@ LocalIndexedDBStoreBackend.prototype = {
             for (let i = 0; i < accountData.length; i++) {
                 store.put(accountData[i]); // put == UPSERT
             }
-            return promiseifyTxn(txn);
+            return txnAsPromise(txn);
         });
     },
 
@@ -323,7 +478,7 @@ LocalIndexedDBStoreBackend.prototype = {
                     event: tuple[1],
                 }); // put == UPSERT
             }
-            return promiseifyTxn(txn);
+            return txnAsPromise(txn);
         });
     },
 
@@ -389,6 +544,28 @@ LocalIndexedDBStoreBackend.prototype = {
             });
         });
     },
+
+    getClientOptions: function() {
+        return Promise.resolve().then(() => {
+            const txn = this.db.transaction(["client_options"], "readonly");
+            const store = txn.objectStore("client_options");
+            return selectQuery(store, undefined, (cursor) => {
+                if (cursor.value && cursor.value && cursor.value.options) {
+                    return cursor.value.options;
+                }
+            }).then((results) => results[0]);
+        });
+    },
+
+    storeClientOptions: async function(options) {
+        const txn = this.db.transaction(["client_options"], "readwrite");
+        const store = txn.objectStore("client_options");
+        store.put({
+            clobber: "-", // constant key so will always clobber
+            options: options,
+        }); // put == UPSERT
+        await txnAsPromise(txn);
+    },
 };
 
 export default LocalIndexedDBStoreBackend;

src/store/indexeddb-remote-backend.js

@@ -65,7 +65,10 @@ RemoteIndexedDBStoreBackend.prototype = {
     clearDatabase: function() {
         return this._ensureStarted().then(() => this._doCmd('clearDatabase'));
     },
-
+    /** @return {Promise<bool>} whether or not the database was newly created in this session. */
+    isNewlyCreated: function() {
+        return this._doCmd('isNewlyCreated');
+    },
     /**
      * @return {Promise} Resolves with a sync response to restore the
      * client state to where it was at the last save, or null if there
@@ -87,6 +90,40 @@ RemoteIndexedDBStoreBackend.prototype = {
         return this._doCmd('syncToDatabase', [users]);
     },
 
+    /**
+     * Returns the out-of-band membership events for this room that
+     * were previously loaded.
+     * @param {string} roomId
+     * @returns {event[]} the events, potentially an empty array if OOB loading didn't yield any new members
+     * @returns {null} in case the members for this room haven't been stored yet
+     */
+    getOutOfBandMembers: function(roomId) {
+        return this._doCmd('getOutOfBandMembers', [roomId]);
+    },
+
+    /**
+     * Stores the out-of-band membership events for this room. Note that
+     * it still makes sense to store an empty array as the OOB status for the room is
+     * marked as fetched, and getOutOfBandMembers will return an empty array instead of null
+     * @param {string} roomId
+     * @param {event[]} membershipEvents the membership events to store
+     * @returns {Promise} when all members have been stored
+     */
+    setOutOfBandMembers: function(roomId, membershipEvents) {
+        return this._doCmd('setOutOfBandMembers', [roomId, membershipEvents]);
+    },
+
+    clearOutOfBandMembers: function(roomId) {
+        return this._doCmd('clearOutOfBandMembers', [roomId]);
+    },
+
+    getClientOptions: function() {
+        return this._doCmd('getClientOptions');
+    },
+
+    storeClientOptions: function(options) {
+        return this._doCmd('storeClientOptions', [options]);
+    },
 
     /**
      * Load all user presence events from the database. This is not cached.
@@ -147,7 +184,9 @@ RemoteIndexedDBStoreBackend.prototype = {
             if (msg.command == 'cmd_success') {
                 def.resolve(msg.result);
             } else {
-                def.reject(msg.error);
+                const error = new Error(msg.error.message);
+                error.name = msg.error.name;
+                def.reject(error);
             }
         } else {
             console.warn("Unrecognised message from worker: " + msg);

src/store/indexeddb-store-worker.js

@@ -67,6 +67,9 @@ class IndexedDBStoreWorker {
             case 'connect':
                 prom = this.backend.connect();
                 break;
+            case 'isNewlyCreated':
+                prom = this.backend.isNewlyCreated();
+                break;
             case 'clearDatabase':
                 prom = this.backend.clearDatabase().then((result) => {
                     // This returns special classes which can't be cloned
@@ -92,10 +95,25 @@ class IndexedDBStoreWorker {
             case 'getNextBatchToken':
                 prom = this.backend.getNextBatchToken();
                 break;
+            case 'getOutOfBandMembers':
+                prom = this.backend.getOutOfBandMembers(msg.args[0]);
+                break;
+            case 'clearOutOfBandMembers':
+                prom = this.backend.clearOutOfBandMembers(msg.args[0]);
+                break;
+            case 'setOutOfBandMembers':
+                prom = this.backend.setOutOfBandMembers(msg.args[0], msg.args[1]);
+                break;
+            case 'getClientOptions':
+                prom = this.backend.getClientOptions();
+                break;
+            case 'storeClientOptions':
+                prom = this.backend.storeClientOptions(msg.args[0]);
+                break;
         }
 
         if (prom === undefined) {
-            postMessage({
+            this.postMessage({
                 command: 'cmd_fail',
                 seq: msg.seq,
                 // Can't be an Error because they're not structured cloneable
@@ -117,7 +135,10 @@ class IndexedDBStoreWorker {
                 command: 'cmd_fail',
                 seq: msg.seq,
                 // Just send a string because Error objects aren't cloneable
-                error: "Error running command",
+                error: {
+                    message: err.message,
+                    name: err.name,
+                },
             });
         });
     }

src/store/indexeddb.js

@@ -146,6 +146,11 @@ IndexedDBStore.prototype.getSavedSync = function() {
     return this.backend.getSavedSync();
 };
 
+/** @return {Promise<bool>} whether or not the database was newly created in this session. */
+IndexedDBStore.prototype.isNewlyCreated = function() {
+    return this.backend.isNewlyCreated();
+};
+
 /**
  * @return {Promise} If there is a saved sync, the nextBatch token
  * for this sync, otherwise null.
@@ -219,4 +224,39 @@ IndexedDBStore.prototype.setSyncData = function(syncData) {
     return this.backend.setSyncData(syncData);
 };
 
+/**
+ * Returns the out-of-band membership events for this room that
+ * were previously loaded.
+ * @param {string} roomId
+ * @returns {event[]} the events, potentially an empty array if OOB loading didn't yield any new members
+ * @returns {null} in case the members for this room haven't been stored yet
+ */
+IndexedDBStore.prototype.getOutOfBandMembers = function(roomId) {
+    return this.backend.getOutOfBandMembers(roomId);
+};
+
+/**
+ * Stores the out-of-band membership events for this room. Note that
+ * it still makes sense to store an empty array as the OOB status for the room is
+ * marked as fetched, and getOutOfBandMembers will return an empty array instead of null
+ * @param {string} roomId
+ * @param {event[]} membershipEvents the membership events to store
+ * @returns {Promise} when all members have been stored
+ */
+IndexedDBStore.prototype.setOutOfBandMembers = function(roomId, membershipEvents) {
+    return this.backend.setOutOfBandMembers(roomId, membershipEvents);
+};
+
+IndexedDBStore.prototype.clearOutOfBandMembers = function(roomId) {
+    return this.backend.clearOutOfBandMembers(roomId);
+};
+
+IndexedDBStore.prototype.getClientOptions = function() {
+    return this.backend.getClientOptions();
+};
+
+IndexedDBStore.prototype.storeClientOptions = function(options) {
+    return this.backend.storeClientOptions(options);
+};
+
 module.exports.IndexedDBStore = IndexedDBStore;

src/store/memory.js

@@ -52,6 +52,10 @@ module.exports.MatrixInMemoryStore = function MatrixInMemoryStore(opts) {
         // type : content
     };
     this.localStorage = opts.localStorage;
+    this._oobMembers = {
+        // roomId: [member events]
+    };
+    this._clientOptions = {};
 };
 
 module.exports.MatrixInMemoryStore.prototype = {
@@ -64,6 +68,10 @@ module.exports.MatrixInMemoryStore.prototype = {
         return this.syncToken;
     },
 
+    /** @return {Promise<bool>} whether or not the database was newly created in this session. */
+    isNewlyCreated: function() {
+        return Promise.resolve(true);
+    },
 
     /**
      * Set the token to stream from.
@@ -377,4 +385,35 @@ module.exports.MatrixInMemoryStore.prototype = {
         };
         return Promise.resolve();
     },
+    /**
+     * Returns the out-of-band membership events for this room that
+     * were previously loaded.
+     * @param {string} roomId
+     * @returns {event[]} the events, potentially an empty array if OOB loading didn't yield any new members
+     * @returns {null} in case the members for this room haven't been stored yet
+     */
+    getOutOfBandMembers: function(roomId) {
+        return Promise.resolve(this._oobMembers[roomId] || null);
+    },
+    /**
+     * Stores the out-of-band membership events for this room. Note that
+     * it still makes sense to store an empty array as the OOB status for the room is
+     * marked as fetched, and getOutOfBandMembers will return an empty array instead of null
+     * @param {string} roomId
+     * @param {event[]} membershipEvents the membership events to store
+     * @returns {Promise} when all members have been stored
+     */
+    setOutOfBandMembers: function(roomId, membershipEvents) {
+        this._oobMembers[roomId] = membershipEvents;
+        return Promise.resolve();
+    },
+
+    getClientOptions: function() {
+        return Promise.resolve(this._clientOptions);
+    },
+
+    storeClientOptions: function(options) {
+        this._clientOptions = Object.assign({}, options);
+        return Promise.resolve();
+    },
 };

src/store/stub.js

@@ -32,6 +32,11 @@ function StubStore() {
 
 StubStore.prototype = {
 
+    /** @return {Promise<bool>} whether or not the database was newly created in this session. */
+    isNewlyCreated: function() {
+        return Promise.resolve(true);
+    },
+
     /**
      * Get the sync token.
      * @return {string}
@@ -264,6 +269,26 @@ StubStore.prototype = {
     deleteAllData: function() {
         return Promise.resolve();
     },
+
+    getOutOfBandMembers: function() {
+        return Promise.resolve(null);
+    },
+
+    setOutOfBandMembers: function() {
+        return Promise.resolve();
+    },
+
+    clearOutOfBandMembers: function() {
+        return Promise.resolve();
+    },
+
+    getClientOptions: function() {
+        return Promise.resolve();
+    },
+
+    storeClientOptions: function() {
+        return Promise.resolve();
+    },
 };
 
 /** Stub Store class. */

src/sync-accumulator.js

@@ -63,6 +63,11 @@ class SyncAccumulator {
             //       { event: $event, token: null|token },
             //       ...
             //    ],
+            //    _summary: {
+            //       m.heroes: [ $user_id ],
+            //       m.joined_member_count: $count,
+            //       m.invited_member_count: $count
+            //    },
             //    _accountData: { $event_type: json },
             //    _unreadNotifications: { ... unread_notifications JSON ... },
             //    _readReceipts: { $user_id: { data: $json, eventId: $event_id }}
@@ -242,6 +247,7 @@ class SyncAccumulator {
                 _timeline: [],
                 _accountData: Object.create(null),
                 _unreadNotifications: {},
+                _summary: {},
                 _readReceipts: {},
             };
         }
@@ -258,6 +264,17 @@ class SyncAccumulator {
         if (data.unread_notifications) {
             currentData._unreadNotifications = data.unread_notifications;
         }
+        if (data.summary) {
+            const HEROES_KEY = "m.heroes";
+            const INVITED_COUNT_KEY = "m.invited_member_count";
+            const JOINED_COUNT_KEY = "m.joined_member_count";
+
+            const acc = currentData._summary;
+            const sum = data.summary;
+            acc[HEROES_KEY] = sum[HEROES_KEY] || acc[HEROES_KEY];
+            acc[JOINED_COUNT_KEY] = sum[JOINED_COUNT_KEY] || acc[JOINED_COUNT_KEY];
+            acc[INVITED_COUNT_KEY] = sum[INVITED_COUNT_KEY] || acc[INVITED_COUNT_KEY];
+        }
 
         if (data.ephemeral && data.ephemeral.events) {
             data.ephemeral.events.forEach((e) => {
@@ -428,6 +445,7 @@ class SyncAccumulator {
                     prev_batch: null,
                 },
                 unread_notifications: roomData._unreadNotifications,
+                summary: roomData._summary,
             };
             // Add account data
             Object.keys(roomData._accountData).forEach((evType) => {

src/sync.js

@@ -33,6 +33,8 @@ const utils = require("./utils");
 const Filter = require("./filter");
 const EventTimeline = require("./models/event-timeline");
 
+import {InvalidStoreError} from './errors';
+
 const DEBUG = true;
 
 // /sync requests allow you to set a timeout= but the request may continue
@@ -100,6 +102,7 @@ function SyncApi(client, opts) {
     this._connectionReturnedDefer = null;
     this._notifEvents = []; // accumulator of sync events in the current sync response
     this._failedSyncCount = 0; // Number of consecutive failed /sync requests
+    this._storeIsInvalid = false; // flag set if the store needs to be cleared before we can start
 
     if (client.getNotifTimelineSet()) {
         client.reEmitter.reEmit(client.getNotifTimelineSet(),
@@ -113,7 +116,8 @@ function SyncApi(client, opts) {
  */
 SyncApi.prototype.createRoom = function(roomId) {
     const client = this.client;
-    const room = new Room(roomId, {
+    const room = new Room(roomId, client, client.getUserId(), {
+        lazyLoadMembers: this.opts.lazyLoadMembers,
         pendingEventOrdering: this.opts.pendingEventOrdering,
         timelineSupport: client.timelineSupport,
     });
@@ -122,6 +126,7 @@ SyncApi.prototype.createRoom = function(roomId) {
                           "Room.timelineReset",
                           "Room.localEchoUpdated",
                           "Room.accountData",
+                          "Room.myMembership",
                          ]);
     this._registerStateListeners(room);
     return room;
@@ -232,7 +237,7 @@ SyncApi.prototype.syncLeftRooms = function() {
 
             self._processRoomEvents(room, stateEvents, timelineEvents);
 
-            room.recalculate(client.credentials.userId);
+            room.recalculate();
             client.store.storeRoom(room);
             client.emit("Room", room);
 
@@ -303,7 +308,7 @@ SyncApi.prototype.peek = function(roomId) {
         peekRoom.currentState.setStateEvents(stateEvents);
 
         self._resolveInvites(peekRoom);
-        peekRoom.recalculate(self.client.credentials.userId);
+        peekRoom.recalculate();
 
         // roll backwards to diverge old state. addEventsToTimeline
         // will overwrite the pagination token, so make sure it overwrites
@@ -420,6 +425,26 @@ SyncApi.prototype.recoverFromSyncStartupError = async function(savedSyncPromise,
     await keepaliveProm;
 };
 
+/**
+ * Is the lazy loading option different than in previous session?
+ * @param {bool} lazyLoadMembers current options for lazy loading
+ * @return {bool} whether or not the option has changed compared to the previous session */
+SyncApi.prototype._wasLazyLoadingToggled = async function(lazyLoadMembers) {
+    lazyLoadMembers = !!lazyLoadMembers;
+    // assume it was turned off before
+    // if we don't know any better
+    let lazyLoadMembersBefore = false;
+    const isStoreNewlyCreated = await this.client.store.isNewlyCreated();
+    if (!isStoreNewlyCreated) {
+        const prevClientOptions = await this.client.store.getClientOptions();
+        if (prevClientOptions) {
+            lazyLoadMembersBefore = !!prevClientOptions.lazyLoadMembers;
+        }
+        return lazyLoadMembersBefore !== lazyLoadMembers;
+    }
+    return false;
+};
+
 /**
  * Main entry point
  */
@@ -442,6 +467,8 @@ SyncApi.prototype.sync = function() {
     //   1) We need to get push rules so we can check if events should bing as we get
     //      them from /sync.
     //   2) We need to get/create a filter which we can use for /sync.
+    //   3) We need to check the lazy loading option matches what was used in the
+    //       stored sync. If it doesn't, we can't use the stored sync.
 
     async function getPushRules() {
         try {
@@ -456,9 +483,47 @@ SyncApi.prototype.sync = function() {
             getPushRules();
             return;
         }
-        getFilter(); // Now get the filter and start syncing
+        checkLazyLoadStatus(); // advance to the next stage
     }
 
+    const checkLazyLoadStatus = async () => {
+        if (this.opts.lazyLoadMembers && client.isGuest()) {
+            this.opts.lazyLoadMembers = false;
+        }
+        if (this.opts.lazyLoadMembers) {
+            const supported = await client.doesServerSupportLazyLoading();
+            if (supported) {
+                this.opts.filter = await client.createFilter(
+                    Filter.LAZY_LOADING_SYNC_FILTER,
+                );
+            } else {
+                console.log("LL: lazy loading requested but not supported " +
+                    "by server, so disabling");
+                this.opts.lazyLoadMembers = false;
+            }
+        }
+        // need to vape the store when enabling LL and wasn't enabled before
+        const shouldClear = await this._wasLazyLoadingToggled(this.opts.lazyLoadMembers);
+        if (shouldClear) {
+            this._storeIsInvalid = true;
+            const reason = InvalidStoreError.TOGGLED_LAZY_LOADING;
+            const error = new InvalidStoreError(reason, !!this.opts.lazyLoadMembers);
+            this._updateSyncState("ERROR", { error });
+            // bail out of the sync loop now: the app needs to respond to this error.
+            // we leave the state as 'ERROR' which isn't great since this normally means
+            // we're retrying. The client must be stopped before clearing the stores anyway
+            // so the app should stop the client, clear the store and start it again.
+            console.warn("InvalidStoreError: store is not usable: stopping sync.");
+            return;
+        }
+        if (this.opts.lazyLoadMembers && this.opts.crypto) {
+            this.opts.crypto.enableLazyLoading();
+        }
+        await this.client._storeClientOptions();
+
+        getFilter(); // Now get the filter and start syncing
+    };
+
     async function getFilter() {
         let filter;
         if (self.opts.filter) {
@@ -586,7 +651,12 @@ SyncApi.prototype._syncFromCache = async function(savedSync) {
         console.error("Error processing cached sync", e.stack || e);
     }
 
-    this._updateSyncState("PREPARED", syncEventData);
+    // Don't emit a prepared if we've bailed because the store is invalid:
+    // in this case the client will not be usable until stopped & restarted
+    // so this would be useless and misleading.
+    if (!this._storeIsInvalid) {
+        this._updateSyncState("PREPARED", syncEventData);
+    }
 };
 
 /**
@@ -778,7 +848,18 @@ SyncApi.prototype._onSyncError = function(err, syncOptions) {
     // erroneous. We set the state to 'reconnecting'
     // instead, so that clients can observe this state
     // if they wish.
-    this._startKeepAlives().then(() => {
+    this._startKeepAlives().then((connDidFail) => {
+        // Only emit CATCHUP if we detected a connectivity error: if we didn't,
+        // it's quite likely the sync will fail again for the same reason and we
+        // want to stay in ERROR rather than keep flip-flopping between ERROR
+        // and CATCHUP.
+        if (connDidFail && this.getSyncState() === 'ERROR') {
+            this._updateSyncState("CATCHUP", {
+                oldSyncToken: null,
+                nextSyncToken: null,
+                catchingUp: true,
+            });
+        }
         this._sync(syncOptions);
     });
 
@@ -823,6 +904,11 @@ SyncApi.prototype._processSyncResponse = async function(
     //          state: { events: [] },
     //          timeline: { events: [], prev_batch: $token, limited: true },
     //          ephemeral: { events: [] },
+    //          summary: {
+    //             m.heroes: [ $user_id ],
+    //             m.joined_member_count: $count,
+    //             m.invited_member_count: $count
+    //          },
     //          account_data: { events: [] },
     //          unread_notifications: {
     //              highlight_count: 0,
@@ -961,9 +1047,11 @@ SyncApi.prototype._processSyncResponse = async function(
         const room = inviteObj.room;
         const stateEvents =
             self._mapSyncEventsFormat(inviteObj.invite_state, room);
+
+        room.updateMyMembership("invite");
         self._processRoomEvents(room, stateEvents);
         if (inviteObj.isBrandNewRoom) {
-            room.recalculate(client.credentials.userId);
+            room.recalculate();
             client.store.storeRoom(room);
             client.emit("Room", room);
         }
@@ -990,6 +1078,8 @@ SyncApi.prototype._processSyncResponse = async function(
             );
         }
 
+        room.updateMyMembership("join");
+
         joinObj.timeline = joinObj.timeline || {};
 
         if (joinObj.isBrandNewRoom) {
@@ -1054,6 +1144,13 @@ SyncApi.prototype._processSyncResponse = async function(
 
         self._processRoomEvents(room, stateEvents, timelineEvents);
 
+        // set summary after processing events,
+        // because it will trigger a name calculation
+        // which needs the room state to be up to date
+        if (joinObj.summary) {
+            room.setSummary(joinObj.summary);
+        }
+
         // XXX: should we be adding ephemeralEvents to the timeline?
         // It feels like that for symmetry with room.addAccountData()
         // there should be a room.addEphemeralEvents() or similar.
@@ -1062,7 +1159,7 @@ SyncApi.prototype._processSyncResponse = async function(
         // we deliberately don't add accountData to the timeline
         room.addAccountData(accountDataEvents);
 
-        room.recalculate(client.credentials.userId);
+        room.recalculate();
         if (joinObj.isBrandNewRoom) {
             client.store.storeRoom(room);
             client.emit("Room", room);
@@ -1097,10 +1194,12 @@ SyncApi.prototype._processSyncResponse = async function(
         const accountDataEvents =
             self._mapSyncEventsFormat(leaveObj.account_data);
 
+        room.updateMyMembership("leave");
+
         self._processRoomEvents(room, stateEvents, timelineEvents);
         room.addAccountData(accountDataEvents);
 
-        room.recalculate(client.credentials.userId);
+        room.recalculate();
         if (leaveObj.isBrandNewRoom) {
             client.store.storeRoom(room);
             client.emit("Room", room);
@@ -1189,13 +1288,16 @@ SyncApi.prototype._startKeepAlives = function(delay) {
  *
  * On failure, schedules a call back to itself. On success, resolves
  * this._connectionReturnedDefer.
+ *
+ * @param {bool} connDidFail True if a connectivity failure has been detected. Optional.
  */
-SyncApi.prototype._pokeKeepAlive = function() {
+SyncApi.prototype._pokeKeepAlive = function(connDidFail) {
+    if (connDidFail === undefined) connDidFail = false;
     const self = this;
     function success() {
         clearTimeout(self._keepAliveTimer);
         if (self._connectionReturnedDefer) {
-            self._connectionReturnedDefer.resolve();
+            self._connectionReturnedDefer.resolve(connDidFail);
             self._connectionReturnedDefer = null;
         }
     }
@@ -1212,7 +1314,7 @@ SyncApi.prototype._pokeKeepAlive = function() {
     ).done(function() {
         success();
     }, function(err) {
-        if (err.httpStatus == 400) {
+        if (err.httpStatus == 400 || err.httpStatus == 404) {
             // treat this as a success because the server probably just doesn't
             // support /versions: point is, we're getting a response.
             // We wait a short time though, just in case somehow the server
@@ -1220,8 +1322,9 @@ SyncApi.prototype._pokeKeepAlive = function() {
             // responses fail, this will mean we don't hammer in a loop.
             self._keepAliveTimer = setTimeout(success, 2000);
         } else {
+            connDidFail = true;
             self._keepAliveTimer = setTimeout(
-                self._pokeKeepAlive.bind(self),
+                self._pokeKeepAlive.bind(self, connDidFail),
                 5000 + Math.floor(Math.random() * 5000),
             );
             // A keepalive has failed, so we emit the
@@ -1229,7 +1332,7 @@ SyncApi.prototype._pokeKeepAlive = function() {
             // first failure).
             // Note we do this after setting the timer:
             // this lets the unit tests advance the mock
-            // clock when the get the error.
+            // clock when they get the error.
             self._updateSyncState("ERROR", { error: err });
         }
     });
@@ -1390,7 +1493,7 @@ SyncApi.prototype._processRoomEvents = function(room, stateEventList,
     // a recalculation (like m.room.name) we won't recalculate until we've
     // finished adding all the events, which will cause the notification to have
     // the old room name rather than the new one.
-    room.recalculate(this.client.credentials.userId);
+    room.recalculate();
 
     // If the timeline wasn't empty, we process the state events here: they're
     // defined as updates to the state before the start of the timeline, so this

travis.sh

@@ -5,7 +5,7 @@ set -ex
 npm run lint
 
 # install Olm so that we can run the crypto tests.
-npm install https://matrix.org/packages/npm/olm/olm-2.2.2.tgz
+npm install https://matrix.org/packages/npm/olm/olm-3.0.0.tgz
 
 npm run test