v1.0.3

Prepare changelog for v1.0.3
Merge pull request #874 from matrix-org/travis/1.0.6-fire/872
2019-04-01 13:31:45 +01:00 · 2019-04-01 13:31:44 +01:00 · 2019-04-01 09:26:39 +01:00 · 2019-03-29 16:19:54 -06:00 · 2019-03-27 17:29:28 +00:00 · 2019-03-27 17:29:27 +00:00
43 changed files with 6094 additions and 7406 deletions
@@ -0,0 +1,24 @@
+steps:
+  - label: ":eslint: Lint"
+    command:
+      - "yarn install"
+      - "yarn lint"
+    plugins:
+      - docker#v3.0.1:
+          image: "node:10"
+
+  - label: ":karma: Tests"
+    command:
+      - "yarn install"
+      - "yarn test"
+    plugins:
+      - docker#v3.0.1:
+          image: "node:10"
+
+  - label: "📃 Docs"
+    command:
+      - "yarn install"
+      - "yarn gendoc"
+    plugins:
+      - docker#v3.0.1:
+          image: "node:10"
@@ -2,6 +2,9 @@
 /.jsdoc

 node_modules
+/.npmrc
+/*.log
+package-lock.json
 .lock-wscript
 build/Release
 coverage
@@ -12,6 +15,6 @@ reports
 /lib
 /specbuild

-# version file and tarball created by 'npm pack'
+# version file and tarball created by `npm pack` / `yarn pack`
 /git-revision.txt
 /matrix-js-sdk-*.tgz
@@ -1,5 +0,0 @@
-language: node_js
-node_js:
-    - "10.11.0"
-script:
-    - ./travis.sh
@@ -1,3 +1,93 @@
+Changes in [1.0.3](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.0.3) (2019-04-01)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.3-rc.1...v1.0.3)
+
+ * Add existence check to local storage based crypto store
+   [\#874](https://github.com/matrix-org/matrix-js-sdk/pull/874)
+
+Changes in [1.0.3-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.0.3-rc.1) (2019-03-27)
+==========================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.2...v1.0.3-rc.1)
+
+ * Add IndexedDB existence checks
+   [\#871](https://github.com/matrix-org/matrix-js-sdk/pull/871)
+ * Emit sync errors for capturing by clients
+   [\#869](https://github.com/matrix-org/matrix-js-sdk/pull/869)
+ * Add functions for getting room upgrade history and leaving those rooms
+   [\#868](https://github.com/matrix-org/matrix-js-sdk/pull/868)
+ * Clarify the meaning of 'real name' for contribution
+   [\#867](https://github.com/matrix-org/matrix-js-sdk/pull/867)
+ * Remove `sessionStore` to `cryptoStore` migration path
+   [\#865](https://github.com/matrix-org/matrix-js-sdk/pull/865)
+ * Add debugging for spurious room version warnings
+   [\#866](https://github.com/matrix-org/matrix-js-sdk/pull/866)
+ * Add investigation notes for browser storage
+   [\#864](https://github.com/matrix-org/matrix-js-sdk/pull/864)
+ * make sure resolve object is defined before calling it
+   [\#862](https://github.com/matrix-org/matrix-js-sdk/pull/862)
+ * Rename `MatrixInMemoryStore` to `MemoryStore`
+   [\#861](https://github.com/matrix-org/matrix-js-sdk/pull/861)
+ * Use Buildkite for CI
+   [\#859](https://github.com/matrix-org/matrix-js-sdk/pull/859)
+ * only create one session at a time per device
+   [\#857](https://github.com/matrix-org/matrix-js-sdk/pull/857)
+
+Changes in [1.0.2](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.0.2) (2019-03-18)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.2-rc.1...v1.0.2)
+
+ * No changes since rc.1
+
+Changes in [1.0.2-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.0.2-rc.1) (2019-03-13)
+==========================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.1...v1.0.2-rc.1)
+
+ * Use modern Yarn version on Travis CI
+   [\#858](https://github.com/matrix-org/matrix-js-sdk/pull/858)
+ * Switch to `yarn` for dependency management
+   [\#856](https://github.com/matrix-org/matrix-js-sdk/pull/856)
+ * More key request fixes
+   [\#855](https://github.com/matrix-org/matrix-js-sdk/pull/855)
+ * Calculate encrypted notification counts
+   [\#851](https://github.com/matrix-org/matrix-js-sdk/pull/851)
+ * Update dependencies
+   [\#854](https://github.com/matrix-org/matrix-js-sdk/pull/854)
+ * make sure key requests get sent
+   [\#850](https://github.com/matrix-org/matrix-js-sdk/pull/850)
+ * Use 'ideal' rather than 'exact' for deviceid
+   [\#852](https://github.com/matrix-org/matrix-js-sdk/pull/852)
+ * handle partially-shared sessions better
+   [\#848](https://github.com/matrix-org/matrix-js-sdk/pull/848)
+
+Changes in [1.0.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.0.1) (2019-03-06)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.1-rc.2...v1.0.1)
+
+ * No changes since rc.2
+
+Changes in [1.0.1-rc.2](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.0.1-rc.2) (2019-03-05)
+==========================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.1-rc.1...v1.0.1-rc.2)
+
+ * dont swallow txn errors in crypto store
+   [\#853](https://github.com/matrix-org/matrix-js-sdk/pull/853)
+ * Don't swallow txn errors in crypto store
+   [\#849](https://github.com/matrix-org/matrix-js-sdk/pull/849)
+
+Changes in [1.0.1-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.0.1-rc.1) (2019-02-28)
+==========================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.0...v1.0.1-rc.1)
+
+ * Fix "e is undefined" masking the original error in MegolmDecryption
+   [\#847](https://github.com/matrix-org/matrix-js-sdk/pull/847)
+
+Changes in [1.0.0](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.0.0) (2019-02-14)
+================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.0-rc.2...v1.0.0)
+
+ * Try again to commit package-lock.json
+   [\#841](https://github.com/matrix-org/matrix-js-sdk/pull/841)
+
 Changes in [1.0.0-rc.2](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v1.0.0-rc.2) (2019-02-14)
 ==========================================================================================================
 [Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v1.0.0-rc.1...v1.0.0-rc.2)
@@ -24,7 +24,7 @@ works. Develop is the unstable branch where all the development actually
 happens: the workflow is that contributors should fork the develop branch to
 make a 'feature' branch for a particular contribution, and then make a pull
 request to merge this back into the matrix.org 'official' develop branch. We
-use github's pull request workflow to review the contribution, and either ask
+use GitHub's pull request workflow to review the contribution, and either ask
 you to make any refinements needed or merge it and make them ourselves. The
 changes will then land on master when we next do a release.

@@ -60,8 +60,8 @@ Sign off
 ~~~~~~~~

 In order to have a concrete record that your contribution is intentional
-and you agree to license it under the same terms as the project's license, we've adopted the
-same lightweight approach that the Linux Kernel
+and you agree to license it under the same terms as the project's license, we've
+adopted the same lightweight approach that the Linux Kernel
 (https://www.kernel.org/doc/Documentation/SubmittingPatches), Docker
 (https://github.com/docker/docker/blob/master/CONTRIBUTING.md), and many other
 projects use: the DCO (Developer Certificate of Origin:
@@ -109,12 +109,16 @@ include the line in your commit or pull request comment::

    Signed-off-by: Your Name <your@email.example.org>

-...using your real name; unfortunately pseudonyms and anonymous contributions
-can't be accepted. Git makes this trivial - just use the -s flag when you do
-``git commit``, having first set ``user.name`` and ``user.email`` git configs
-(which you should have done anyway :)
+We accept contributions under a legally identifiable name, such as your name on
+government documentation or common-law names (names claimed by legitimate usage
+or repute). Unfortunately, we cannot accept anonymous contributions at this
+time.

-If you forgot to sign off your commits before making your pull request and are on git 2.17+ 
-you can mass signoff using rebase::
+Git allows you to add this signoff automatically when using the ``-s`` flag to
+``git commit``, which uses the name and email set in your ``user.name`` and
+``user.email`` git configs.
+
+If you forgot to sign off your commits before making your pull request and are
+on Git 2.17+ you can mass signoff using rebase::

    git rebase --signoff origin/develop
@@ -21,7 +21,11 @@ Please check [the working browser example](examples/browser) for more informatio
 In Node.js
 ----------

-``npm install matrix-js-sdk``
+Ensure you have the latest LTS version of Node.js installed.
+
+Using `yarn` instead of `npm` is recommended. Please see the Yarn [install guide](https://yarnpkg.com/docs/install/) if you do not have it already.
+
+``yarn add matrix-js-sdk``

 ```javascript
  var sdk = require("matrix-js-sdk");
@@ -76,7 +80,7 @@ client.on("Room.timeline", function(event, room, toStartOfTimeline) {
 });
 ```

-By default, the `matrix-js-sdk` client uses the `MatrixInMemoryStore` to store events as they are received. For example to iterate through the currently stored timeline for a room:
+By default, the `matrix-js-sdk` client uses the `MemoryStore` to store events as they are received. For example to iterate through the currently stored timeline for a room:

 ```javascript
 Object.keys(client.store.rooms).forEach((roomId) => {
@@ -283,7 +287,7 @@ This SDK uses JSDoc3 style comments. You can manually build and
 host the API reference from the source files like this:

 ```
-  $ npm run gendoc
+  $ yarn gendoc
  $ cd .jsdoc
  $ python -m SimpleHTTPServer 8005
 ```
@@ -319,15 +323,15 @@ To provide the Olm library in a browser application:
 
 To provide the Olm library in a node.js application:

- * ``npm install https://matrix.org/packages/npm/olm/olm-3.0.0.tgz``
+ * ``yarn add https://matrix.org/packages/npm/olm/olm-3.0.0.tgz``
   (replace the URL with the latest version you want to use from
    https://matrix.org/packages/npm/olm/)
 * ``global.Olm = require('olm');`` *before* loading ``matrix-js-sdk``.

-If you want to package Olm as dependency for your node.js application, you
-can use ``npm install https://matrix.org/packages/npm/olm/olm-3.0.0.tgz
--save-optional`` (if your application also works without e2e crypto enabled)
-or ``--save`` (if it doesn't) to do so.
+If you want to package Olm as dependency for your node.js application, you can
+use ``yarn add https://matrix.org/packages/npm/olm/olm-3.0.0.tgz``. If your
+application also works without e2e crypto enabled, add ``--optional`` to mark it
+as an optional dependency.


 Contributing
@@ -337,7 +341,7 @@ want to use this SDK, skip this section.*

 First, you need to pull in the right build tools:
 ```
- $ npm install
+ $ yarn install
 ```

 Building
@@ -345,20 +349,20 @@ Building

 To build a browser version from scratch when developing::
 ```
- $ npm run build
+ $ yarn build
 ```

 To constantly do builds when files are modified (using ``watchify``)::
 ```
- $ npm run watch
+ $ yarn watch
 ```

 To run tests (Jasmine)::
 ```
- $ npm test
+ $ yarn test
 ```

 To run linting:
 ```
- $ npm run lint
+ $ yarn lint
 ```
@@ -0,0 +1,70 @@
+# Browser Storage Notes
+
+## Overview
+
+Browsers examined: Firefox 67, Chrome 75
+
+The examination below applies to the default, non-persistent storage policy.
+
+## Quota Measurement
+
+Browsers appear to enforce and measure the quota in terms of space on disk, not
+data stored, so you may be able to store more data than the simple sum of all
+input data depending on how compressible your data is.
+
+## Quota Limit
+
+Specs and documentation suggest we should consistently receive
+`QuotaExceededError` when we're near space limits, but the reality is a bit
+blurrier.
+
+When we are low on disk space overall or near the group limit / origin quota:
+
+* Chrome
+    * Log database may fail to start with AbortError
+    * IndexedDB fails to start for crypto: AbortError in connect from
+      indexeddb-store-worker
+    * When near the quota, QuotaExceededError is used more consistently
+* Firefox
+    * The first error will be QuotaExceededError
+    * Future write attempts will fail with various errors when space is low,
+      including nonsense like "InvalidStateError: A mutation operation was
+      attempted on a database that did not allow mutations."
+    * Once you start getting errors, the DB is effectively wedged in read-only
+      mode
+    * Can revive access if you reopen the DB
+
+## Cache Eviction
+
+While the Storage Standard says all storage for an origin group should be
+limited by a single quota, in practice, browsers appear to handle `localStorage`
+separately from the others, so it has a separate quota limit and isn't evicted
+when low on space.
+
+* Chrome, Firefox
+    * IndexedDB for origin deleted
+    * Local Storage remains in place
+
+## Persistent Storage
+
+Storage Standard offers a `navigator.storage.persist` API that can be used to
+request persistent storage that won't be deleted by the browser because of low
+space.
+
+* Chrome
+    * Chrome 75 seems to grant this without any prompt based on [interaction
+      criteria](https://developers.google.com/web/updates/2016/06/persistent-storage)
+* Firefox
+    * Firefox 67 shows a prompt to grant
+    * Reverting persistent seems to require revoking permission _and_ clearing
+      site data
+
+## Storage Estimation
+
+Storage Standard offers a `navigator.storage.estimate` API to get some clue of
+how much space remains.
+
+* Chrome, Firefox
+    * Can run this at any time to request an estimate of space remaining
+* Firefox
+    * Returns `0` for `usage` if a site is persisted
@@ -21,4 +21,4 @@ export PATH="$rootdir/node_modules/.bin:$PATH"

 # now run our checks
 cd "$tmpdir"
-npm run lint
+yarn lint
@@ -6,7 +6,7 @@ export NVM_DIR="$HOME/.nvm"
 [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

 nvm use 10 || exit $?
-npm install || exit $?
+yarn install || exit $?

 RC=0

@@ -18,17 +18,19 @@ function fail {
 # don't use last time's test reports
 rm -rf reports coverage || exit $?

-npm test || fail "npm test finished with return code $?"
+yarn test || fail "yarn test finished with return code $?"

-npm run -s lint -- -f checkstyle > eslint.xml ||
+yarn -s lint -f checkstyle > eslint.xml ||
    fail "eslint finished with return code $?"

 # delete the old tarball, if it exists
 rm -f matrix-js-sdk-*.tgz

-npm pack ||
-    fail "npm pack finished with return code $?"
+# `yarn pack` doesn't seem to run scripts, however that seems okay here as we
+# just built as part of `install` above.
+yarn pack ||
+    fail "yarn pack finished with return code $?"

-npm run gendoc || fail "JSDoc failed with code $?"
+yarn gendoc || fail "JSDoc failed with code $?"

 exit $RC
@@ -1,24 +1,24 @@
 {
  "name": "matrix-js-sdk",
-  "version": "1.0.0-rc.2",
+  "version": "1.0.3",
  "description": "Matrix Client-Server SDK for Javascript",
  "main": "index.js",
  "scripts": {
    "test:build": "babel -s -d specbuild spec",
    "test:run": "istanbul cover --report text --report cobertura --config .istanbul.yml -i \"lib/**/*.js\" node_modules/mocha/bin/_mocha -- --recursive specbuild --colors --reporter mocha-jenkins-reporter --reporter-options junit_report_path=reports/test-results.xml",
    "test:watch": "mocha --watch --compilers js:babel-core/register --recursive spec --colors",
-    "test": "npm run test:build && npm run test:run",
-    "check": "npm run test:build && _mocha --recursive specbuild --colors",
+    "test": "yarn test:build && yarn test:run",
+    "check": "yarn test:build && _mocha --recursive specbuild --colors",
    "gendoc": "babel --no-babelrc -d .jsdocbuild src && jsdoc -r .jsdocbuild -P package.json -R README.md -d .jsdoc",
-    "start": "npm run start:init && npm run start:watch",
+    "start": "yarn start:init && yarn start:watch",
    "start:watch": "babel -s -w --skip-initial-build -d lib src",
    "start:init": "babel -s -d lib src",
    "clean": "rimraf lib dist",
    "build": "babel -s -d lib src && rimraf dist && mkdir dist && browserify -d browser-index.js | exorcist dist/browser-matrix.js.map > dist/browser-matrix.js && uglifyjs -c -m -o dist/browser-matrix.min.js --source-map dist/browser-matrix.min.js.map --in-source-map dist/browser-matrix.js.map dist/browser-matrix.js",
-    "dist": "npm run build",
+    "dist": "yarn build",
    "watch": "watchify -d browser-index.js -o 'exorcist dist/browser-matrix.js.map > dist/browser-matrix.js' -v",
    "lint": "eslint --max-warnings 101 src spec",
-    "prepublish": "npm run clean && npm run build && git rev-parse HEAD > git-revision.txt"
+    "prepare": "yarn clean && yarn build && git rev-parse HEAD > git-revision.txt"
  },
  "repository": {
    "type": "git",
@@ -79,14 +79,15 @@
    "istanbul": "^0.4.5",
    "jsdoc": "^3.5.5",
    "lolex": "^1.5.2",
-    "matrix-mock-request": "^1.2.2",
+    "matrix-mock-request": "^1.2.3",
    "mocha": "^5.2.0",
    "mocha-jenkins-reporter": "^0.4.0",
+    "olm": "https://matrix.org/packages/npm/olm/olm-3.1.0-pre1.tgz",
    "rimraf": "^2.5.4",
    "source-map-support": "^0.4.11",
    "sourceify": "^0.1.0",
    "uglify-js": "^2.8.26",
-    "watchify": "^3.11.0"
+    "watchify": "^3.11.1"
  },
  "browserify": {
    "transform": [
@@ -6,7 +6,9 @@
 #   github-changelog-generator; install via:
 #     pip install git+https://github.com/matrix-org/github-changelog-generator.git
 #   jq; install from your distribution's package manager (https://stedolan.github.io/jq/)
-#   hub; install via brew (OSX) or source/pre-compiled binaries (debian) (https://github.com/github/hub) - Tested on v2.2.9
+#   hub; install via brew (macOS) or source/pre-compiled binaries (debian) (https://github.com/github/hub) - Tested on v2.2.9
+#   npm; typically installed by Node.js
+#   yarn; install via brew (macOS) or similar (https://yarnpkg.com/docs/install/)

 set -e

@@ -22,6 +24,8 @@ else
    echo "hub is required: please install it"
    exit
 fi
+npm --version > /dev/null || (echo "npm is required: please install it"; kill $$)
+yarn --version > /dev/null || (echo "yarn is required: please install it"; kill $$)

 USAGE="$0 [-xz] [-c changelog_file] vX.Y.Z"

@@ -88,6 +92,8 @@ if [ -z "$skip_changelog" ]; then
    update_changelog -h > /dev/null || (echo "github-changelog-generator is required: please install it"; exit)
 fi

+# Login and publish continues to use `npm`, as it seems to have more clearly
+# defined options and semantics than `yarn` for writing to the registry.
 actual_npm_user=`npm whoami`;
 if [ $expected_npm_user != $actual_npm_user ]; then
    echo "you need to be logged into npm as $expected_npm_user, but you are logged in as $actual_npm_user" >&2
@@ -147,18 +153,18 @@ cat "${changelog_file}" | `dirname $0`/scripts/changelog_head.py > "${latest_cha
 set -x

 # Bump package.json and build the dist
-echo "npm version"
-# npm version will automatically commit its modification
+echo "yarn version"
+# yarn version will automatically commit its modification
 # and make a release tag. We don't want it to create the tag
 # because it can only sign with the default key, but we can
 # only turn off both of these behaviours, so we have to
 # manually commit the result.
-npm version --no-git-tag-version "$release"
+yarn version --no-git-tag-version --new-version "$release"

-# commit package-lock.json if it exists and is versioned
-if [[ -f package-lock.json && -z `git status --porcelain --ignored package-lock.json` ]];
+# commit yarn.lock if it exists, is versioned, and is modified
+if [[ -f yarn.lock && `git status --porcelain yarn.lock | grep '^ M'` ]];
 then
-    pkglock='package-lock.json'
+    pkglock='yarn.lock'
 else
    pkglock=''
 fi
@@ -178,7 +184,7 @@ fi
 # assets.
 # We make a completely separate checkout to be sure
 # we're using released versions of the dependencies
-# (rather than whatever we're pulling in from npm link)
+# (rather than whatever we're pulling in from yarn link)
 assets=''
 dodist=0
 jq -e .scripts.dist package.json 2> /dev/null || dodist=$?
@@ -189,10 +195,10 @@ if [ $dodist -eq 0 ]; then
    pushd "$builddir"
    git clone "$projdir" .
    git checkout "$rel_branch"
-    npm install
+    yarn install
    # We haven't tagged yet, so tell the dist script what version
    # it's building
-    DIST_VERSION="$tag" npm run dist
+    DIST_VERSION="$tag" yarn dist

    popd

@@ -281,12 +287,13 @@ fi
 rm "${release_text}"
 rm "${latest_changes}"

-# publish to npmjs
+# Login and publish continues to use `npm`, as it seems to have more clearly
+# defined options and semantics than `yarn` for writing to the registry.
 npm publish

 if [ -z "$skip_jsdoc" ]; then
    echo "generating jsdocs"
-    npm run gendoc
+    yarn gendoc

    echo "copying jsdocs to gh-pages branch"
    git checkout gh-pages
@@ -311,7 +318,7 @@ git checkout master
 git pull
 git merge "$rel_branch"

-# push master  and docs (if generated) to github
+# push master and docs (if generated) to github
 git push origin master
 if [ -z "$skip_jsdoc" ]; then
    git push origin gh-pages
@@ -4,7 +4,7 @@ const sdk = require("../..");
 const HttpBackend = require("matrix-mock-request");
 const publicGlobals = require("../../lib/matrix");
 const Room = publicGlobals.Room;
-const MatrixInMemoryStore = publicGlobals.MatrixInMemoryStore;
+const MemoryStore = publicGlobals.MemoryStore;
 const Filter = publicGlobals.Filter;
 const utils = require("../test-utils");
 const MockStorageApi = require("../MockStorageApi");
@@ -23,7 +23,7 @@ describe("MatrixClient", function() {
    beforeEach(function() {
        utils.beforeEach(this); // eslint-disable-line no-invalid-this
        httpBackend = new HttpBackend();
-        store = new MatrixInMemoryStore();
+        store = new MemoryStore();

        const mockStorage = new MockStorageApi();
        sessionStore = new sdk.WebStorageSessionStore(mockStorage);
@@ -128,7 +128,7 @@ describe("MatrixClient opts", function() {
        beforeEach(function() {
            client = new MatrixClient({
                request: httpBackend.requestFn,
-                store: new sdk.MatrixInMemoryStore(),
+                store: new sdk.MemoryStore(),
                baseUrl: baseUrl,
                userId: userId,
                accessToken: accessToken,
@@ -8,6 +8,11 @@ import expect from 'expect';
 import WebStorageSessionStore from '../../lib/store/session/webstorage';
 import MemoryCryptoStore from '../../lib/crypto/store/memory-crypto-store.js';
 import MockStorageApi from '../MockStorageApi';
+import TestClient from '../TestClient';
+import {MatrixEvent} from '../../lib/models/event';
+import Room from '../../lib/models/room';
+import olmlib from '../../lib/crypto/olmlib';
+import lolex from 'lolex';

 const EventEmitter = require("events").EventEmitter;

@@ -119,4 +124,243 @@ describe("Crypto", function() {
            await prom;
        });
    });
+
+    describe('Key requests', function() {
+        let aliceClient;
+        let bobClient;
+
+        beforeEach(async function() {
+            aliceClient = (new TestClient(
+                "@alice:example.com", "alicedevice",
+            )).client;
+            bobClient = (new TestClient(
+                "@bob:example.com", "bobdevice",
+            )).client;
+            await aliceClient.initCrypto();
+            await bobClient.initCrypto();
+        });
+
+        afterEach(async function() {
+            aliceClient.stopClient();
+            bobClient.stopClient();
+        });
+
+        it(
+            "does not cancel keyshare requests if some messages are not decrypted",
+            async function() {
+                function awaitEvent(emitter, event) {
+                    return new Promise((resolve, reject) => {
+                        emitter.once(event, (result) => {
+                            resolve(result);
+                        });
+                    });
+                }
+
+                async function keyshareEventForEvent(event, index) {
+                    const eventContent = event.getWireContent();
+                    const key = await aliceClient._crypto._olmDevice
+                        .getInboundGroupSessionKey(
+                            roomId, eventContent.sender_key, eventContent.session_id,
+                            index,
+                        );
+                    const ksEvent = new MatrixEvent({
+                        type: "m.forwarded_room_key",
+                        sender: "@alice:example.com",
+                        content: {
+                            algorithm: olmlib.MEGOLM_ALGORITHM,
+                            room_id: roomId,
+                            sender_key: eventContent.sender_key,
+                            sender_claimed_ed25519_key: key.sender_claimed_ed25519_key,
+                            session_id: eventContent.session_id,
+                            session_key: key.key,
+                            chain_index: key.chain_index,
+                            forwarding_curve25519_key_chain:
+                            key.forwarding_curve_key_chain,
+                        },
+                    });
+                    // make onRoomKeyEvent think this was an encrypted event
+                    ksEvent._senderCurve25519Key = "akey";
+                    return ksEvent;
+                }
+
+                const encryptionCfg = {
+                    "algorithm": "m.megolm.v1.aes-sha2",
+                };
+                const roomId = "!someroom";
+                const aliceRoom = new Room(roomId, aliceClient, "@alice:example.com", {});
+                const bobRoom = new Room(roomId, bobClient, "@bob:example.com", {});
+                aliceClient.store.storeRoom(aliceRoom);
+                bobClient.store.storeRoom(bobRoom);
+                await aliceClient.setRoomEncryption(roomId, encryptionCfg);
+                await bobClient.setRoomEncryption(roomId, encryptionCfg);
+                const events = [
+                    new MatrixEvent({
+                        type: "m.room.message",
+                        sender: "@alice:example.com",
+                        room_id: roomId,
+                        event_id: "$1",
+                        content: {
+                            msgtype: "m.text",
+                            body: "1",
+                        },
+                    }),
+                    new MatrixEvent({
+                        type: "m.room.message",
+                        sender: "@alice:example.com",
+                        room_id: roomId,
+                        event_id: "$2",
+                        content: {
+                            msgtype: "m.text",
+                            body: "2",
+                        },
+                    }),
+                ];
+                await Promise.all(events.map(async (event) => {
+                    // alice encrypts each event, and then bob tries to decrypt
+                    // them without any keys, so that they'll be in pending
+                    await aliceClient._crypto.encryptEvent(event, aliceRoom);
+                    event._clearEvent = {};
+                    event._senderCurve25519Key = null;
+                    event._claimedEd25519Key = null;
+                    try {
+                        await bobClient._crypto.decryptEvent(event);
+                    } catch (e) {
+                        // we expect this to fail because we don't have the
+                        // decryption keys yet
+                    }
+                }));
+
+                const bobDecryptor = bobClient._crypto._getRoomDecryptor(
+                    roomId, olmlib.MEGOLM_ALGORITHM,
+                );
+
+                let eventPromise = Promise.all(events.map((ev) => {
+                    return awaitEvent(ev, "Event.decrypted");
+                }));
+
+                // keyshare the session key starting at the second message, so
+                // the first message can't be decrypted yet, but the second one
+                // can
+                let ksEvent = await keyshareEventForEvent(events[1], 1);
+                await bobDecryptor.onRoomKeyEvent(ksEvent);
+                await eventPromise;
+                expect(events[0].getContent().msgtype).toBe("m.bad.encrypted");
+                expect(events[1].getContent().msgtype).toNotBe("m.bad.encrypted");
+
+                const cryptoStore = bobClient._cryptoStore;
+                const eventContent = events[0].getWireContent();
+                const senderKey = eventContent.sender_key;
+                const sessionId = eventContent.session_id;
+                const roomKeyRequestBody = {
+                    algorithm: olmlib.MEGOLM_ALGORITHM,
+                    room_id: roomId,
+                    sender_key: senderKey,
+                    session_id: sessionId,
+                };
+                // the room key request should still be there, since we haven't
+                // decrypted everything
+                expect(await cryptoStore.getOutgoingRoomKeyRequest(roomKeyRequestBody))
+                    .toExist();
+
+                // keyshare the session key starting at the first message, so
+                // that it can now be decrypted
+                eventPromise = awaitEvent(events[0], "Event.decrypted");
+                ksEvent = await keyshareEventForEvent(events[0], 0);
+                await bobDecryptor.onRoomKeyEvent(ksEvent);
+                await eventPromise;
+                expect(events[0].getContent().msgtype).toNotBe("m.bad.encrypted");
+                // the room key request should be gone since we've now decypted everything
+                expect(await cryptoStore.getOutgoingRoomKeyRequest(roomKeyRequestBody))
+                    .toNotExist();
+            },
+        );
+
+        it("creates a new keyshare request if we request a keyshare", async function() {
+            // make sure that cancelAndResend... creates a new keyshare request
+            // if there wasn't an already-existing one
+            const event = new MatrixEvent({
+                sender: "@bob:example.com",
+                room_id: "!someroom",
+                content: {
+                    algorithm: olmlib.MEGOLM_ALGORITHM,
+                    session_id: "sessionid",
+                    sender_key: "senderkey",
+                },
+            });
+            await aliceClient.cancelAndResendEventRoomKeyRequest(event);
+            const cryptoStore = aliceClient._cryptoStore;
+            const roomKeyRequestBody = {
+                algorithm: olmlib.MEGOLM_ALGORITHM,
+                room_id: "!someroom",
+                session_id: "sessionid",
+                sender_key: "senderkey",
+            };
+            expect(await cryptoStore.getOutgoingRoomKeyRequest(roomKeyRequestBody))
+                .toExist();
+        });
+
+        it("uses a new txnid for re-requesting keys", async function() {
+            const event = new MatrixEvent({
+                sender: "@bob:example.com",
+                room_id: "!someroom",
+                content: {
+                    algorithm: olmlib.MEGOLM_ALGORITHM,
+                    session_id: "sessionid",
+                    sender_key: "senderkey",
+                },
+            });
+            /* return a promise and a function. When the function is called,
+             * the promise will be resolved.
+             */
+            function awaitFunctionCall() {
+                let func;
+                const promise = new Promise((resolve, reject) => {
+                    func = function(...args) {
+                        resolve(args);
+                        return new Promise((resolve, reject) => {
+                            // give us some time to process the result before
+                            // continuing
+                            global.setTimeout(resolve, 1);
+                        });
+                    };
+                });
+                return {func, promise};
+            }
+
+            aliceClient.startClient();
+
+            const clock = lolex.install();
+
+            try {
+                let promise;
+                // make a room key request, and record the transaction ID for the
+                // sendToDevice call
+                ({promise, func: aliceClient.sendToDevice} = awaitFunctionCall());
+                await aliceClient.cancelAndResendEventRoomKeyRequest(event);
+                clock.runToLast();
+                let args = await promise;
+                const txnId = args[2];
+                clock.runToLast();
+
+                // give the room key request manager time to update the state
+                // of the request
+                await Promise.resolve();
+
+                // cancel and resend the room key request
+                ({promise, func: aliceClient.sendToDevice} = awaitFunctionCall());
+                await aliceClient.cancelAndResendEventRoomKeyRequest(event);
+                clock.runToLast();
+                // the first call to sendToDevice will be the cancellation
+                args = await promise;
+                // the second call to sendToDevice will be the key request
+                ({promise, func: aliceClient.sendToDevice} = awaitFunctionCall());
+                clock.runToLast();
+                args = await promise;
+                clock.runToLast();
+                expect(args[2]).toNotBe(txnId);
+            } finally {
+                clock.uninstall();
+            }
+        });
+    });
 });
@@ -1,6 +1,6 @@
 /*
 Copyright 2017 Vector Creations Ltd
-Copyright 2018 New Vector Ltd
+Copyright 2018, 2019 New Vector Ltd

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -16,8 +16,6 @@ limitations under the License.
 */

 import DeviceList from '../../../lib/crypto/DeviceList';
-import MockStorageApi from '../../MockStorageApi';
-import WebStorageSessionStore from '../../../lib/store/session/webstorage';
 import MemoryCryptoStore from '../../../lib/crypto/store/memory-crypto-store.js';
 import testUtils from '../../test-utils';
 import utils from '../../../lib/utils';
@@ -57,7 +55,6 @@ const signedDeviceList = {

 describe('DeviceList', function() {
    let downloadSpy;
-    let sessionStore;
    let cryptoStore;
    let deviceLists = [];

@@ -67,8 +64,6 @@ describe('DeviceList', function() {
        deviceLists = [];

        downloadSpy = expect.createSpy();
-        const mockStorage = new MockStorageApi();
-        sessionStore = new WebStorageSessionStore(mockStorage);
        cryptoStore = new MemoryCryptoStore();
    });

@@ -85,7 +80,7 @@ describe('DeviceList', function() {
        const mockOlm = {
            verifySignature: function(key, message, signature) {},
        };
-        const dl = new DeviceList(baseApis, cryptoStore, sessionStore, mockOlm);
+        const dl = new DeviceList(baseApis, cryptoStore, mockOlm);
        deviceLists.push(dl);
        return dl;
    }
@@ -5,7 +5,6 @@ import Promise from 'bluebird';

 import sdk from '../../../..';
 import algorithms from '../../../../lib/crypto/algorithms';
-import WebStorageSessionStore from '../../../../lib/store/session/webstorage';
 import MemoryCryptoStore from '../../../../lib/crypto/store/memory-crypto-store.js';
 import MockStorageApi from '../../../MockStorageApi';
 import testUtils from '../../../test-utils';
@@ -40,10 +39,9 @@ describe("MegolmDecryption", function() {
        mockBaseApis = {};

        const mockStorage = new MockStorageApi();
-        const sessionStore = new WebStorageSessionStore(mockStorage);
        const cryptoStore = new MemoryCryptoStore(mockStorage);

-        const olmDevice = new OlmDevice(sessionStore, cryptoStore);
+        const olmDevice = new OlmDevice(cryptoStore);

        megolmDecryption = new MegolmDecryption({
            userId: '@user:id',
@@ -264,10 +262,9 @@ describe("MegolmDecryption", function() {

        it("re-uses sessions for sequential messages", async function() {
            const mockStorage = new MockStorageApi();
-            const sessionStore = new WebStorageSessionStore(mockStorage);
            const cryptoStore = new MemoryCryptoStore(mockStorage);

-            const olmDevice = new OlmDevice(sessionStore, cryptoStore);
+            const olmDevice = new OlmDevice(cryptoStore);
            olmDevice.verifySignature = expect.createSpy();
            await olmDevice.init();

@@ -1,5 +1,5 @@
 /*
-Copyright 2018 New Vector Ltd
+Copyright 2018,2019 New Vector Ltd

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -17,18 +17,18 @@ limitations under the License.
 import '../../../olm-loader';

 import expect from 'expect';
-import WebStorageSessionStore from '../../../../lib/store/session/webstorage';
 import MemoryCryptoStore from '../../../../lib/crypto/store/memory-crypto-store.js';
 import MockStorageApi from '../../../MockStorageApi';
 import testUtils from '../../../test-utils';

 import OlmDevice from '../../../../lib/crypto/OlmDevice';
+import olmlib from '../../../../lib/crypto/olmlib';
+import DeviceInfo from '../../../../lib/crypto/deviceinfo';

 function makeOlmDevice() {
    const mockStorage = new MockStorageApi();
-    const sessionStore = new WebStorageSessionStore(mockStorage);
    const cryptoStore = new MemoryCryptoStore(mockStorage);
-    const olmDevice = new OlmDevice(sessionStore, cryptoStore);
+    const olmDevice = new OlmDevice(cryptoStore);
    return olmDevice;
 }

@@ -82,5 +82,61 @@ describe("OlmDecryption", function() {
                "The olm or proteus is an aquatic salamander in the family Proteidae",
            );
        });
+
+        it("creates only one session at a time", async function() {
+            // if we call ensureOlmSessionsForDevices multiple times, it should
+            // only try to create one session at a time, even if the server is
+            // slow
+            let count = 0;
+            const baseApis = {
+                claimOneTimeKeys: () => {
+                    // simulate a very slow server (.5 seconds to respond)
+                    count++;
+                    return new Promise((resolve, reject) => {
+                        setTimeout(reject, 500);
+                    });
+                },
+            };
+            const devicesByUser = {
+                "@bob:example.com": [
+                    DeviceInfo.fromStorage({
+                        keys: {
+                            "curve25519:ABCDEFG": "akey",
+                        },
+                    }, "ABCDEFG"),
+                ],
+            };
+            function alwaysSucceed(promise) {
+                // swallow any exception thrown by a promise, so that
+                // Promise.all doesn't abort
+                return promise.catch(() => {});
+            }
+
+            // start two tasks that try to ensure that there's an olm session
+            const promises = Promise.all([
+                alwaysSucceed(olmlib.ensureOlmSessionsForDevices(
+                    aliceOlmDevice, baseApis, devicesByUser,
+                )),
+                alwaysSucceed(olmlib.ensureOlmSessionsForDevices(
+                    aliceOlmDevice, baseApis, devicesByUser,
+                )),
+            ]);
+
+            await new Promise((resolve) => {
+                setTimeout(resolve, 200);
+            });
+
+            // after .2s, both tasks should have started, but one should be
+            // waiting on the other before trying to create a session, so
+            // claimOneTimeKeys should have only been called once
+            expect(count).toBe(1);
+
+            await promises;
+
+            // after waiting for both tasks to complete, the first task should
+            // have failed, so the second task should have tried to create a
+            // new session and will have called claimOneTimeKeys
+            expect(count).toBe(2);
+        });
    });
 });
@@ -138,7 +138,7 @@ describe("MegolmBackup", function() {
        sessionStore = new WebStorageSessionStore(mockStorage);
        cryptoStore = new MemoryCryptoStore(mockStorage);

-        olmDevice = new OlmDevice(sessionStore, cryptoStore);
+        olmDevice = new OlmDevice(cryptoStore);

        // we stub out the olm encryption bits
        mockOlmLib = {};
@@ -106,19 +106,26 @@ function keyFromRecoverySession(session, decryptionKey) {
 *
 * @param {string} opts.userId The user ID for this user.
 *
- * @param {Object=} opts.store The data store to use. If not specified,
- * this client will not store any HTTP responses.
+ * @param {Object=} opts.store
+ *    The data store used for sync data from the homeserver. If not specified,
+ *    this client will not store any HTTP responses. The `createClient` helper
+ *    will create a default store if needed.
+ *
+ * @param {module:store/session/webstorage~WebStorageSessionStore} opts.sessionStore
+ *    A store to be used for end-to-end crypto session data. Most data has been
+ *    migrated out of here to `cryptoStore` instead. If not specified,
+ *    end-to-end crypto will be disabled. The `createClient` helper
+ *    _will not_ create this store at the moment.
+ *
+ * @param {module:crypto.store.base~CryptoStore} opts.cryptoStore
+ *    A store to be used for end-to-end crypto session data. If not specified,
+ *    end-to-end crypto will be disabled. The `createClient` helper will create
+ *    a default store if needed.
 *
 * @param {string=} opts.deviceId A unique identifier for this device; used for
 *    tracking things like crypto keys and access tokens.  If not specified,
 *    end-to-end crypto will be disabled.
 *
- * @param {Object=} opts.sessionStore A store to be used for end-to-end crypto
- *    session data. This should be a {@link
- *    module:store/session/webstorage~WebStorageSessionStore|WebStorageSessionStore},
- *    or an object implementing the same interface. If not specified,
- *    end-to-end crypto will be disabled.
- *
 * @param {Object} opts.scheduler Optional. The scheduler to use. If not
 * specified, this client will not retry requests on failure. This client
 * will supply its own processing function to
@@ -141,9 +148,6 @@ function keyFromRecoverySession(session, decryptionKey) {
 * maintain support for back-paginating the live timeline after a '/sync'
 * result with a gap.
 *
- * @param {module:crypto.store.base~CryptoStore} opts.cryptoStore
- *    crypto store implementation.
- *
 * @param {Array} [opts.verificationMethods] Optional. The verification method
 * that the application can handle.  Each element should be an item from {@link
 * module:crypto~verificationMethods verificationMethods}, or a class that
@@ -220,7 +224,7 @@ function MatrixClient(opts) {
    // List of which rooms have encryption enabled: separate from crypto because
    // we still want to know which rooms are encrypted even if crypto is disabled:
    // we don't want to start sending unencrypted events to them.
-    this._roomList = new RoomList(this._cryptoStore, this._sessionStore);
+    this._roomList = new RoomList(this._cryptoStore);

    // The pushprocessor caches useful things, so keep one and re-use it
    this._pushProcessor = new PushProcessor(this);
@@ -228,6 +232,32 @@ function MatrixClient(opts) {
    this._serverSupportsLazyLoading = null;

    this._cachedCapabilities = null; // { capabilities: {}, lastUpdated: timestamp }
+
+    // The SDK doesn't really provide a clean way for events to recalculate the push
+    // actions for themselves, so we have to kinda help them out when they are encrypted.
+    // We do this so that push rules are correctly executed on events in their decrypted
+    // state, such as highlights when the user's name is mentioned.
+    this.on("Event.decrypted", (event) => {
+        const oldActions = event.getPushActions();
+        const actions = this._pushProcessor.actionsForEvent(event);
+        event.setPushActions(actions); // Might as well while we're here
+
+        // Ensure the unread counts are kept up to date if the event is encrypted
+        const oldHighlight = oldActions && oldActions.tweaks
+            ? !!oldActions.tweaks.highlight : false;
+        const newHighlight = actions && actions.tweaks
+            ? !!actions.tweaks.highlight : false;
+        if (oldHighlight !== newHighlight) {
+            const room = this.getRoom(event.getRoomId());
+            // TODO: Handle mentions received while the client is offline
+            // See also https://github.com/vector-im/riot-web/issues/9069
+            if (room && !room.hasUserReadEvent(this.getUserId(), event.getId())) {
+                const current = room.getUnreadNotificationCount("highlight");
+                const newCount = newHighlight ? current + 1 : current - 1;
+                room.setUnreadNotificationCount("highlight", newCount);
+            }
+        }
+    });
 }
 utils.inherits(MatrixClient, EventEmitter);
 utils.extend(MatrixClient.prototype, MatrixBaseApis.prototype);
@@ -405,6 +435,7 @@ MatrixClient.prototype.getCapabilities = function() {
    if (this._cachedCapabilities) {
        const now = new Date().getTime();
        if (now - this._cachedCapabilities.lastUpdated <= CAPABILITIES_CACHE_MS) {
+            console.log("Returning cached capabilities");
            return Promise.resolve(this._cachedCapabilities.capabilities);
        }
    }
@@ -419,6 +450,8 @@ MatrixClient.prototype.getCapabilities = function() {
            capabilities: capabilities,
            lastUpdated: new Date().getTime(),
        };
+
+        console.log("Caching capabilities: ", capabilities);
        return capabilities;
    });
 };
@@ -760,9 +793,10 @@ MatrixClient.prototype.isEventSenderVerified = async function(event) {
 * request.
 * @param  {MatrixEvent} event event of which to cancel and resend the room
 *                            key request.
+ * @return {Promise} A promise that will resolve when the key request is queued
 */
 MatrixClient.prototype.cancelAndResendEventRoomKeyRequest = function(event) {
-    event.cancelAndResendKeyRequest(this._crypto);
+    return event.cancelAndResendKeyRequest(this._crypto, this.getUserId());
 };

 /**
@@ -2096,6 +2130,80 @@ MatrixClient.prototype.sendTyping = function(roomId, isTyping, timeoutMs, callba
    );
 };

+/**
+ * Determines the history of room upgrades for a given room, as far as the
+ * client can see. Returns an array of Rooms where the first entry is the
+ * oldest and the last entry is the newest (likely current) room. If the
+ * provided room is not found, this returns an empty list. This works in
+ * both directions, looking for older and newer rooms of the given room.
+ * @param {string} roomId The room ID to search from
+ * @param {boolean} verifyLinks If true, the function will only return rooms
+ * which can be proven to be linked. For example, rooms which have a create
+ * event pointing to an old room which the client is not aware of or doesn't
+ * have a matching tombstone would not be returned.
+ * @return {Room[]} An array of rooms representing the upgrade
+ * history.
+ */
+MatrixClient.prototype.getRoomUpgradeHistory = function(roomId, verifyLinks=false) {
+    let currentRoom = this.getRoom(roomId);
+    if (!currentRoom) return [];
+
+    const upgradeHistory = [currentRoom];
+
+    // Work backwards first, looking at create events.
+    let createEvent = currentRoom.currentState.getStateEvents("m.room.create", "");
+    while (createEvent) {
+        console.log(`Looking at ${createEvent.getId()}`);
+        const predecessor = createEvent.getContent()['predecessor'];
+        if (predecessor && predecessor['room_id']) {
+            console.log(`Looking at predecessor ${predecessor['room_id']}`);
+            const refRoom = this.getRoom(predecessor['room_id']);
+            if (!refRoom) break; // end of the chain
+
+            if (verifyLinks) {
+                const tombstone = refRoom.currentState
+                    .getStateEvents("m.room.tombstone", "");
+
+                if (!tombstone
+                    || tombstone.getContent()['replacement_room'] !== refRoom.roomId) {
+                    break;
+                }
+            }
+
+            // Insert at the front because we're working backwards from the currentRoom
+            upgradeHistory.splice(0, 0, refRoom);
+            createEvent = refRoom.currentState.getStateEvents("m.room.create", "");
+        } else {
+            // No further create events to look at
+            break;
+        }
+    }
+
+    // Work forwards next, looking at tombstone events
+    let tombstoneEvent = currentRoom.currentState.getStateEvents("m.room.tombstone", "");
+    while (tombstoneEvent) {
+        const refRoom = this.getRoom(tombstoneEvent.getContent()['replacement_room']);
+        if (!refRoom) break; // end of the chain
+
+        if (verifyLinks) {
+            createEvent = refRoom.currentState.getStateEvents("m.room.create", "");
+            if (!createEvent || !createEvent.getContent()['predecessor']) break;
+
+            const predecessor = createEvent.getContent()['predecessor'];
+            if (predecessor['room_id'] !== currentRoom.roomId) break;
+        }
+
+        // Push to the end because we're looking forwards
+        upgradeHistory.push(refRoom);
+
+        // Set the current room to the reference room so we know where we're at
+        currentRoom = refRoom;
+        tombstoneEvent = currentRoom.currentState.getStateEvents("m.room.tombstone", "");
+    }
+
+    return upgradeHistory;
+};
+
 /**
 * @param {string} roomId
 * @param {string} userId
@@ -2163,6 +2271,50 @@ MatrixClient.prototype.leave = function(roomId, callback) {
        callback);
 };

+/**
+ * Leaves all rooms in the chain of room upgrades based on the given room. By
+ * default, this will leave all the previous and upgraded rooms, including the
+ * given room. To only leave the given room and any previous rooms, keeping the
+ * upgraded (modern) rooms untouched supply `false` to `includeFuture`.
+ * @param {string} roomId The room ID to start leaving at
+ * @param {boolean} includeFuture If true, the whole chain (past and future) of
+ * upgraded rooms will be left.
+ * @return {module:client.Promise} Resolves when completed with an object keyed
+ * by room ID and value of the error encountered when leaving or null.
+ */
+MatrixClient.prototype.leaveRoomChain = function(roomId, includeFuture=true) {
+    const upgradeHistory = this.getRoomUpgradeHistory(roomId);
+
+    let eligibleToLeave = upgradeHistory;
+    if (!includeFuture) {
+        eligibleToLeave = [];
+        for (const room of upgradeHistory) {
+            eligibleToLeave.push(room);
+            if (room.roomId === roomId) {
+                break;
+            }
+        }
+    }
+
+    const populationResults = {}; // {roomId: Error}
+    const promises = [];
+
+    const doLeave = (roomId) => {
+        return this.leave(roomId).then(() => {
+            populationResults[roomId] = null;
+        }).catch((err) => {
+            populationResults[roomId] = err;
+            return null; // suppress error
+        });
+    };
+
+    for (const room of eligibleToLeave) {
+        promises.push(doLeave(room.roomId));
+    }
+
+    return Promise.all(promises).then(() => populationResults);
+};
+
 /**
 * @param {string} roomId
 * @param {string} userId
@@ -1,6 +1,6 @@
 /*
 Copyright 2017 Vector Creations Ltd
-Copyright 2018 New Vector Ltd
+Copyright 2018, 2019 New Vector Ltd

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -61,9 +61,8 @@ const TRACKING_STATUS_UP_TO_DATE = 3;
 * @alias module:crypto/DeviceList
 */
 export default class DeviceList {
-    constructor(baseApis, cryptoStore, sessionStore, olmDevice) {
+    constructor(baseApis, cryptoStore, olmDevice) {
        this._cryptoStore = cryptoStore;
-        this._sessionStore = sessionStore;

        // userId -> {
        //     deviceId -> {
@@ -108,30 +107,13 @@ export default class DeviceList {
     * Load the device tracking state from storage
     */
    async load() {
-        let shouldDeleteSessionStore = false;
        await this._cryptoStore.doTxn(
-            // migrate from session store if there's data there and not here
-            'readwrite', [IndexedDBCryptoStore.STORE_DEVICE_DATA], (txn) => {
+            'readonly', [IndexedDBCryptoStore.STORE_DEVICE_DATA], (txn) => {
                this._cryptoStore.getEndToEndDeviceData(txn, (deviceData) => {
-                    if (deviceData === null) {
-                        logger.log("Migrating e2e device data...");
-                        this._devices = this._sessionStore.getAllEndToEndDevices() || {};
-                        this._deviceTrackingStatus = (
-                            this._sessionStore.getEndToEndDeviceTrackingStatus() || {}
-                        );
-                        this._syncToken = this._sessionStore.getEndToEndDeviceSyncToken();
-                        this._cryptoStore.storeEndToEndDeviceData({
-                            devices: this._devices,
-                            trackingStatus: this._deviceTrackingStatus,
-                            syncToken: this._syncToken,
-                        }, txn);
-                        shouldDeleteSessionStore = true;
-                    } else {
-                        this._devices = deviceData ? deviceData.devices : {},
-                        this._deviceTrackingStatus = deviceData ?
-                            deviceData.trackingStatus : {};
-                        this._syncToken = deviceData ? deviceData.syncToken : null;
-                    }
+                    this._devices = deviceData ? deviceData.devices : {},
+                    this._deviceTrackingStatus = deviceData ?
+                        deviceData.trackingStatus : {};
+                    this._syncToken = deviceData ? deviceData.syncToken : null;
                    this._userByIdentityKey = {};
                    for (const user of Object.keys(this._devices)) {
                        const userDevices = this._devices[user];
@@ -146,11 +128,6 @@ export default class DeviceList {
            },
        );

-        if (shouldDeleteSessionStore) {
-            // migrated data is now safely persisted: remove from old store
-            this._sessionStore.removeEndToEndDeviceData();
-        }
-
        for (const u of Object.keys(this._deviceTrackingStatus)) {
            // if a download was in progress when we got shut down, it isn't any more.
            if (this._deviceTrackingStatus[u] == TRACKING_STATUS_DOWNLOAD_IN_PROGRESS) {
@@ -475,10 +452,10 @@ export default class DeviceList {
        if (!this._deviceTrackingStatus[userId]) {
            logger.log('Now tracking device list for ' + userId);
            this._deviceTrackingStatus[userId] = TRACKING_STATUS_PENDING_DOWNLOAD;
+            // we don't yet persist the tracking status, since there may be a lot
+            // of calls; we save all data together once the sync is done
+            this._dirty = true;
        }
-        // we don't yet persist the tracking status, since there may be a lot
-        // of calls; we save all data together once the sync is done
-        this._dirty = true;
    }

    /**
@@ -1,6 +1,6 @@
 /*
 Copyright 2016 OpenMarket Ltd
-Copyright 2017 New Vector Ltd
+Copyright 2017, 2019 New Vector Ltd

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -59,20 +59,17 @@ function checkPayloadLength(payloadString) {
 * Manages the olm cryptography functions. Each OlmDevice has a single
 * OlmAccount and a number of OlmSessions.
 *
- * Accounts and sessions are kept pickled in a sessionStore.
+ * Accounts and sessions are kept pickled in the cryptoStore.
 *
 * @constructor
 * @alias module:crypto/OlmDevice
 *
- * @param {Object} sessionStore A store to be used for data in end-to-end
- *    crypto. This is deprecated and being replaced by cryptoStore.
 * @param {Object} cryptoStore A store for crypto data
 *
 * @property {string} deviceCurve25519Key   Curve25519 key for the account
 * @property {string} deviceEd25519Key      Ed25519 key for the account
 */
-function OlmDevice(sessionStore, cryptoStore) {
-    this._sessionStore = sessionStore;
+function OlmDevice(cryptoStore) {
    this._cryptoStore = cryptoStore;
    this._pickleKey = "DEFAULT_KEY";

@@ -81,7 +78,7 @@ function OlmDevice(sessionStore, cryptoStore) {
    this.deviceEd25519Key = null;
    this._maxOneTimeKeys = null;

-    // we don't bother stashing outboundgroupsessions in the sessionstore -
+    // we don't bother stashing outboundgroupsessions in the cryptoStore -
    // instead we keep them here.
    this._outboundGroupSessionStore = {};

@@ -102,6 +99,10 @@ function OlmDevice(sessionStore, cryptoStore) {
    // Keys are strings of form "<senderKey>|<session_id>|<message_index>"
    // Values are objects of the form "{id: <event id>, timestamp: <ts>}"
    this._inboundGroupSessionMessageIndexes = {};
+
+    // Keep track of sessions that we're starting, so that we don't start
+    // multiple sessions for the same device at the same time.
+    this._sessionsInProgress = {};
 }

 /**
@@ -114,14 +115,10 @@ function OlmDevice(sessionStore, cryptoStore) {
 * Reads the device keys from the OlmAccount object.
 */
 OlmDevice.prototype.init = async function() {
-    await this._migrateFromSessionStore();
-
    let e2eKeys;
    const account = new global.Olm.Account();
    try {
-        await _initialiseAccount(
-            this._sessionStore, this._cryptoStore, this._pickleKey, account,
-        );
+        await _initialiseAccount(this._cryptoStore, this._pickleKey, account);
        e2eKeys = JSON.parse(account.identity_keys());

        this._maxOneTimeKeys = account.max_number_of_one_time_keys();
@@ -133,7 +130,7 @@ OlmDevice.prototype.init = async function() {
    this.deviceEd25519Key = e2eKeys.ed25519;
 };

-async function _initialiseAccount(sessionStore, cryptoStore, pickleKey, account) {
+async function _initialiseAccount(cryptoStore, pickleKey, account) {
    await cryptoStore.doTxn('readwrite', [IndexedDBCryptoStore.STORE_ACCOUNT], (txn) => {
        cryptoStore.getAccount(txn, (pickledAccount) => {
            if (pickledAccount !== null) {
@@ -154,95 +151,6 @@ OlmDevice.getOlmVersion = function() {
    return global.Olm.get_library_version();
 };

-OlmDevice.prototype._migrateFromSessionStore = async function() {
-    // account
-    await this._cryptoStore.doTxn(
-        'readwrite', [IndexedDBCryptoStore.STORE_ACCOUNT], (txn) => {
-            this._cryptoStore.getAccount(txn, (pickledAccount) => {
-                if (pickledAccount === null) {
-                    // Migrate from sessionStore
-                    pickledAccount = this._sessionStore.getEndToEndAccount();
-                    if (pickledAccount !== null) {
-                        logger.log("Migrating account from session store");
-                        this._cryptoStore.storeAccount(txn, pickledAccount);
-                    }
-                }
-            });
-        },
-    );
-
-    // remove the old account now the transaction has completed. Either we've
-    // migrated it or decided not to, either way we want to blow away the old data.
-    this._sessionStore.removeEndToEndAccount();
-
-    // sessions
-    const sessions = this._sessionStore.getAllEndToEndSessions();
-    if (Object.keys(sessions).length > 0) {
-        await this._cryptoStore.doTxn(
-            'readwrite', [IndexedDBCryptoStore.STORE_SESSIONS], (txn) => {
-                // Don't migrate sessions from localstorage if we already have sessions
-                // in indexeddb, since this means we've already migrated and an old version
-                // has run against the same localstorage and created some spurious sessions.
-                this._cryptoStore.countEndToEndSessions(txn, (count) => {
-                    if (count) {
-                        logger.log("Crypto store already has sessions: not migrating");
-                        return;
-                    }
-                    let numSessions = 0;
-                    for (const deviceKey of Object.keys(sessions)) {
-                        for (const sessionId of Object.keys(sessions[deviceKey])) {
-                            numSessions++;
-                            this._cryptoStore.storeEndToEndSession(
-                                deviceKey, sessionId, sessions[deviceKey][sessionId], txn,
-                            );
-                        }
-                    }
-                    logger.log(
-                        "Migrating " + numSessions + " sessions from session store",
-                    );
-                });
-            },
-        );
-
-        this._sessionStore.removeAllEndToEndSessions();
-    }
-
-    // inbound group sessions
-    const ibGroupSessions = this._sessionStore.getAllEndToEndInboundGroupSessionKeys();
-    if (Object.keys(ibGroupSessions).length > 0) {
-        let numIbSessions = 0;
-        await this._cryptoStore.doTxn(
-            'readwrite', [IndexedDBCryptoStore.STORE_INBOUND_GROUP_SESSIONS], (txn) => {
-                // We always migrate inbound group sessions, even if we already have some
-                // in the new store. They should be be safe to migrate.
-                for (const s of ibGroupSessions) {
-                    try {
-                        this._cryptoStore.addEndToEndInboundGroupSession(
-                            s.senderKey, s.sessionId,
-                            JSON.parse(
-                                this._sessionStore.getEndToEndInboundGroupSession(
-                                    s.senderKey, s.sessionId,
-                                ),
-                            ), txn,
-                        );
-                    } catch (e) {
-                        logger.warn(
-                            "Failed to migrate session " + s.senderKey + "/" +
-                            s.sessionId + ": " + e.stack || e,
-                        );
-                    }
-                    ++numIbSessions;
-                }
-                logger.log(
-                    "Migrated " + numIbSessions +
-                    " inbound group sessions from session store",
-                );
-            },
-        );
-        this._sessionStore.removeAllEndToEndInboundGroupSessions();
-    }
-};
-
 /**
 * extract our OlmAccount from the crypto store and call the given function
 * with the account object
@@ -553,6 +461,15 @@ OlmDevice.prototype.createInboundSession = async function(
 * @return {Promise<string[]>}  a list of known session ids for the device
 */
 OlmDevice.prototype.getSessionIdsForDevice = async function(theirDeviceIdentityKey) {
+    if (this._sessionsInProgress[theirDeviceIdentityKey]) {
+        console.log("waiting for session to be created");
+        try {
+            await this._sessionsInProgress[theirDeviceIdentityKey];
+        } catch (e) {
+            // if the session failed to be created, just fall through and
+            // return an empty result
+        }
+    }
    let sessionIds;
    await this._cryptoStore.doTxn(
        'readonly', [IndexedDBCryptoStore.STORE_SESSIONS],
@@ -573,10 +490,18 @@ OlmDevice.prototype.getSessionIdsForDevice = async function(theirDeviceIdentityK
 *
 * @param {string} theirDeviceIdentityKey Curve25519 identity key for the
 *     remote device
+ * @param {boolean} nowait Don't wait for an in-progress session to complete.
+ *     This should only be set to true of the calling function is the function
+ *     that marked the session as being in-progress.
 * @return {Promise<?string>}  session id, or null if no established session
 */
-OlmDevice.prototype.getSessionIdForDevice = async function(theirDeviceIdentityKey) {
-    const sessionInfos = await this.getSessionInfoForDevice(theirDeviceIdentityKey);
+OlmDevice.prototype.getSessionIdForDevice = async function(
+    theirDeviceIdentityKey, nowait,
+) {
+    const sessionInfos = await this.getSessionInfoForDevice(
+        theirDeviceIdentityKey, nowait,
+    );
+
    if (sessionInfos.length === 0) {
        return null;
    }
@@ -611,9 +536,21 @@ OlmDevice.prototype.getSessionIdForDevice = async function(theirDeviceIdentityKe
 * message and is therefore past the pre-key stage), and 'sessionId'.
 *
 * @param {string} deviceIdentityKey Curve25519 identity key for the device
+ * @param {boolean} nowait Don't wait for an in-progress session to complete.
+ *     This should only be set to true of the calling function is the function
+ *     that marked the session as being in-progress.
 * @return {Array.<{sessionId: string, hasReceivedMessage: Boolean}>}
 */
-OlmDevice.prototype.getSessionInfoForDevice = async function(deviceIdentityKey) {
+OlmDevice.prototype.getSessionInfoForDevice = async function(deviceIdentityKey, nowait) {
+    if (this._sessionsInProgress[deviceIdentityKey] && !nowait) {
+        logger.log("waiting for session to be created");
+        try {
+            await this._sessionsInProgress[deviceIdentityKey];
+        } catch (e) {
+            // if the session failed to be created, then just fall through and
+            // return an empty result
+        }
+    }
    const info = [];

    await this._cryptoStore.doTxn(
@@ -915,14 +852,6 @@ OlmDevice.prototype.addInboundGroupSession = async function(
            this._getInboundGroupSession(
                roomId, senderKey, sessionId, txn,
                (existingSession, existingSessionData) => {
-                    if (existingSession) {
-                        logger.log(
-                            "Update for megolm session " + senderKey + "/" + sessionId,
-                        );
-                        // for now we just ignore updates. TODO: implement something here
-                        return;
-                    }
-
                    // new session.
                    const session = new global.Olm.InboundGroupSession();
                    try {
@@ -938,6 +867,20 @@ OlmDevice.prototype.addInboundGroupSession = async function(
                            );
                        }

+                        if (existingSession) {
+                            logger.log(
+                                "Update for megolm session "
+                                    + senderKey + "/" + sessionId,
+                            );
+                            if (existingSession.first_known_index()
+                                <= session.first_known_index()) {
+                                // existing session has lower index (i.e. can
+                                // decrypt more), so keep it
+                                logger.log("Keeping existing session");
+                                return;
+                            }
+                        }
+
                        const sessionData = {
                            room_id: roomId,
                            session: session.pickle(this._pickleKey),
@@ -945,7 +888,7 @@ OlmDevice.prototype.addInboundGroupSession = async function(
                            forwardingCurve25519KeyChain: forwardingCurve25519KeyChain,
                        };

-                        this._cryptoStore.addEndToEndInboundGroupSession(
+                        this._cryptoStore.storeEndToEndInboundGroupSession(
                            senderKey, sessionId, sessionData, txn,
                        );
                    } finally {
@@ -124,35 +124,118 @@ export default class OutgoingRoomKeyRequestManager {
     *
     * @param {module:crypto~RoomKeyRequestBody} requestBody
     * @param {Array<{userId: string, deviceId: string}>} recipients
+     * @param {boolean} resend whether to resend the key request if there is
+     *    already one
     *
     * @returns {Promise} resolves when the request has been added to the
     *    pending list (or we have established that a similar request already
     *    exists)
     */
-    sendRoomKeyRequest(requestBody, recipients) {
-        return this._cryptoStore.getOrAddOutgoingRoomKeyRequest({
-            requestBody: requestBody,
-            recipients: recipients,
-            requestId: this._baseApis.makeTxnId(),
-            state: ROOM_KEY_REQUEST_STATES.UNSENT,
-        }).then((req) => {
-            if (req.state === ROOM_KEY_REQUEST_STATES.UNSENT) {
-                this._startTimer();
+    async sendRoomKeyRequest(requestBody, recipients, resend=false) {
+        const req = await this._cryptoStore.getOutgoingRoomKeyRequest(
+            requestBody,
+        );
+        if (!req) {
+            await this._cryptoStore.getOrAddOutgoingRoomKeyRequest({
+                requestBody: requestBody,
+                recipients: recipients,
+                requestId: this._baseApis.makeTxnId(),
+                state: ROOM_KEY_REQUEST_STATES.UNSENT,
+            });
+        } else {
+            switch (req.state) {
+            case ROOM_KEY_REQUEST_STATES.CANCELLATION_PENDING_AND_WILL_RESEND:
+            case ROOM_KEY_REQUEST_STATES.UNSENT:
+                // nothing to do here, since we're going to send a request anyways
+                return;
+
+            case ROOM_KEY_REQUEST_STATES.CANCELLATION_PENDING: {
+                // existing request is about to be cancelled.  If we want to
+                // resend, then change the state so that it resends after
+                // cancelling.  Otherwise, just cancel the cancellation.
+                const state = resend ?
+                    ROOM_KEY_REQUEST_STATES.CANCELLATION_PENDING_AND_WILL_RESEND :
+                    ROOM_KEY_REQUEST_STATES.SENT;
+                await this._cryptoStore.updateOutgoingRoomKeyRequest(
+                    req.requestId, ROOM_KEY_REQUEST_STATES.CANCELLATION_PENDING, {
+                        state,
+                        cancellationTxnId: this._baseApis.makeTxnId(),
+                    },
+                );
+                break;
            }
-        });
+            case ROOM_KEY_REQUEST_STATES.SENT: {
+                // a request has already been sent.  If we don't want to
+                // resend, then do nothing.  If we do want to, then cancel the
+                // existing request and send a new one.
+                if (resend) {
+                    const state =
+                          ROOM_KEY_REQUEST_STATES.CANCELLATION_PENDING_AND_WILL_RESEND;
+                    const updatedReq =
+                          await this._cryptoStore.updateOutgoingRoomKeyRequest(
+                              req.requestId, ROOM_KEY_REQUEST_STATES.SENT, {
+                                  state,
+                                  cancellationTxnId: this._baseApis.makeTxnId(),
+                                  // need to use a new transaction ID so that
+                                  // the request gets sent
+                                  requestTxnId: this._baseApis.makeTxnId(),
+                              },
+                          );
+                    if (!updatedReq) {
+                        // updateOutgoingRoomKeyRequest couldn't find the request
+                        // in state ROOM_KEY_REQUEST_STATES.SENT, so we must have
+                        // raced with another tab to mark the request cancelled.
+                        // Try again, to make sure the request is resent.
+                        return await this.sendRoomKeyRequest(
+                            requestBody, recipients, resend,
+                        );
+                    }
+
+                    // We don't want to wait for the timer, so we send it
+                    // immediately. (We might actually end up racing with the timer,
+                    // but that's ok: even if we make the request twice, we'll do it
+                    // with the same transaction_id, so only one message will get
+                    // sent).
+                    //
+                    // (We also don't want to wait for the response from the server
+                    // here, as it will slow down processing of received keys if we
+                    // do.)
+                    try {
+                        await this._sendOutgoingRoomKeyRequestCancellation(
+                            updatedReq,
+                            true,
+                        );
+                    } catch (e) {
+                        logger.error(
+                            "Error sending room key request cancellation;"
+                                + " will retry later.", e,
+                        );
+                    }
+                    // The request has transitioned from
+                    // CANCELLATION_PENDING_AND_WILL_RESEND to UNSENT. We
+                    // still need to resend the request which is now UNSENT, so
+                    // start the timer if it isn't already started.
+                }
+                break;
+            }
+            default:
+                throw new Error('unhandled state: ' + req.state);
+            }
+        }
+        // some of the branches require the timer to be started.  Just start it
+        // all the time, because it doesn't hurt to start it.
+        this._startTimer();
    }

    /**
     * Cancel room key requests, if any match the given requestBody
     *
     * @param {module:crypto~RoomKeyRequestBody} requestBody
-     * @param {boolean} andResend if true, transition to UNSENT instead of
-     *                            deleting after sending cancellation.
     *
     * @returns {Promise} resolves when the request has been updated in our
     *    pending list.
     */
-    cancelRoomKeyRequest(requestBody, andResend=false) {
+    cancelRoomKeyRequest(requestBody) {
        return this._cryptoStore.getOutgoingRoomKeyRequest(
            requestBody,
        ).then((req) => {
@@ -183,15 +266,10 @@ export default class OutgoingRoomKeyRequestManager {
                    );

                case ROOM_KEY_REQUEST_STATES.SENT: {
-                    // If `andResend` is set, transition to UNSENT once the
-                    // cancellation has successfully been sent.
-                    const state = andResend ?
-                        ROOM_KEY_REQUEST_STATES.CANCELLATION_PENDING_AND_WILL_RESEND :
-                        ROOM_KEY_REQUEST_STATES.CANCELLATION_PENDING;
                    // send a cancellation.
                    return this._cryptoStore.updateOutgoingRoomKeyRequest(
                        req.requestId, ROOM_KEY_REQUEST_STATES.SENT, {
-                            state,
+                            state: ROOM_KEY_REQUEST_STATES.CANCELLATION_PENDING,
                            cancellationTxnId: this._baseApis.makeTxnId(),
                        },
                    ).then((updatedReq) => {
@@ -221,20 +299,12 @@ export default class OutgoingRoomKeyRequestManager {
                        // do.)
                        this._sendOutgoingRoomKeyRequestCancellation(
                            updatedReq,
-                            andResend,
                        ).catch((e) => {
                            logger.error(
                                "Error sending room key request cancellation;"
                                + " will retry later.", e,
                            );
                            this._startTimer();
-                        }).then(() => {
-                            if (!andResend) return;
-                            // The request has transitioned from
-                            // CANCELLATION_PENDING_AND_WILL_RESEND to UNSENT. We
-                            // still need to resend the request which is now UNSENT, so
-                            // start the timer if it isn't already started.
-                            this._startTimer();
                        });
                    });
                }
@@ -280,7 +350,7 @@ export default class OutgoingRoomKeyRequestManager {
                logger.warn(
                    `error in OutgoingRoomKeyRequestManager: ${e}`,
                );
-            }).done();
+            });
        };

        this._sendOutgoingRoomKeyRequestsTimer = global.setTimeout(
@@ -331,7 +401,7 @@ export default class OutgoingRoomKeyRequestManager {
                logger.error("Error sending room key request; will retry later.", e);
                this._sendOutgoingRoomKeyRequestsTimer = null;
                this._startTimer();
-            }).done();
+            });
        });
    }

@@ -351,7 +421,7 @@ export default class OutgoingRoomKeyRequestManager {
        };

        return this._sendMessageToDevices(
-            requestMessage, req.recipients, req.requestId,
+            requestMessage, req.recipients, req.requestTxnId || req.requestId,
        ).then(() => {
            return this._cryptoStore.updateOutgoingRoomKeyRequest(
                req.requestId, ROOM_KEY_REQUEST_STATES.UNSENT,
@@ -1,5 +1,5 @@
 /*
-Copyright 2018 New Vector Ltd
+Copyright 2018, 2019 New Vector Ltd

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -26,40 +26,21 @@ import IndexedDBCryptoStore from './store/indexeddb-crypto-store';
 * @alias module:crypto/RoomList
 */
 export default class RoomList {
-    constructor(cryptoStore, sessionStore) {
+    constructor(cryptoStore) {
        this._cryptoStore = cryptoStore;
-        this._sessionStore = sessionStore;

        // Object of roomId -> room e2e info object (body of the m.room.encryption event)
        this._roomEncryption = {};
    }

    async init() {
-        let removeSessionStoreRooms = false;
        await this._cryptoStore.doTxn(
            'readwrite', [IndexedDBCryptoStore.STORE_ROOMS], (txn) => {
                this._cryptoStore.getEndToEndRooms(txn, (result) => {
-                    if (result === null || Object.keys(result).length === 0) {
-                        // migrate from session store, if there's data there
-                        const sessStoreRooms = this._sessionStore.getAllEndToEndRooms();
-                        if (sessStoreRooms !== null) {
-                            for (const roomId of Object.keys(sessStoreRooms)) {
-                                this._cryptoStore.storeEndToEndRoom(
-                                    roomId, sessStoreRooms[roomId], txn,
-                                );
-                            }
-                        }
-                        this._roomEncryption = sessStoreRooms;
-                        removeSessionStoreRooms = true;
-                    } else {
-                        this._roomEncryption = result;
-                    }
+                    this._roomEncryption = result;
                });
            },
        );
-        if (removeSessionStoreRooms) {
-            this._sessionStore.removeAllEndToEndRooms();
-        }
    }

    getRoomEncryption(roomId) {
@@ -758,7 +758,7 @@ MegolmDecryption.prototype.decryptEvent = async function(event) {
    } catch (e) {
        let errorCode = "OLM_DECRYPT_GROUP_MESSAGE_ERROR";

-        if (e.message === 'OLM.UNKNOWN_MESSAGE_INDEX') {
+        if (e && e.message === 'OLM.UNKNOWN_MESSAGE_INDEX') {
            this._requestKeysForEvent(event);

            errorCode = 'OLM_UNKNOWN_MESSAGE_INDEX';
@@ -766,7 +766,7 @@ MegolmDecryption.prototype.decryptEvent = async function(event) {

        throw new base.DecryptionError(
            errorCode,
-            e.toString(), {
+            e ? e.toString() : "Unknown Error: Error is undefined", {
                session: content.sender_key + '|' + content.session_id,
            },
        );
@@ -815,19 +815,9 @@ MegolmDecryption.prototype.decryptEvent = async function(event) {
 };

 MegolmDecryption.prototype._requestKeysForEvent = function(event) {
-    const sender = event.getSender();
    const wireContent = event.getWireContent();

-    // send the request to all of our own devices, and the
-    // original sending device if it wasn't us.
-    const recipients = [{
-        userId: this._userId, deviceId: '*',
-    }];
-    if (sender != this._userId) {
-        recipients.push({
-            userId: sender, deviceId: wireContent.device_id,
-        });
-    }
+    const recipients = event.getKeyRequestRecipients(this._userId);

    this._crypto.requestRoomKey({
        room_id: event.getRoomId(),
@@ -938,16 +928,23 @@ MegolmDecryption.prototype.onRoomKeyEvent = function(event) {
        content.session_key, keysClaimed,
        exportFormat,
    ).then(() => {
-        // cancel any outstanding room key requests for this session
-        this._crypto.cancelRoomKeyRequest({
-            algorithm: content.algorithm,
-            room_id: content.room_id,
-            session_id: content.session_id,
-            sender_key: senderKey,
-        });
-
        // have another go at decrypting events sent with this session.
-        this._retryDecryption(senderKey, sessionId);
+        this._retryDecryption(senderKey, sessionId)
+            .then((success) => {
+                // cancel any outstanding room key requests for this session.
+                // Only do this if we managed to decrypt every message in the
+                // session, because if we didn't, we leave the other key
+                // requests in the hopes that someone sends us a key that
+                // includes an earlier index.
+                if (success) {
+                    this._crypto.cancelRoomKeyRequest({
+                        algorithm: content.algorithm,
+                        room_id: content.room_id,
+                        session_id: content.session_id,
+                        sender_key: senderKey,
+                    });
+                }
+            });
    }).then(() => {
        if (this._crypto.backupInfo) {
            // don't wait for the keys to be backed up for the server
@@ -1105,19 +1102,27 @@ MegolmDecryption.prototype.importRoomKey = function(session) {
 * @private
 * @param {String} senderKey
 * @param {String} sessionId
+ *
+ * @return {Boolean} whether all messages were successfully decrypted
 */
-MegolmDecryption.prototype._retryDecryption = function(senderKey, sessionId) {
+MegolmDecryption.prototype._retryDecryption = async function(senderKey, sessionId) {
    const k = senderKey + "|" + sessionId;
    const pending = this._pendingEvents[k];
    if (!pending) {
-        return;
+        return true;
    }

    delete this._pendingEvents[k];

-    for (const ev of pending) {
-        ev.attemptDecryption(this._crypto);
-    }
+    await Promise.all([...pending].map(async (ev) => {
+        try {
+            await ev.attemptDecryption(this._crypto);
+        } catch (e) {
+            // don't die if something goes wrong
+        }
+    }));
+
+    return !this._pendingEvents[k];
 };

 base.registerAlgorithm(
@@ -134,9 +134,9 @@ export default function Crypto(baseApis, sessionStore, userId, deviceId,
    this._checkedForBackup = false; // Have we checked the server for a backup we can use?
    this._sendingBackups = false; // Are we currently sending backups?

-    this._olmDevice = new OlmDevice(sessionStore, cryptoStore);
+    this._olmDevice = new OlmDevice(cryptoStore);
    this._deviceList = new DeviceList(
-        baseApis, cryptoStore, sessionStore, this._olmDevice,
+        baseApis, cryptoStore, this._olmDevice,
    );

    // the last time we did a check for the number of one-time-keys on the
@@ -198,25 +198,6 @@ utils.inherits(Crypto, EventEmitter);
 */
 Crypto.prototype.init = async function() {
    await global.Olm.init();
-
-    const sessionStoreHasAccount = Boolean(this._sessionStore.getEndToEndAccount());
-    let cryptoStoreHasAccount;
-    await this._cryptoStore.doTxn(
-        'readonly', [IndexedDBCryptoStore.STORE_ACCOUNT], (txn) => {
-            this._cryptoStore.getAccount(txn, (pickledAccount) => {
-                cryptoStoreHasAccount = Boolean(pickledAccount);
-            });
-        },
-    );
-    if (sessionStoreHasAccount && !cryptoStoreHasAccount) {
-        // we're about to migrate to the crypto store
-        this.emit("crypto.warning", 'CRYPTO_WARNING_ACCOUNT_MIGRATED');
-    } else if (sessionStoreHasAccount && cryptoStoreHasAccount) {
-        // There's an account in both stores: an old version of
-        // the code has been run against this store.
-        this.emit("crypto.warning", 'CRYPTO_WARNING_OLD_VERSION_DETECTED');
-    }
-
    await this._olmDevice.init();
    await this._deviceList.load();

@@ -235,7 +216,7 @@ Crypto.prototype.init = async function() {
    }

    if (!myDevices[this._deviceId]) {
-        // add our own deviceinfo to the sessionstore
+        // add our own deviceinfo to the cryptoStore
        const deviceInfo = {
            keys: this._deviceKeys,
            algorithms: this._supportedAlgorithms,
@@ -942,7 +923,7 @@ Crypto.prototype.forceDiscardSession = function(roomId) {
 };

 /**
- * Configure a room to use encryption (ie, save a flag in the sessionstore).
+ * Configure a room to use encryption (ie, save a flag in the cryptoStore).
 *
 * @param {string} roomId The room ID to enable encryption in.
 *
@@ -1426,10 +1407,14 @@ Crypto.prototype.handleDeviceListChanges = async function(syncData, syncDeviceLi
 *
 * @param {module:crypto~RoomKeyRequestBody} requestBody
 * @param {Array<{userId: string, deviceId: string}>} recipients
+ * @param {boolean} resend whether to resend the key request if there is
+ *    already one
+ *
+ * @return {Promise} a promise that resolves when the key request is queued
 */
-Crypto.prototype.requestRoomKey = function(requestBody, recipients) {
-    this._outgoingRoomKeyRequestManager.sendRoomKeyRequest(
-        requestBody, recipients,
+Crypto.prototype.requestRoomKey = function(requestBody, recipients, resend=false) {
+    return this._outgoingRoomKeyRequestManager.sendRoomKeyRequest(
+        requestBody, recipients, resend,
    ).catch((e) => {
        // this normally means we couldn't talk to the store
        logger.error(
@@ -1443,11 +1428,9 @@ Crypto.prototype.requestRoomKey = function(requestBody, recipients) {
 *
 * @param {module:crypto~RoomKeyRequestBody} requestBody
 *    parameters to match for cancellation
- * @param {boolean} andResend
- *    if true, resend the key request after cancelling.
 */
-Crypto.prototype.cancelRoomKeyRequest = function(requestBody, andResend) {
-    this._outgoingRoomKeyRequestManager.cancelRoomKeyRequest(requestBody, andResend)
+Crypto.prototype.cancelRoomKeyRequest = function(requestBody) {
+    this._outgoingRoomKeyRequestManager.cancelRoomKeyRequest(requestBody)
    .catch((e) => {
        logger.warn("Error clearing pending room key requests", e);
    }).done();
@@ -1984,7 +1967,7 @@ Crypto.prototype._onToDeviceBadEncrypted = async function(event) {
            sender, device.deviceId,
        );
    for (const keyReq of requestsToResend) {
-        this.cancelRoomKeyRequest(keyReq.requestBody, true);
+        this.requestRoomKey(keyReq.requestBody, keyReq.recipients, true);
    }
 };

@@ -2356,17 +2339,6 @@ class IncomingRoomKeyRequestCancellation {
 * Fires when the app may wish to warn the user about something related
 * the end-to-end crypto.
 *
- * Comes with a type which is one of:
- * * CRYPTO_WARNING_ACCOUNT_MIGRATED: Account data has been migrated from an older
- *       version of the store in such a way that older clients will no longer be
- *       able to read it. The app may wish to warn the user against going back to
- *       an older version of the app.
- * * CRYPTO_WARNING_OLD_VERSION_DETECTED: js-sdk has detected that an older version
- *       of js-sdk has been run against the same store after a migration has been
- *       performed. This is likely have caused unexpected behaviour in the old
- *       version. For example, the old version and the new version may have two
- *       different identity keys.
- *
 * @event module:client~MatrixClient#"crypto.warning"
 * @param {string} type One of the strings listed above
 */
@@ -1,5 +1,6 @@
 /*
 Copyright 2016 OpenMarket Ltd
+Copyright 2019 New Vector Ltd

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -137,6 +138,7 @@ module.exports.ensureOlmSessionsForDevices = async function(
        // [userId, deviceId], ...
    ];
    const result = {};
+    const resolveSession = {};

    for (const userId in devicesByUser) {
        if (!devicesByUser.hasOwnProperty(userId)) {
@@ -148,7 +150,36 @@ module.exports.ensureOlmSessionsForDevices = async function(
            const deviceInfo = devices[j];
            const deviceId = deviceInfo.deviceId;
            const key = deviceInfo.getIdentityKey();
-            const sessionId = await olmDevice.getSessionIdForDevice(key);
+            if (!olmDevice._sessionsInProgress[key]) {
+                // pre-emptively mark the session as in-progress to avoid race
+                // conditions.  If we find that we already have a session, then
+                // we'll resolve
+                olmDevice._sessionsInProgress[key] = new Promise(
+                    (resolve, reject) => {
+                        resolveSession[key] = {
+                            resolve: (...args) => {
+                                delete olmDevice._sessionsInProgress[key];
+                                resolve(...args);
+                            },
+                            reject: (...args) => {
+                                delete olmDevice._sessionsInProgress[key];
+                                reject(...args);
+                            },
+                        };
+                    },
+                );
+            }
+            const sessionId = await olmDevice.getSessionIdForDevice(
+                key, resolveSession[key],
+            );
+            if (sessionId !== null && resolveSession[key]) {
+                // we found a session, but we had marked the session as
+                // in-progress, so unmark it and unblock anything that was
+                // waiting
+                delete olmDevice._sessionsInProgress[key];
+                resolveSession[key].resolve();
+                delete resolveSession[key];
+            }
            if (sessionId === null || force) {
                devicesWithoutSession.push([userId, deviceId]);
            }
@@ -163,16 +194,19 @@ module.exports.ensureOlmSessionsForDevices = async function(
        return result;
    }

-    // TODO: this has a race condition - if we try to send another message
-    // while we are claiming a key, we will end up claiming two and setting up
-    // two sessions.
-    //
-    // That should eventually resolve itself, but it's poor form.
-
    const oneTimeKeyAlgorithm = "signed_curve25519";
-    const res = await baseApis.claimOneTimeKeys(
-        devicesWithoutSession, oneTimeKeyAlgorithm,
-    );
+    let res;
+    try {
+        res = await baseApis.claimOneTimeKeys(
+            devicesWithoutSession, oneTimeKeyAlgorithm,
+        );
+    } catch (e) {
+        for (const resolver of Object.values(resolveSession)) {
+            resolver.resolve();
+        }
+        logger.log("failed to claim one-time keys", e, devicesWithoutSession);
+        throw e;
+    }

    const otk_res = res.one_time_keys || {};
    const promises = [];
@@ -185,6 +219,7 @@ module.exports.ensureOlmSessionsForDevices = async function(
        for (let j = 0; j < devices.length; j++) {
            const deviceInfo = devices[j];
            const deviceId = deviceInfo.deviceId;
+            const key = deviceInfo.getIdentityKey();
            if (result[userId][deviceId].sessionId && !force) {
                // we already have a result for this device
                continue;
@@ -199,10 +234,12 @@ module.exports.ensureOlmSessionsForDevices = async function(
            }

            if (!oneTimeKey) {
-                logger.warn(
-                    "No one-time keys (alg=" + oneTimeKeyAlgorithm +
-                        ") for device " + userId + ":" + deviceId,
-                );
+                const msg = "No one-time keys (alg=" + oneTimeKeyAlgorithm +
+                      ") for device " + userId + ":" + deviceId;
+                logger.warn(msg);
+                if (resolveSession[key]) {
+                    resolveSession[key].resolve();
+                }
                continue;
            }

@@ -210,7 +247,15 @@ module.exports.ensureOlmSessionsForDevices = async function(
                _verifyKeyAndStartSession(
                    olmDevice, oneTimeKey, userId, deviceInfo,
                ).then((sid) => {
+                    if (resolveSession[key]) {
+                        resolveSession[key].resolve(sid);
+                    }
                    result[userId][deviceId].sessionId = sid;
+                }, (e) => {
+                    if (resolveSession[key]) {
+                        resolveSession[key].resolve();
+                    }
+                    throw e;
                }),
            );
        }
@@ -690,11 +690,12 @@ function promiseifyTxn(txn) {
            }
            resolve();
        };
-        txn.onerror = () => {
+        txn.onerror = (event) => {
            if (txn._mx_abortexception !== undefined) {
                reject(txn._mx_abortexception);
+            } else {
+                reject(event.target.error);
            }
-            reject();
        };
        txn.onabort = () => reject(txn._mx_abortexception);
    });
@@ -22,6 +22,7 @@ import LocalStorageCryptoStore from './localStorage-crypto-store';
 import MemoryCryptoStore from './memory-crypto-store';
 import * as IndexedDBCryptoStoreBackend from './indexeddb-crypto-store-backend';
 import {InvalidCryptoStoreError} from '../../errors';
+import * as IndexedDBHelpers from "../../indexeddb-helpers";

 /**
 * Internal module. indexeddb storage for e2e.
@@ -48,6 +49,10 @@ export default class IndexedDBCryptoStore {
        this._backendPromise = null;
    }

+    static exists(indexedDB, dbName) {
+        return IndexedDBHelpers.exists(indexedDB, dbName);
+    }
+
    /**
     * Ensure the database exists and is up-to-date, or fall back to
     * a local storage or in-memory store.
@@ -57,6 +57,16 @@ export default class LocalStorageCryptoStore extends MemoryCryptoStore {
        this.store = webStore;
    }

+    static exists(webStore) {
+        const length = webStore.length;
+        for (let i = 0; i < length; i++) {
+            if (webStore.key(i).startsWith(E2E_PREFIX)) {
+                return true;
+            }
+        }
+        return false;
+    }
+
    // Olm Sessions

    countEndToEndSessions(txn, func) {
@@ -107,7 +107,7 @@ const emojiMapping = [
    ["📕", "book"],       // 42
    ["✏️", "pencil"],     // 43
    ["📎", "paperclip"],  // 44
-    ["✂️", "scisors"],    // 45
+    ["✂️", "scissors"],    // 45
    ["🔒", "padlock"],    // 46
    ["🔑", "key"],        // 47
    ["🔨", "hammer"],     // 48
@@ -0,0 +1,52 @@
+/*
+Copyright 2019 New Vector Ltd
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+import Promise from 'bluebird';
+
+/**
+ * Check if an IndexedDB database exists. The only way to do so is to try opening it, so
+ * we do that and then delete it did not exist before.
+ *
+ * @param {Object} indexedDB The `indexedDB` interface
+ * @param {string} dbName The database name to test for
+ * @returns {boolean} Whether the database exists
+ */
+export function exists(indexedDB, dbName) {
+    return new Promise((resolve, reject) => {
+        let exists = true;
+        const req = indexedDB.open(dbName);
+        req.onupgradeneeded = () => {
+            // Since we did not provide an explicit version when opening, this event
+            // should only fire if the DB did not exist before at any version.
+            exists = false;
+        };
+        req.onblocked = () => reject();
+        req.onsuccess = () => {
+            const db = req.result;
+            db.close();
+            if (!exists) {
+                // The DB did not exist before, but has been created as part of this
+                // existence check. Delete it now to restore previous state. Delete can
+                // actually take a while to complete in some browsers, so don't wait for
+                // it. This won't block future open calls that a store might issue next to
+                // properly set up the DB.
+                indexedDB.deleteDatabase(dbName);
+            }
+            resolve(exists);
+        };
+        req.onerror = ev => reject(ev.target.error);
+    });
+}
@@ -22,8 +22,14 @@ module.exports.ContentHelpers = require("./content-helpers");
 module.exports.MatrixEvent = require("./models/event").MatrixEvent;
 /** The {@link module:models/event.EventStatus|EventStatus} enum. */
 module.exports.EventStatus = require("./models/event").EventStatus;
-/** The {@link module:store/memory.MatrixInMemoryStore|MatrixInMemoryStore} class. */
-module.exports.MatrixInMemoryStore = require("./store/memory").MatrixInMemoryStore;
+/** The {@link module:store/memory.MemoryStore|MemoryStore} class. */
+module.exports.MemoryStore = require("./store/memory").MemoryStore;
+/**
+ * The {@link module:store/memory.MemoryStore|MemoryStore} class was previously
+ * exported as `MatrixInMemoryStore`, so this is preserved for SDK consumers.
+ * @deprecated Prefer `MemoryStore` going forward.
+ */
+module.exports.MatrixInMemoryStore = module.exports.MemoryStore;
 /** The {@link module:store/indexeddb.IndexedDBStore|IndexedDBStore} class. */
 module.exports.IndexedDBStore = require("./store/indexeddb").IndexedDBStore;
 /** The {@link module:store/indexeddb.IndexedDBStoreBackend|IndexedDBStoreBackend} class. */
@@ -88,21 +94,21 @@ module.exports.createNewMatrixCall = require("./webrtc/call").createNewMatrixCal


 /**
- * Set an audio output device to use for MatrixCalls
+ * Set a preferred audio output device to use for MatrixCalls
 * @function
 * @param {string=} deviceId the identifier for the device
 * undefined treated as unset
 */
 module.exports.setMatrixCallAudioOutput = require('./webrtc/call').setAudioOutput;
 /**
- * Set an audio input device to use for MatrixCalls
+ * Set a preferred audio input device to use for MatrixCalls
 * @function
 * @param {string=} deviceId the identifier for the device
 * undefined treated as unset
 */
 module.exports.setMatrixCallAudioInput = require('./webrtc/call').setAudioInput;
 /**
- * Set a video input device to use for MatrixCalls
+ * Set a preferred video input device to use for MatrixCalls
 * @function
 * @param {string=} deviceId the identifier for the device
 * undefined treated as unset
@@ -164,7 +170,7 @@ module.exports.setCryptoStoreFactory = function(fac) {
 * this is a string, it is assumed to be the base URL. These configuration
 * options will be passed directly to {@link module:client~MatrixClient}.
 * @param {Object} opts.store If not set, defaults to
- * {@link module:store/memory.MatrixInMemoryStore}.
+ * {@link module:store/memory.MemoryStore}.
 * @param {Object} opts.scheduler If not set, defaults to
 * {@link module:scheduler~MatrixScheduler}.
 * @param {requestFunction} opts.request If not set, defaults to the function
@@ -187,7 +193,7 @@ module.exports.createClient = function(opts) {
        };
    }
    opts.request = opts.request || request;
-    opts.store = opts.store || new module.exports.MatrixInMemoryStore({
+    opts.store = opts.store || new module.exports.MemoryStore({
      localStorage: global.localStorage,
    });
    opts.scheduler = opts.scheduler || new module.exports.MatrixScheduler();
@@ -382,15 +382,41 @@ utils.extend(module.exports.MatrixEvent.prototype, {
     * Cancel any room key request for this event and resend another.
     *
     * @param {module:crypto} crypto crypto module
+     * @param {string} userId the user who received this event
+     *
+     * @returns {Promise} a promise that resolves when the request is queued
     */
-    cancelAndResendKeyRequest: function(crypto) {
+    cancelAndResendKeyRequest: function(crypto, userId) {
        const wireContent = this.getWireContent();
-        crypto.cancelRoomKeyRequest({
+        return crypto.requestRoomKey({
            algorithm: wireContent.algorithm,
            room_id: this.getRoomId(),
            session_id: wireContent.session_id,
            sender_key: wireContent.sender_key,
-        }, true);
+        }, this.getKeyRequestRecipients(userId), true);
+    },
+
+    /**
+     * Calculate the recipients for keyshare requests.
+     *
+     * @param {string} userId the user who received this event.
+     *
+     * @returns {Array} array of recipients
+     */
+    getKeyRequestRecipients: function(userId) {
+        // send the request to all of our own devices, and the
+        // original sending device if it wasn't us.
+        const wireContent = this.getWireContent();
+        const recipients = [{
+            userId, deviceId: '*',
+        }];
+        const sender = this.getSender();
+        if (sender !== userId) {
+            recipients.push({
+                userId: sender, deviceId: wireContent.device_id,
+            });
+        }
+        return recipients;
    },

    _decryptionLoop: async function(crypto) {
@@ -471,6 +497,14 @@ utils.extend(module.exports.MatrixEvent.prototype, {
            this._retryDecryption = false;
            this._setClearData(res);

+            // Before we emit the event, clear the push actions so that they can be recalculated
+            // by relevant code. We do this because the clear event has now changed, making it
+            // so that existing rules can be re-run over the applicable properties. Stuff like
+            // highlighting when the user's name is mentioned rely on this happening. We also want
+            // to set the push actions before emitting so that any notification listeners don't
+            // pick up the wrong contents.
+            this.setPushActions(null);
+
            this.emit("Event.decrypted", this, err);

            return;
@@ -511,6 +545,17 @@ utils.extend(module.exports.MatrixEvent.prototype, {
            decryptionResult.forwardingCurve25519KeyChain || [];
    },

+    /**
+     * Gets the cleartext content for this event. If the event is not encrypted,
+     * or encryption has not been completed, this will return null.
+     *
+     * @returns {Object} The cleartext (decrypted) content for the event
+     */
+    getClearContent: function() {
+        const ev = this._clearEvent;
+        return ev && ev.content ? ev.content : null;
+    },
+
    /**
     * Check if the event is encrypted.
     * @return {boolean} True if this event is encrypted.
@@ -260,6 +260,8 @@ Room.prototype.getRecommendedVersion = async function() {
    }

    const currentVersion = this.getVersion();
+    console.log(`[${this.roomId}] Current version: ${currentVersion}`);
+    console.log(`[${this.roomId}] Version capability: `, versionCap);

    const result = {
        version: currentVersion,
@@ -280,6 +282,11 @@ Room.prototype.getRecommendedVersion = async function() {
        result.version = versionCap.default;
        result.needsUpgrade = true;
        result.urgent = !!this.getVersion().match(/^[0-9]+[0-9.]*$/g);
+        if (result.urgent) {
+            console.warn(`URGENT upgrade required on ${this.roomId}`);
+        } else {
+            console.warn(`Non-urgent upgrade required on ${this.roomId}`);
+        }
        return Promise.resolve(result);
    }

@@ -1424,6 +1431,40 @@ Room.prototype.getEventReadUpTo = function(userId, ignoreSynthesized) {
    return receipts["m.read"][userId].eventId;
 };

+/**
+ * Determines if the given user has read a particular event ID with the known
+ * history of the room. This is not a definitive check as it relies only on
+ * what is available to the room at the time of execution.
+ * @param {String} userId The user ID to check the read state of.
+ * @param {String} eventId The event ID to check if the user read.
+ * @returns {Boolean} True if the user has read the event, false otherwise.
+ */
+Room.prototype.hasUserReadEvent = function(userId, eventId) {
+    const readUpToId = this.getEventReadUpTo(userId, false);
+    if (readUpToId === eventId) return true;
+
+    if (this.timeline.length
+        && this.timeline[this.timeline.length - 1].getSender()
+        && this.timeline[this.timeline.length - 1].getSender() === userId) {
+        // It doesn't matter where the event is in the timeline, the user has read
+        // it because they've sent the latest event.
+        return true;
+    }
+
+    for (let i = this.timeline.length - 1; i >= 0; --i) {
+        const ev = this.timeline[i];
+
+        // If we encounter the target event first, the user hasn't read it
+        // however if we encounter the readUpToId first then the user has read
+        // it. These rules apply because we're iterating bottom-up.
+        if (ev.getId() === eventId) return false;
+        if (ev.getId() === readUpToId) return true;
+    }
+
+    // We don't know if the user has read it, so assume not.
+    return false;
+};
+
 /**
 * Get a list of receipts for the given event.
 * @param {MatrixEvent} event the event to get receipts for
@@ -184,7 +184,10 @@ function PushProcessor(client) {
    };

    const eventFulfillsDisplayNameCondition = function(cond, ev) {
-        const content = ev.getContent();
+        let content = ev.getContent();
+        if (ev.isEncrypted() && ev.getClearContent()) {
+            content = ev.getClearContent();
+        }
        if (!content || !content.body || typeof content.body != 'string') {
            return false;
        }
@@ -18,6 +18,7 @@ limitations under the License.
 import Promise from 'bluebird';
 import SyncAccumulator from "../sync-accumulator";
 import utils from "../utils";
+import * as IndexedDBHelpers from "../indexeddb-helpers";

 const VERSION = 3;

@@ -132,6 +133,10 @@ const LocalIndexedDBStoreBackend = function LocalIndexedDBStoreBackend(
    this._isNewlyCreated = false;
 };

+LocalIndexedDBStoreBackend.exists = function(indexedDB, dbName) {
+    dbName = "matrix-js-sdk:" + (dbName || "default");
+    return IndexedDBHelpers.exists(indexedDB, dbName);
+};

 LocalIndexedDBStoreBackend.prototype = {
    /**
@@ -16,7 +16,7 @@ limitations under the License.
 */

 import Promise from 'bluebird';
-import {MatrixInMemoryStore} from "./memory";
+import {MemoryStore} from "./memory";
 import utils from "../utils";
 import LocalIndexedDBStoreBackend from "./indexeddb-local-backend.js";
 import RemoteIndexedDBStoreBackend from "./indexeddb-remote-backend.js";
@@ -37,9 +37,9 @@ const WRITE_DELAY_MS = 1000 * 60 * 5; // once every 5 minutes


 /**
- * Construct a new Indexed Database store, which extends MatrixInMemoryStore.
+ * Construct a new Indexed Database store, which extends MemoryStore.
 *
- * This store functions like a MatrixInMemoryStore except it periodically persists
+ * This store functions like a MemoryStore except it periodically persists
 * the contents of the store to an IndexedDB backend.
 *
 * All data is still kept in-memory but can be loaded from disk by calling
@@ -62,7 +62,7 @@ const WRITE_DELAY_MS = 1000 * 60 * 5; // once every 5 minutes
 * </pre>
 *
 * @constructor
- * @extends MatrixInMemoryStore
+ * @extends MemoryStore
 * @param {Object} opts Options object.
 * @param {Object} opts.indexedDB The Indexed DB interface e.g.
 * <code>window.indexedDB</code>
@@ -79,7 +79,7 @@ const WRITE_DELAY_MS = 1000 * 60 * 5; // once every 5 minutes
 * database.
 */
 const IndexedDBStore = function IndexedDBStore(opts) {
-    MatrixInMemoryStore.call(this, opts);
+    MemoryStore.call(this, opts);

    if (!opts.indexedDB) {
        throw new Error('Missing required option: indexedDB');
@@ -109,7 +109,11 @@ const IndexedDBStore = function IndexedDBStore(opts) {
        // user_id : timestamp
    };
 };
-utils.inherits(IndexedDBStore, MatrixInMemoryStore);
+utils.inherits(IndexedDBStore, MemoryStore);
+
+IndexedDBStore.exists = function(indexedDB, dbName) {
+    return LocalIndexedDBStoreBackend.exists(indexedDB, dbName);
+};

 /**
 * @return {Promise} Resolved when loaded from indexed db.
@@ -164,7 +168,7 @@ IndexedDBStore.prototype.getSavedSyncToken = function() {
 * @return {Promise} Resolves if the data was deleted from the database.
 */
 IndexedDBStore.prototype.deleteAllData = function() {
-    MatrixInMemoryStore.prototype.deleteAllData.call(this);
+    MemoryStore.prototype.deleteAllData.call(this);
    return this.backend.clearDatabase().then(() => {
        console.log("Deleted indexeddb data.");
    }, (err) => {
@@ -17,7 +17,7 @@ limitations under the License.
 */
 "use strict";
 /**
- * This is an internal module. See {@link MatrixInMemoryStore} for the public class.
+ * This is an internal module. See {@link MemoryStore} for the public class.
 * @module store/memory
 */
 const utils = require("../utils");
@@ -31,7 +31,7 @@ import Promise from 'bluebird';
 * @param {LocalStorage} opts.localStorage The local storage instance to persist
 * some forms of data such as tokens. Rooms will NOT be stored.
 */
-module.exports.MatrixInMemoryStore = function MatrixInMemoryStore(opts) {
+module.exports.MemoryStore = function MemoryStore(opts) {
    opts = opts || {};
    this.rooms = {
        // roomId: Room
@@ -58,7 +58,7 @@ module.exports.MatrixInMemoryStore = function MatrixInMemoryStore(opts) {
    this._clientOptions = {};
 };

-module.exports.MatrixInMemoryStore.prototype = {
+module.exports.MemoryStore.prototype = {

    /**
     * Retrieve the token to stream from.
@@ -724,6 +724,9 @@ SyncApi.prototype._sync = async function(syncOptions) {
        // log the exception with stack if we have it, else fall back
        // to the plain description
        console.error("Caught /sync error", e.stack || e);
+
+        // Emit the exception for client handling
+        this.client.emit("sync.unexpectedError", e);
    }

    // update this as it may have changed
@@ -583,7 +583,7 @@ MatrixCall.prototype._maybeGotUserMediaForInvite = function(stream) {
            ' Or possibly you are using an insecure domain. Receiving only.');
        this.peerConn = _createPeerConnection(this);
    } else {
-        debuglog('Failed to getUserMedia.');
+        debuglog('Failed to getUserMedia: ' + error.name);
        this._getUserMediaFailed(error);
        return;
    }
@@ -652,7 +652,7 @@ MatrixCall.prototype._maybeGotUserMediaForAnswer = function(stream) {
        debuglog('User denied access to camera/microphone.' +
            ' Or possibly you are using an insecure domain. Receiving only.');
    } else {
-        debuglog('Failed to getUserMedia.');
+        debuglog('Failed to getUserMedia: ' + error.name);
        this._getUserMediaFailed(error);
        return;
    }
@@ -1274,15 +1274,15 @@ const _getUserMediaVideoContraints = function(callType) {
        case 'voice':
            return {
                audio: {
-                    deviceId: audioInput ? {exact: audioInput} : undefined,
+                    deviceId: audioInput ? {ideal: audioInput} : undefined,
                }, video: false,
            };
        case 'video':
            return {
                audio: {
-                    deviceId: audioInput ? {exact: audioInput} : undefined,
+                    deviceId: audioInput ? {ideal: audioInput} : undefined,
                }, video: {
-                    deviceId: videoInput ? {exact: videoInput} : undefined,
+                    deviceId: videoInput ? {ideal: videoInput} : undefined,
                    /* We want 640x360.  Chrome will give it only if we ask exactly,
                       FF refuses entirely if we ask exactly, so have to ask for ideal
                       instead */
@@ -1,13 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-npm run lint
-
-# install Olm so that we can run the crypto tests.
-npm install https://matrix.org/packages/npm/olm/olm-3.1.0-pre1.tgz
-
-npm run test
-
-npm run gendoc
-
Author	SHA1	Message	Date
David Baker	e2c17528c2	v1.0.3	2019-04-01 13:31:45 +01:00
David Baker	e00f565f37	Prepare changelog for v1.0.3	2019-04-01 13:31:44 +01:00
J. Ryan Stinnett	085e797c30	Merge pull request #874 from matrix-org/travis/1.0.6-fire/872 Add existence check to local storage based crypto store	2019-04-01 09:26:39 +01:00
J. Ryan Stinnett	d753db590b	Add existence check to local storage based crypto store This supports additional diagnostics of stores in https://github.com/vector-im/riot-web/issues/9309.	2019-03-29 16:19:54 -06:00
David Baker	feac096dc2	v1.0.3-rc.1	2019-03-27 17:29:28 +00:00
David Baker	67985d449a	Prepare changelog for v1.0.3-rc.1	2019-03-27 17:29:27 +00:00
J. Ryan Stinnett	a6de59c198	Update matrix-mock-request	2019-03-27 15:02:48 +00:00
J. Ryan Stinnett	01eeb98e35	Merge pull request #871 from jryans/storage-edge-cases Add IndexedDB existence checks	2019-03-26 15:32:23 +00:00
J. Ryan Stinnett	49a7defbf0	Close the DB right away This seems to be needed in Chrome to avoid blocking future opens.	2019-03-26 14:58:45 +00:00
J. Ryan Stinnett	c1ba5de686	Add missing word	2019-03-26 11:25:59 +00:00
J. Ryan Stinnett	81428f23d1	Tweak code style	2019-03-26 11:21:22 +00:00
Travis Ralston	8af86bb746	Merge pull request #869 from matrix-org/travis/timeline-explosion-workaround Emit sync errors for capturing by clients	2019-03-25 11:33:01 -06:00
J. Ryan Stinnett	8513f5c413	Add an IndexedDB existence check to the crypto store This will be useful for future storage diagnostics as part of https://github.com/vector-im/riot-web/issues/9271.	2019-03-25 16:11:59 +00:00
J. Ryan Stinnett	eadec35093	Add an IndexedDB existence check to the main store This will be useful for future storage diagnostics as part of https://github.com/vector-im/riot-web/issues/9271.	2019-03-25 16:11:51 +00:00
Travis Ralston	d6dbd621b8	Merge pull request #868 from matrix-org/travis/leave-room-chain Add functions for getting room upgrade history and leaving those rooms	2019-03-25 09:57:10 -06:00
Travis Ralston	7168f76614	Emit sync errors for capturing by clients For https://github.com/vector-im/riot-web/issues/9260	2019-03-22 20:19:38 -06:00
Travis Ralston	1cda95f23c	Add functions for getting room upgrade history and leaving those rooms Required for https://github.com/vector-im/riot-web/issues/8539	2019-03-22 17:34:05 -06:00
J. Ryan Stinnett	bb1cd2bbce	Merge pull request #867 from jryans/real-name Clarify the meaning of 'real name' for contribution	2019-03-22 17:26:21 +00:00
J. Ryan Stinnett	3f90ac5712	Merge pull request #865 from jryans/storage-edge-cases Remove `sessionStore` to `cryptoStore` migration path	2019-03-22 17:25:51 +00:00
J. Ryan Stinnett	8d249a843c	Reformat contribution doc to 80 chars	2019-03-22 11:53:43 +00:00
J. Ryan Stinnett	858b41d835	Clarify the meaning of 'real name' for contribution This applies the clarification already adopted by Synapse: https://github.com/matrix-org/synapse/commit/ec766b25303b420850e6d2875f156f23109acf6a	2019-03-22 11:51:07 +00:00
Travis Ralston	61aea05af0	Merge pull request #866 from matrix-org/travis/verbose-capabilities Add debugging for spurious room version warnings	2019-03-20 17:10:23 -06:00
Travis Ralston	e7c764d5f5	Add debugging for spurious room version warnings See https://github.com/vector-im/riot-web/issues/9225	2019-03-20 17:05:09 -06:00
J. Ryan Stinnett	09a9afe4e7	Clarify the current state of each store This moves docs about the 3 stores used next to each other and clarifies their purpose for future readers.	2019-03-20 16:56:47 +00:00
J. Ryan Stinnett	5a26503da7	Remove `sessionStore` to `cryptoStore` migration path The code to migrate from the `sessionStore` to `cryptoStore` originally appeared in https://github.com/matrix-org/matrix-js-sdk/pull/584 (2017-12-06). At this point, it seems safe to assume most sessions that need migrating have already done so. Removing this code simplifies store handling and removes the `sessionStore` from most places in JS SDK.	2019-03-20 16:56:47 +00:00
J. Ryan Stinnett	5faf5ea1f8	Merge pull request #864 from jryans/storage-notes Add investigation notes for browser storage	2019-03-20 16:35:13 +00:00
J. Ryan Stinnett	0754c29c22	Add investigation notes for browser storage This collects my notes from investigating the state of browser storage as part of https://github.com/vector-im/riot-web/issues/9109.	2019-03-20 16:33:47 +00:00
Hubert Chathi	d5c6dcf111	Merge pull request #862 from uhoreg/check_resolver make sure resolve object is defined before calling it	2019-03-20 10:32:10 -04:00
J. Ryan Stinnett	6a57ddd33c	Merge pull request #861 from matrix-org/jryans/storage-edge-cases Rename `MatrixInMemoryStore` to `MemoryStore`	2019-03-20 11:05:55 +00:00
Hubert Chathi	bd711cdc1f	make sure resolve object is defined before calling it	2019-03-19 23:21:49 -04:00
J. Ryan Stinnett	e669e493c9	Add deprecation notice to `MatrixInMemoryStore`	2019-03-19 15:04:05 +00:00
J. Ryan Stinnett	48f290196c	Rename `MatrixInMemoryStore` to `MemoryStore` None of the other store classes use the `Matrix` prefix, and I find the mismatch confusing (it leads me to think it might have a different purpose than the others). This change removes the prefix from the store for consistency. The old name is left as an export for existing SDK consumers.	2019-03-19 14:24:47 +00:00
David Baker	f8985dbb39	Merge branch 'master' into develop	2019-03-18 13:50:07 +00:00
David Baker	ef594d52e4	v1.0.2	2019-03-18 13:47:54 +00:00
David Baker	23bbb2f8c6	Prepare changelog for v1.0.2	2019-03-18 13:47:53 +00:00
Travis Ralston	73c7733ebc	Merge pull request #859 from matrix-org/travis/buildkite Use Buildkite for CI	2019-03-15 11:43:50 -06:00
Travis Ralston	87f7f9443e	Delete Travis (CI) We now use BuildKite	2019-03-13 17:27:17 -06:00
Travis Ralston	af6bbbc59b	Add a basic BuildKite :pipeline:	2019-03-13 17:23:30 -06:00
J. Ryan Stinnett	5b35a364a9	Ignore package-lock.json in case of confused npm users	2019-03-13 15:49:18 +00:00
Hubert Chathi	d56ebadbc4	Merge pull request #857 from uhoreg/fix_ensureolm_race only create one session at a time per device	2019-03-13 11:21:35 -04:00
J. Ryan Stinnett	04accdeddc	Ignore *.log files, such as from npm or Yarn	2019-03-13 14:40:42 +00:00
David Baker	70575f9e33	v1.0.2-rc.1	2019-03-13 14:24:57 +00:00
David Baker	8e16586d84	Merge branch 'develop' into release-v1.0.2	2019-03-13 14:24:26 +00:00
David Baker	6920dfb800	Yarn needs --new-version to create a new version	2019-03-13 14:23:51 +00:00
David Baker	02d93770aa	Prepare changelog for v1.0.2-rc.1	2019-03-13 14:20:18 +00:00
David Baker	cd124231c5	Merge pull request #858 from jryans/yarn-ci Use modern Yarn version on Travis CI	2019-03-13 13:22:39 +00:00
J. Ryan Stinnett	cd75848882	Use modern Yarn version on Travis CI Travis CI uses a quite old version of Yarn by default. This adds Yarn's recommended incantation for using the latest stable version.	2019-03-13 12:28:07 +00:00
J. Ryan Stinnett	1bae15ede9	Recommend using the latest Node LTS version	2019-03-13 10:50:58 +00:00
Hubert Chathi	8c2001adbf	don't reject on error because we ignore it anyways, and it makes an unrelated unit test fail	2019-03-12 17:24:34 -04:00
Hubert Chathi	79ca235e7c	only create one session at a time per device	2019-03-12 16:04:26 -04:00
J. Ryan Stinnett	4570fcaa8a	Ignore .npmrc	2019-03-12 12:02:04 +00:00
J. Ryan Stinnett	90670cf1be	Merge pull request #856 from jryans/yarn Switch to `yarn` for dependency management	2019-03-11 13:27:04 +00:00
J. Ryan Stinnett	cc86f427d2	Convert Olm to a dev dependency This gives more natural behavior of downloading Olm by default when working on this module, so that all tests including crypto are run.	2019-03-08 15:58:08 +00:00
J. Ryan Stinnett	2144791d52	Update scripts and docs to use `yarn` where appropriate Most `npm` operations are replaced with `yarn`, which generally has better behavior. However, steps like publish that write to the NPM registry are left to `npm`, which currently handles these tasks best.	2019-03-08 15:45:57 +00:00
J. Ryan Stinnett	33aabf44e7	Convert from npm to yarn lock file The npm lock file was imported into yarn. A yarn install pass was then run to double-check the lock file for sanity.	2019-03-08 10:47:58 +00:00
Hubert Chathi	fc1ea27380	Merge pull request #855 from uhoreg/fix_key_requests More key request fixes	2019-03-06 14:46:43 -05:00
Hubert Chathi	81946294d8	use lolex to fake the timer	2019-03-06 14:42:52 -05:00
Matthew Hodgson	77270fa78c	Merge pull request #851 from matrix-org/travis/e2e-notifs Calculate encrypted notification counts	2019-03-06 17:11:22 +00:00
Hubert Chathi	9e29289dcc	use a different transaction ID when re-sending a key request	2019-03-06 12:02:48 -05:00
Travis Ralston	6198943976	Add a mention that we should be handling gaps in /sync	2019-03-06 09:56:36 -07:00
David Baker	8beb836ccd	Merge pull request #854 from matrix-org/dbkr/deps_190206 Update dependencies	2019-03-06 14:08:26 +00:00
David Baker	b7c0e39c1a	Update dependencies To get non-vulnerable versions	2019-03-06 13:50:35 +00:00
David Baker	777acae2e5	Merge branch 'master' into develop	2019-03-06 11:17:25 +00:00
David Baker	e77389c1ce	v1.0.1	2019-03-06 11:10:42 +00:00
David Baker	eb24e2e1f1	Prepare changelog for v1.0.1	2019-03-06 11:10:41 +00:00
Hubert Chathi	114244f8bb	fix typo	2019-03-06 00:23:01 -05:00
Travis Ralston	54769d9136	Add jsdoc	2019-03-05 14:25:32 -07:00
Travis Ralston	2f2deb5333	Expose the clear event content directly from an event	2019-03-05 14:04:51 -07:00
Travis Ralston	37f106d4af	More safely set the push actions for an encrypted event	2019-03-05 14:04:39 -07:00
David Baker	36ee7cdbfc	v1.0.1-rc.2	2019-03-05 18:53:16 +00:00
David Baker	2b564498ee	Prepare changelog for v1.0.1-rc.2	2019-03-05 18:53:15 +00:00
David Baker	0ddba16fa1	Merge pull request #853 from matrix-org/bwindels/dontswallowcryptotxnerrors dont swallow txn errors in crypto store	2019-03-05 18:36:15 +00:00
Hubert Chathi	550086eb67	Merge pull request #850 from uhoreg/ensure_key_request make sure key requests get sent	2019-03-05 10:57:15 -05:00
Hubert Chathi	055ce673cd	fix jsdoc	2019-03-05 10:54:02 -05:00
David Baker	5308595658	Merge pull request #852 from matrix-org/dbkr/getusermedia_deviceid_ideal Use 'ideal' rather than 'exact' for deviceid	2019-03-05 14:23:39 +00:00
David Baker	e726e29f39	Better logging here too	2019-03-05 13:02:55 +00:00
David Baker	33b12fa6b5	Use 'ideal' rather than 'exact' for deviceid We were using 'exact' which means we fail outright if the device we wanted isn't available. This means if a user selects a specific device then later unplugs it, we fail to open a capture device the next time they make a call even if there's one available. Using 'ideal' uses the chosen device in preference, but something else if it isn't available. Also log the name of the exception when we fail to open a capture device to give us more of an idea of what's gone wrong. Should help fix https://github.com/vector-im/riot-web/issues/8993	2019-03-05 12:59:29 +00:00
Travis Ralston	829cd05cba	Appease the linter	2019-03-04 21:46:48 -07:00
Travis Ralston	4834e12a3a	Calculate unread badges for encrypted events	2019-03-04 21:41:20 -07:00
Travis Ralston	bcd4ad130c	Use the decrypted event content when checking the push rules Otherwise we'll be looking at the encrypted source, and that doesn't help anyone.	2019-03-04 21:33:57 -07:00
Travis Ralston	998d9e010e	Support flushing the cache on calculated push rules Needed for encrypted events to be able to pass some push rules.	2019-03-04 21:24:25 -07:00
Hubert Chathi	5480e8e1d5	refactor key sharing requests use sendRoomKeyRequest with a new resend flag, instead of cancelRoomKeyRequest, when requesting keys, so that we make sure that we send a new request if there is no previous request fixes https://github.com/vector-im/riot-web/issues/6838	2019-03-04 17:09:56 -05:00
Hubert Chathi	98fdcabc00	stop client after each test	2019-03-04 16:59:54 -05:00
Bruno Windels	236397816d	Merge pull request #849 from matrix-org/bwindels/dontswallowcryptotxnerrors Don't swallow txn errors in crypto store	2019-03-04 13:05:17 +01:00
Bruno Windels	755c55de3e	dont swallow txn errors in crypto store	2019-03-04 12:58:05 +01:00
Hubert Chathi	526da71992	Merge pull request #848 from uhoreg/fix_partial_keyshare handle partially-shared sessions better	2019-03-01 12:18:54 -05:00
Hubert Chathi	86ef262799	fix c+p comment	2019-03-01 12:15:59 -05:00
Hubert Chathi	282904d4be	restore the order of backup vs retry decryption	2019-03-01 08:54:04 -05:00
Hubert Chathi	a1be24307a	lint	2019-02-28 22:54:46 -05:00
Hubert Chathi	4b5623691b	handle partially-shared sessions better - don't cancel key requests if we can't decrypt everything in the session - overwrite the session key if we get a better version	2019-02-28 16:01:29 -05:00
David Baker	7bdf1e9b92	v1.0.1-rc.1	2019-02-28 14:28:36 +00:00
David Baker	af1db8a606	Prepare changelog for v1.0.1-rc.1	2019-02-28 14:28:36 +00:00
Travis Ralston	a99bb3c4c9	Merge pull request #847 from matrix-org/travis/fix-megolm-error Fix "e is undefined" masking the original error in MegolmDecryption	2019-02-26 13:59:16 -07:00
Travis Ralston	fd155c15bd	Excessive checks are excessive	2019-02-26 13:50:09 -07:00
Travis Ralston	aaa43631aa	Fix "e is undefined" masking the original error in MegolmDecryption	2019-02-26 13:15:03 -07:00
Hubert Chathi	d2557bc943	only set the dirty flag if something was actually changed	2019-02-22 08:37:34 -05:00
Hubert Chathi	33a3506981	speling is hard	2019-02-15 17:14:29 -05:00
J. Ryan Stinnett	03a54353be	v1.0.0	2019-02-14 16:48:12 +00:00
J. Ryan Stinnett	c6328923e6	Prepare changelog for v1.0.0	2019-02-14 16:48:11 +00:00
J. Ryan Stinnett	1ecb820bb0	Merge branch 'develop' into release-v1.0.0	2019-02-14 16:46:34 +00:00
J. Ryan Stinnett	0be2319288	Merge pull request #841 from jryans/package-lock-release Try again to commit package-lock.json	2019-02-14 15:57:21 +00:00
J. Ryan Stinnett	073a025b83	Try again to commit package-lock.json	2019-02-14 14:20:31 +00:00