From 905a884f72e8c9cff559b18da015166317aba52b Mon Sep 17 00:00:00 2001 From: James Salter Date: Thu, 17 Mar 2022 09:10:12 +0000 Subject: [PATCH] Formalise guidelines around writing tests (#2179) --- CONTRIBUTING.md | 50 +++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 40 insertions(+), 10 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 696f4df88..61516817e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -100,18 +100,48 @@ checks, so please check back after a few minutes. Tests ----- -If your PR is a feature (ie. if it's being labelled with the 'T-Enhancement' -label) then we require that the PR also includes tests. These need to test that -your feature works as expected and ideally test edge cases too. For the js-sdk -itself, your tests should generally be unit tests. matrix-react-sdk also uses -these guidelines, so for that your tests can be unit tests using -react-test-utils, snapshot tests or screenshot tests. +Your PR should include tests. -We don't require tests for bug fixes (T-Defect) but strongly encourage regression -tests for the bug itself wherever possible. +For new user facing features in `matrix-react-sdk` or `element-web`, you +must include: -In the future we may formalise this more with a minimum test coverage -percentage for the diff. +1. Comprehensive unit tests written in Jest. These are located in `/test`. +2. "happy path" end-to-end tests. + These are located in `/test/end-to-end-tests` in `matrix-react-sdk`, and + are run using `element-web`. Ideally, you would also include tests for edge + and error cases. + +Unit tests are expected even when the feature is in labs. It's good practice +to write tests alongside the code as it ensures the code is testable from +the start, and gives you a fast feedback loop while you're developing the +functionality. End-to-end tests should be added prior to the feature +leaving labs, but don't have to be present from the start (although it might +be beneficial to have some running early, so you can test things faster). + +For bugs in those repos, your change must include at least one unit test or +end-to-end test; which is best depends on what sort of test most concisely +exercises the area. + +Changes to `matrix-js-sdk` must be accompanied by unit tests written in Jest. +These are located in `/spec/`. + +When writing unit tests, please aim for a high level of test coverage +for new code - 80% or greater. If you cannot achieve that, please document +why it's not possible in your PR. + +Tests validate that your change works as intended and also document +concisely what is being changed. Ideally, your new tests fail +prior to your change, and succeed once it has been applied. You may +find this simpler to achieve if you write the tests first. + +If you're spiking some code that's experimental and not being used to support +production features, exceptions can be made to requirements for tests. +Note that tests will still be required in order to ship the feature, and it's +strongly encouraged to think about tests early in the process, as adding +tests later will become progressively more difficult. + +If you're not sure how to approach writing tests for your change, ask for help +in [#element-dev](https://matrix.to/#/#element-dev:matrix.org). Code style ----------