|
| 1 | +# Cypress in Element Web |
| 2 | + |
| 3 | +## Scope of this Document |
| 4 | +This doc is about our Cypress tests in Element Web and how we use Cypress to write tests. |
| 5 | +It aims to cover: |
| 6 | + * How to run the tests yourself |
| 7 | + * How the tests work |
| 8 | + * How to write great Cypress tests |
| 9 | + |
| 10 | +## Running the Tests |
| 11 | +Our Cypress tests run automatically as part of our CI along with our other tests, |
| 12 | +on every pull request and on every merge to develop. |
| 13 | + |
| 14 | +However the Cypress tests are run, an element-web must be running on |
| 15 | +http://localhost:8080 (this is configured in `cypress.json`) - this is what will |
| 16 | +be tested. When running Cypress tests yourself, the standard `yarn start` from the |
| 17 | +element-web project is fine: leave it running it a different terminal as you would |
| 18 | +when developing. |
| 19 | + |
| 20 | +The tests use Docker to launch Synapse instances to test against, so you'll also |
| 21 | +need to have Docker installed and working in order to run the Cypress tests. |
| 22 | + |
| 23 | +There are a few different ways to run the tests yourself. The simplest is to run: |
| 24 | + |
| 25 | +``` |
| 26 | +yarn run test:cypress |
| 27 | +``` |
| 28 | + |
| 29 | +This will run the Cypress tests once, non-interactively. |
| 30 | + |
| 31 | +You can also run individual tests this way too, as you'd expect: |
| 32 | + |
| 33 | +``` |
| 34 | +yarn run test:cypress cypress/integration/1-register/register.spec.ts |
| 35 | +``` |
| 36 | + |
| 37 | +Cypress also has its own UI that you can use to run and debug the tests. |
| 38 | +To launch it: |
| 39 | + |
| 40 | +``` |
| 41 | +yarn run test:cypress:open |
| 42 | +``` |
| 43 | + |
| 44 | +## How the Tests Work |
| 45 | +Everything Cypress-related lives in the `cypress/` subdirectory of react-sdk |
| 46 | +as is typical for Cypress tests. Likewise, tests live in `cypress/integration`. |
| 47 | + |
| 48 | +`cypress/plugins/synapsedocker` contains a Cypress plugin that starts instances |
| 49 | +of Synapse in Docker containers. These synapses are what Element-web runs against |
| 50 | +in the Cypress tests. |
| 51 | + |
| 52 | +Synapse can be launched with different configurations in order to test element |
| 53 | +in different configurations. `cypress/plugins/synapsedocker/templates` contains |
| 54 | +template configuration files for each different configuration. |
| 55 | + |
| 56 | +Each test suite can then launch whatever Syanpse instances it needs it whatever |
| 57 | +configurations. |
| 58 | + |
| 59 | +Note that although tests should stop the Synapse instances after running and the |
| 60 | +plugin also stop any remaining instances after all tests have run, it is possible |
| 61 | +to be left with some stray containers if, for example, you terminate a test such |
| 62 | +that the `after()` does not run and also exit Cypress uncleanly. All the containers |
| 63 | +it starts are prefixed so they are easy to recognise. They can be removed safely. |
| 64 | + |
| 65 | +After each test run, logs from the Syanpse instances are saved in `cypress/synapselogs` |
| 66 | +with each instance in a separate directory named after it's ID. These logs are removed |
| 67 | +at the start of each test run. |
| 68 | + |
| 69 | +## Writing Tests |
| 70 | +Mostly this is the same advice as for writing any other Cypress test: the Cypress |
| 71 | +docs are well worth a read if you're not already familiar with Cypress testing, eg. |
| 72 | +https://docs.cypress.io/guides/references/best-practices . |
| 73 | + |
| 74 | +### Getting a Synapse |
| 75 | +The key difference is in starting Synapse instances. Tests use this plugin via |
| 76 | +`cy.task()` to provide a Synapse instance to log into: |
| 77 | + |
| 78 | +``` |
| 79 | +cy.task<SynapseInstance>("synapseStart", "consent").then(result => { |
| 80 | + synapseId = result.synapseId; |
| 81 | + synapsePort = result.port; |
| 82 | +}); |
| 83 | +``` |
| 84 | + |
| 85 | +This returns an object with information about the Synapse instance, including what port |
| 86 | +it was started on and the ID that needs to be passed to shut it down again. It also |
| 87 | +returns the registration shared secret (`registrationSecret`) that can be used to |
| 88 | +register users via the REST API. |
| 89 | + |
| 90 | +Synapse instances should be reasonably cheap to start (you may see the first one take a |
| 91 | +while as it pulls the Docker image), so it's generally expected that tests will start a |
| 92 | +Synapse instance for each test suite, ie. in `before()`, and then tear it down in `after()`. |
| 93 | + |
| 94 | +### Synapse Config Templates |
| 95 | +When a Synapse instance is started, it's given a config generated from one of the config |
| 96 | +templates in `cypress/plugins/synapsedocker/templates`. There are a couple of special files |
| 97 | +in these templates: |
| 98 | + * `homeserver.yaml`: |
| 99 | + Template substitution happens in this file. Template variables are: |
| 100 | + * `REGISTRATION_SECRET`: The secret used to register users via the REST API. |
| 101 | + * `MACAROON_SECRET_KEY`: Generated each time for security |
| 102 | + * `FORM_SECRET`: Generated each time for security |
| 103 | + * `localhost.signing.key`: A signing key is auto-generated and saved to this file. |
| 104 | + Config templates should not contain a signing key and instead assume that one will exist |
| 105 | + in this file. |
| 106 | + |
| 107 | +All other files in the template are copied recursively to `/data/`, so the file `foo.html` |
| 108 | +in a template can be referenced in the config as `/data/foo.html`. |
| 109 | + |
| 110 | +### Logging In |
| 111 | +This doesn't quite exist yet. Most tests will just want to start with the client in a 'logged in' |
| 112 | +state, so we should provide an easy way to start a test with element in this state. The |
| 113 | +`registrationSecret` provided when starting a Synapse can be used to create a user (porting |
| 114 | +the code from https://github.com/matrix-org/matrix-react-sdk/blob/develop/test/end-to-end-tests/src/rest/creator.ts#L49). |
| 115 | +We'd then need to log in as this user. Ways of doing this would be: |
| 116 | + |
| 117 | +1. Fill in the login form. This isn't ideal as it's effectively testing the login process in each |
| 118 | + test, and will just be slower. |
| 119 | +1. Mint an access token using https://matrix-org.github.io/synapse/develop/admin_api/user_admin_api.html#login-as-a-user |
| 120 | + then inject this into element-web. This would probably be fastest, although also relies on correctly |
| 121 | + setting up localstorage |
| 122 | +1. Mint a login token, inject the Homeserver URL into localstorage and then load element, passing the login |
| 123 | + token as a URL parameter. This is a supported way of logging in to element-web, but there's no API |
| 124 | + on Synapse to make such a token currently. It would be fairly easy to add a synapse-specific admin API |
| 125 | + to do so. We should write tests for token login (and the rest of SSO) at some point anyway though. |
| 126 | + |
| 127 | +If we make this as a convenience API, it can easily be swapped out later: we could start with option 1 |
| 128 | +and then switch later. |
| 129 | + |
| 130 | +### Joining a Room |
| 131 | +Many tests will also want to start with the client in a room, ready to send & receive messages. Best |
| 132 | +way to do this may be to get an access token for the user and use this to create a room with the REST |
| 133 | +API before logging the user in. |
| 134 | + |
| 135 | +### Convenience APIs |
| 136 | +We should probably end up with convenience APIs that wrap the synapse creation, logging in and room |
| 137 | +creation that can be called to set up tests. |
| 138 | + |
| 139 | +## Good Test Hygiene |
| 140 | +This section mostly summarises general good Cypress testing practice, and should not be news to anyone |
| 141 | +already familiar with Cypress. |
| 142 | + |
| 143 | +1. Test a well-isolated unit of functionality. The more specific, the easier it will be to tell what's |
| 144 | + wrong when they fail. |
| 145 | +1. Don't depend on state from other tests: any given test should be able to run in isolation. |
| 146 | +1. Try to avoid driving the UI for anything other than the UI you're trying to test. eg. if you're |
| 147 | + testing that the user can send a reaction to a message, it's best to send a message using a REST |
| 148 | + API, then react to it using the UI, rather than using the element-web UI to send the message. |
| 149 | +1. Avoid explicit waits. `cy.get()` will implicitly wait for the specified element to appear and |
| 150 | + all assertions are retired until they either pass or time out, so you should never need to |
| 151 | + manually wait for an element. |
| 152 | + * For example, for asserting about editing an already-edited message, you can't wait for the |
| 153 | + 'edited' element to appear as there was already one there, but you can assert that the body |
| 154 | + of the message is what is should be after the second edit and this assertion will pass once |
| 155 | + it becomes true. You can then assert that the 'edited' element is still in the DOM. |
| 156 | + * You can also wait for other things like network requests in the |
| 157 | + browser to complete (https://docs.cypress.io/guides/guides/network-requests#Waiting). |
| 158 | + Needing to wait for things can also be because of race conditions in the app itself, which ideally |
| 159 | + shouldn't be there! |
| 160 | + |
| 161 | +This is a small selection - the Cypress best practices guide, linked above, has more good advice, and we |
| 162 | +should generally try to adhere to them. |
0 commit comments