Richer tests for Synapse modules

[matrixbot]

This issue has been migrated from #11084.

---

It would be useful to have richer tests for Synapse modules.

The idea of making Complement extensible (if it's not already) and doing tests that way has been suggested but it seems like quite a chunk of work.

One simple-seeming way of doing better tests for Synapse modules might be to do tests in the style of actual Synapse features. For example, using HomeserverTestCase and checking that various REST requests all give the expected results.

(We seem happy to acknowledge for now that this could be a bit brittle if we wind up refactoring Synapse's test helpers, but that's infrequent enough)

This is not as simple as I first thought; Synapse's test helpers are contained in the tests package which is explicitly excluded by setup.py: https://github.com/matrix-org/synapse/blob/b3698f945cabb44649df03617db3c51d9fc62674/setup.py#L141 .

The effect of this seems to be that it's not possible (at least, without imagining some nasty hacks) to import those test helpers into a module test.

What should we do about this?

Naïvely, I would perhaps like to see these pulled out into a separate package which is installable as e.g. matrix-synapse-test-helpers, but really not sure what is ergonomic or typical here in Python-land (it also seems like we'd have a circular dependency if we did that, synapse <---> test helpers).