remove room without listeners from Notifier.room_to_user_streams

[skazancev]

Context: #17791 (comment)

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct
  (run the linters)

[CLAassistant]

All committers have signed the CLA.

[anoadragon453]

Looks harmless, tests pass, and apparently this removes a memory leak.

Let's try it out, thanks!