To explain more specifically. Matrix is federated, so you can talk to other matrix servers like Lemmy instances can. However, matrix not part of the Fediverse, which is built on activity pub, which lets Lemmy, Masotodon, Pixfed and others all talk to each other.
That does sound rad. Mostly my point was just about not having push-to-talk or toggle mute in the jitsi plug in, but I am far from deep on what can be done with the matrix protocol.