encryption regularly breaks in weird ways, usually you see a message that you can’t read

This was once common, but it’s somewhat rare now in my experience, and the upcoming Matrix 2.0 apparently addresses most (all?) of the remaining causes.

if you enable encryption in a chat room you cannot disable it

I consider this a good thing, for the sake of the people who joined or wrote in the chat with the understanding that what they write is and will remain encrypted. If you want to abandon encryption, you can always create a new room.

we now have two official clients for Android (Element and Element X) in the first one encryption breaks in weird ways, in the later there is no way to use Spaces properly

No, there is one officially released client for android: Element. Element X is in beta. When it leaves beta, it will take over as the one officially released client.

direct messages between people don’t work well - it is like they are a room with the two people

It works well for me. How is it a problem for you? It looks just like the person-to-person chats on other platforms I use, including SMS.

privacy wise matrix is weak,

Privacy of message content is not weak at all.

leaks metadata,

It’s true that some metadata can be read by admins of the servers that have been invited into a chat. Given all the features that Matrix uniquely offers, that’s an acceptable tradeoff for many of us. Also, the developers have stated that moving most of that metadata to the encrypted channel is planned.

attachments are not encrypted, etc.

This is just plain false.

https://spec.matrix.org/latest/client-server-api/#sending-encrypted-attachments