There's been steady progress since the last news entry, so I thought I'd write a small progress update again.

These crates saw a new release:

• ruma-api-macros 0.9.0 (with an alpha beforehand)
• ruma-api 0.12.0 (with an alpha beforehand)
• ruma-client-api 0.5.0
• ruma-client 0.3.0-beta.2
• ruma-identifiers 0.14.1

What changed?

The biggest thing that changed is that the temporary non-support of server-side usage of ruma-api that I wrote about last time has ended. Restoring server-side support also required some changes to ruma-client-api, but those were rather minimal because all of the additional logic that deals with the client / server side distinction is hidden away in ruma-api(-macros).

Moreover, ruma-client now has no more dependencies with alpha or beta versions, and will have its 0.3.0 release soon (pending a small update from hyper). ruma-identifiers now supports historical user IDs, which are user IDs can't actually be created on a matrix homeserver because they contain invalid characeters, but nevertheless have to be supported because older versions of the specification allowed these characters and as such some users still have them in their IDs.

Server-side support

To support fallible deserialization on both ends of the protocol, the request-sending / response-receiving side (when dealing with the client-server protocl, this is the client) and the request-receiving / response-sending side (the server) now use different types if request or response structs contain events. For every Request or Response that contains an event (or another struct that contains another struct that contains a Vec of events and such) we now generate a corresponding IncomingRequest / IncomingResponse. The non-Incoming structs don't use ruma-events' EventResult type, while the Incoming types do.

For details about how these new structs are generated, see the documentation for the new Outgoing derive macro.

Last week, we uploaded 7 (!) new versions of our crates to crates.io:

• ruma-events-macros 0.2.0
• ruma-events 0.15.0
• ruma-events 0.15.1 (bugfix release because ruma-events 0.15.0 was released a tad too early)
• ruma-api-macros 0.8.0
• ruma-api 0.11.0
• ruma-client-api 0.4.0
• ruma-client 0.3.0-beta.1

What changed?

A quick overview:

• ruma-events 0.15 improves event deserialization and validation
• the latest ruma-api & ruma-client-api are only useable for implementing client-side stuff, server-side functionality will be added back in a later release
• ruma-client 0.3.0-beta.1 supports async/await, the final version will be published after its dependencies are final too

And now to the technical details...

Fallible event deserialization

The most important change is one in ruma-events, and it is the reason all of the other crates saw a new release too. ruma-events 0.15.0 introduces a type EventResult<T>, which is almost the same as Result<T, ruma_events::InvalidEvent> (and it is in fact convertible to that) but with one important difference: It implements serde::Deserialize in a different way. Deserializing an EventResult<T> almost always succeeds. It tries to deserialize the T, and if that fails, falls back to serde_json::Value deserialization. However, in contrast to the fallback one would get with Result<T, serde_json::Value> deserialization which is supported by serde directly, we also capture the error message from Ts deserialization... or validation:

Event validation

I left out one detail in the previous section, and that is that the T in EventResult<T> doesn't actually need to implement Deserialize. Instead, it needs to implement TryFromRaw. This trait is very similar to TryFrom from the standard library, and in fact we would have used TryFrom directly, were it not for coherence issues¹. This trait is implemented for every event (content) type in ruma-events and allows conversion of a private raw version of that type, which does implement Deserialize, to the actual event (content) type. All of this is done to enable validation of events during deserialization, without having to write the actual deserialization code manually. There are currently few events that need this validation step after deserialization (most of the constraints the matrix spec puts on json fields beyond their types are captured in their Rust types, e.g. using UserId instead of String), but it is nevertheless useful to have the required mechanisms in place.

¹ RFC 2451 "Re-Rebalancing Coherence" might allow us to switch to TryFrom in a future release, though I have not tested this yet

Temporary non-support of server-side usage

Because of the whole fallible deserialization thing, we might not be able to use the same Rust types on both client & server anymore. Due to that and the server not being worked on currently, we decided to temporarily remove request deserialization and response serialization from ruma-api and ruma-api-macros, leaving request serialization and response deserialization, as needed for the client use case. This affects ruma-client-api as well, through its use of the aforementioned crates.

Async/await in ruma-client

As listed at the beginning, we published ruma-client 0.3.0-beta.1. This release supports async / await syntax in addition to using all the latest lower-level ruma crates. While async / await will become stable with Rust 1.39.0 on 2019-11-09, tokio 0.2.0 is expected to take some more time to come out, which will in turn affect hyper 0.13.0's final release and thus our 0.3.0 release. Nevertheless, ruma-client 0.3.0-beta.1 can be used today on Rust beta or nightly.

From the editor

I did another live stream of Ruma development on my Twitch channel this past week. If you missed it, you can watch the recording on YouTube. In the stream, I worked on another revision to the ruma-events API in which a special result type was introduced to handle deserialization/validation errors. Jonas quickly discovered, upon trying to update ruma-client-api to ruma-events 0.10, that the new deserialization API didn't quite work and that further changes were needed. I updated all the manually events to the new new API, but the events generated by ruma-events-macros still need to be updated. I will do that on the next stream in the coming week.

A couple weeks ago I recorded an interview with the Rustacean Station podcast, and today the episode was released. You can find it here: Ruma and the Matrix Communication Protocol: An Interview with Jimmy Cuadra. A huge thanks to Ben Striegel and Jon Gjengset for starting the podcast, and to Ben and Abdou Seck for taking the time to record this episode with me. It was a good time, and I hope it gets some new people interested in Matrix and/or Ruma.

From the editor

I forgot to provide an update last week, so this week we've got two weeks of stuff to cover. The revamp of ruma-events I mentioned several times previously was released as 0.13.0, followed shortly by an update to 0.14.0 to bump the url crate (which is a re-exported dependency) to version 2.0. When trying to integrate the new ruma-events into ruma-client-api, Jonas quickly discovered that it wasn't actually possible, because API requests/responses that contain events need to be deserialized, and can't because the events don't have a public implementation of serde::Deserialize anymore. I felt like an idiot for not foreseeing this seemingly obvious problem, but that's the way it goes sometimes. I'll be working on addressing this next week.

A new version of ruma-api, 0.10.0, was released this week. This release contains two significant changes: The first is that the Endpoint trait has been revised to be implemented in terms of the endpoint's request type, rather than having the request be specified via an associated type. This creates a more straightforward mapping between request and response types for each endpoint, removes a bunch of unnecessary unit structs that were only used to link associated request and response types together, and lets us remove some use of the turbofish operator in ruma-client. The second change in this release is that ruma-api-macros has been merged into the repo, and now exists in a Cargo workspace. The ruma_api procedural macro is now included and re-export from ruma-api itself under a default feature. The macro has also been updated to refer to its dependencies via a hidden module in ruma-api. The result is that downstream crates like ruma-client-api no longer have to specify dependnecies on all the crates referenced by the code generated by the macro.

Another exiciting update: This past week, both Jonas and I started live streaming our development on Twitch. I made a separate post on my own blog about this, but the short version is that I intend to live stream most of my work on Ruma from now on to help people learn about both Rust and Matrix. You can find my live streams on my Twitch channel, and you can watch the recordings of past streams on my YouTube channel. I'll announce my streams in #ruma:matrix.org as well as on my Twitter feed. Jonas's live streams can be watched on his Twitch channel.

In the last update, I mentioned that Ruma finally accepts donations via Liberapay. Thank you very much to the people who have already signed on to donate to the project! I thought it wouldn't hurt to mention again in case anyone missed it. :}

From the editor

This week I'd like to officially welcome Jonas Platte as a member of the Ruma team. Jonas has been involved in the project for a long time and has provided many significant contributions, both via code and via discussions about the project's design. Jonas is the original author of ruma-gtk, a graphical Matrix client built on ruma-client, which later became Fest, which in turn was forked to create Fractal. Thank you very much for all your work, Jonas!

A new addition to the website this week is the how to contribute page, which offers an entry point for anyone interested in contributing to the project's software. Thank you to everyone who has expressed interest in contributing in the past (and for everyone who will in the future!) Hopefully this guide will help you get started and give you some ideas for how to help.

On several occasions I've been asked if there is a way to donate financially to the project. I'm happy to report that Ruma now accepts donations via Liberapay. Donations to the project will be divided amongst team members, which as of today consists of only Jonas and me, but will hopefully grow over time. In the past I was hesitant to accept donations because I didn't want financial interest to affect (or have the appearance of affecting) my decisions about the project. I am building Ruma because I think the world needs Matrix, not as a way to support myself. Being able to share donations to Ruma with other developers makes me much more comfortable with accepting donations. It serves both to dilute my own financial interest in the project as well as to provide benefit in a way that's more in tune with the project's goals: making the world better for all of us. For anyone that decides to support the project on Liberapay, I offer my most sincere thanks.

As for the software itself, the main update this week is that the revamp of ruma-events I've been working on for the past month is complete and has been merged into the master branch. I still need to do one quick pass over everything to be sure I didn't make any obvious mistakes, but once that's done it will be ready for a new release. As part of this revamp, a new supporting crate, ruma-events-macros also had its first release this week. Like ruma-api-macros, it's really only used as an internal dependency for the project, and doesn't have much use to other developers directly. But it could be interesting to look at if you want an example of a real-world procedural macro.

There was also a new release of ruma-api, version 0.9.0, which revises the API to support the upcoming futures 0.3, ironically by removing the dependency on futures entirely. This release also removes the library's dependency on hyper, as it's always been a goal for the foundational Ruma libraries to allow other developers to build Matrix software with other HTTP libraries if they wish.

Matrix at large

Riot, the flagship Matrix client, released version 1.3 this week, which includes the ability to edit sent messages, and to add reactions to messages. Both of these features are highly requested and bring Riot in line with other high-profile collaboration software.

Other notable news is the announcement to get Dendrite, a Matrix homeserver written in Go, into a state where it can have some practical use for Matrix users. The first goal is for it to support enough functionality to be used for Matrix bots. Read more about this announcement on the lastest This Week in Matrix.