Recently, types from the ruma-identifiers crate were integrated into ruma-events and ruma, so that values representing a Matrix ID can guarantee some important invariants.
Previously these values were just strings.
There was still a source of confusion and errors in that the ID types had to be converted to and from strings for the structs that represent database records using the Diesel ORM.
To fix this, I added the necessary implementations to ruma-identifiers (under the "diesel" feature) so that the ID types can serve directly as the database columns' data types.
Of course, these are still stored as text in the database, but the conversion happens only during data serialization and deserialization, and so cannot be broken by application logic.
This was a pretty hairy change and it took a while to get it all working and to get the test suite updated and passing.
Big thanks to Sean Griffin (author of Diesel) for helping me figure out how to achieve this.
Notable changes to ruma
- Increase the portability of the Cargo wrapper script.
- Use ruma-identifiers ID types for relevant database columns.
- Crate version 0.4.0 and then 0.4.1 were released, adding Diesel integration and a bug fix for case-sensitivity in user IDs.
New contributors
Rust at large
- Sean Griffin shared some details of generic implementations that don't work right now, but ideally could help make it easier for apps to create their own custom Diesel data types.
- An important step for the future prospects of moving Ruma to stable Rust, the implementation for macros 1.1 will land soon.
This will allow Diesel and Serde to use custom derive without requiring nightly features, which is one of the main reasons Ruma targets nightly Rust.
Apologies for the missed update last week!
I was sick and didn't get a chance to write it.
However, there was work on the project last week.
I will just include those changes in this update.
Notable changes to ruma
- Previous work to update Diesel and Serde to the latest versions have been completed.
- Ruma now uses the types from ruma-identifiers to represent Matrix IDs instead of strings.
- Ruma now uses the latest version of ruma-events which also uses ruma-identifiers.
- The Docker Compose file now locks the Ruma development image to a specific version, so breaking changes to the image are now coordinated with Ruma.
As part of the same change, the image is now locked to a specific version of the nightly compiler.
- Database migrations are now embedded in the ruma binary, which means a check out of the source code is no longer necessary.
- Add initial support for the message event creation API.
This is a big milestone and one of the core APIs that makes Matrix work!
- Added simple type aliases for custom events of each kind (basic, room, and state events).
- I'm experimenting again with a version of the API where the event kinds are traits instead of generic structs.
They really feel like they should be traits, but the actual usage pattern for the events in the main ruma application may not work well like this.
- Crate version 0.3.0 was released, which implements
Clone for the ID types and Copy for the error type.
- I'm working on a change that would allow the ID types to be used directly as struct fields for Diesel table types.
Right now it seems like more trouble than it's worth, but I'm following the issue on user-defined
FromSql/ToSql types.
Matrix at large
- Paul Evans of the Matrix team is working on testing Ruma with SyTest, a blackbox integration testing framework for Matrix homeservers.
Even though few of the APIs are complete in Ruma so far, there are already a few tests passing!
- SPEC-442: Explain requirements and guarantees around transaction IDs
- SPEC-444: Add an API for getting auth flows
The May 22, 2016 news update mentioned that Vector and Synapse use unspecified APIs for negotiating authentication flows, but an issue was never filed for improving that.
Rust at large
- Starting to think about how permissions checks will work in Ruma, I opened an issue for transaction isolation support in Diesel as a possible way to prevent ToCToU errors.
This week I began working on upgrading the libraries used by Ruma.
In particular, Diesel and Serde have new minor versions (which prior to 1.0 are effectively major versions.)
I'm also working on integrating ruma-identifiers and the recent round of changes to ruma-events back into Ruma.
Because of library changes, there are a lot of code updates needed.
I got the compiler errors down from about 60 to about 10, but there are a few blocking issues to fix the rest.
These are noted in the section on Rust below.
- Replace macro for enum implementation with more straightforward usage of Serde annotations where possible.
This means that
Display and FromStr are no longer implemented for these types, but they may not be used by Ruma anyway.
They'll be added back if and when there is an actual need.
Rust at large
- diesel-rs/diesel#394 Remove the
Copy constraint from Identifiable.
In Diesel 0.7.0, struct fields representing a database table's primary key must be Copy.
Because strings are used for this in Ruma, this PR must be merged in order to avoid a bunch of otherwise unnecessary newtypes in Ruma.
- iron/body-parser#72: Bump Serde to 0.8.
Needed due to version conflicts with
serde_json::Value.
This week serde 0.8.0 was released, offering a new serialization API.
Most of the work on Ruma was towards serialization of the ID and event types.
This will support the next step of implementing APIs in the homeserver that take events as user input.
In addition to the changes landed in ruma-events, I've been working on figuring out how to approach Matrix types that are represented as enums in Rust code but involve checking the value of a key in the JSON structure of the serialized form to determine which variant it is.
I hope to get this nailed down and committed by next week's update.
- Implement serialization and deserialization for ID types.
- Change the
new constructor to an implementation of TryFrom and add a separate generate constructor for generating new IDs with random localparts.
- Begin running the test suite on Travis CI.
- Publish the crate to crates.io.
ruma-identifiers is the first Ruma component to get a crates.io release!
- Convert the types of struct fields that represent a Matrix ID from
String to the relevant type from ruma-identifiers.
- Manually implement serialization and deserialization for a few enum types where the variant name appears in JSON camel cased.
This was initially done by copy/pasting similar code between types, but was replaced with a macro that reduced a lot of duplication (and actually increased the test suite coverage by testing every single variant of each enum.)
- Begin running the test suite on Travis CI.
This was a relatively quiet week in terms of code changes.
I'm working on nailing down ruma-identifiers in preparation for using it in ruma-events before I start working on serialization in ruma-events, which is needed for the next step in ruma itself.
Serde is about to release 0.8, which has some breaking changes including a revamped API for manually implementing serialization.
I started to learn how it works since I'll need to use it as soon as it's released.
- Add
EventId, RoomId, and RoomAliasId.
Matrix at large
- I discovered that the new spec introduction that includes the grammar for Matrix IDs is missing some important details about event and room IDs.
The spec currently defines them as
sigil opaque_id ":" server_name but does not define opaque_id at all.
Certain limitations are important to note for the purposes of parsing and compatibility, such as allowed character set and minimum and maximum lengths.
I think it's also important to disallow characters that would be problematic for display or parsing, such as whitespace, newlines, and colons, which are also used as the delimiter between opaque ID, IP/hostname, and port.
There was already a JIRA issue for specifying these types of IDs, SPEC-389, so I added my concerns there.
Rust at large
- rustup 0.4 was released, which contains the change I made previously for allowing arbitrary toolchains to be installed alongside rustup itself.
I'll be using this to improve Ruma's Docker development image.
- serde-rs/serde#462: A simple doc fix I made to Serde while looking through the 0.8 changes.
- rust-lang/rfcs#1681: Procedural macros 1.1.
Only tangentially related to Ruma, but an important effort that could help in getting Ruma onto stable Rust.