News News feed (Atom)

GSoC 2021: API Coverage Final Summary

› published on by

Time Flies

It's been 10 weeks already! This year's Google Summer of Code is coming to an end. I really enjoyed working on this project, which turned out to be very varied in what parts of Ruma I worked with: I've added endpoints, modified and added new event types, created new identifier types and worked on macros.

This year for GSoC I had a project of extending Ruma's API coverage. More specifically, I would complete Ruma's coverage of Matrix's Identity Service API. On this front, I was successful, and now Ruma has 100% coverage of those endpoints! I also had plans to implement new MSCs, namely MSC1946: Secure Secret Storage and Sharing and MSC2785: Event Notification Attributes and Actions. Along the way, I would also work on resolving a few issues here and there, which helped get myself better acquainted with Ruma's codebase.

Working on Ruma was a good learning experience for me. I had an opportunity to do actual work on a project which is used in the real world. All the while, I was learning and practicing many skills. Working with git, coding against a formal specification (and dealing with the problems caused by this). Although I did learn a lot at University, this was a very useful hands-on experience. And as a bonus, I got to program in Rust for the whole Summer! Rust is a language I particularly enjoy using, but which I unfortunately rarely have the opportunity to use at school.

The Final Parts of the Project

The end of my project did not turn out to be as smooth as planned. As mentioned in the previous blog post. I had originally planned to work on adding support for Secure Secret Storage and Sharing, introduced by MSC1946. Although the sharing part was possible to implement with the current macros available in Ruma, storage requires a major refactoring of these macros. This means that although I was able to do the groundwork for supporting MSC1946, secret storage will not be usable until those macros are reworked.

The reason for all this trouble is that MSC1946 introduces an event m.secret_storage.profile.[key_ID] where [key_ID] is a placeholder for the ID of a key to be stored. This means there could be any number of event types possible, instead of the until now predefined set of event types. I thus opened an issue for this to be supported in Ruma.

Although secret storage will need to wait to be merged, secret sharing has already been merged. This part of MSC1946, as its name suggests, allows clients to share secrets between devices. As such, keys can be exchanged between devices using the part of the Matrix Spec introduced by this MSC. In fact, matrix-rust-sdk has already implemented the gossiping of cross signing keys, building on top of Ruma's implementation of Secret Sharing.

As it wasn't realistic to include the major macro refactor in my GSoC project, I decided to move onto MSC2785: Event notification attributes and actions. I had kept this as an extra goal for my project, and seemed to be a good fit for the situation. Unluckily though, when I read more into the MSC's text, I realized I would encounter the exact same problem as I had with Secret Storage. This MSC introduced the event type m.notifications_profile.<profile> where <profile> is the name of the profile to store. I still powered ahead, implementing the different endpoints and events, temporarily setting a fixed event type name for m.notifications_profile.<profile>. This allowed the grunt of the work to be done, meaning it would be ready for when Ruma gets support for variable event names. Although the MSC isn't finalized as of yet, all the currently laid out endpoints and events have been implemented in a branch ready to be merged once possible.

Thus there are a couple of pull requests which are blocked and that will be merged in the future, once variable event type names are supported:

Finally, I also worked on this issue:

To solve this last issue, I decided to learn how to write procedural macros. In the first half of GSoC, I had created a new macro_rules! macro, largely based on a pre-existing one. As I had very little experience in writing macros, and zero experience in writing procedural ones, I decided to read a short series of blog posts to get myself up to speed. Once that was done, I got to work: I added the generation of From trait implementations inside the pre-existing proc-macro code which generate the event enums. I also added a derive macro for enums created manually. I really enjoyed working on this, which was my first foray into Rust's procedural macros.

Recap of PRs for my GSoC

Start of GSoC with the Identity Service API

Middle of GSoc, implementing MSCs and adding types

End of GSoC

GSoC 2021: Tooling Final Summary

› published on by

What a fun summer! I was hoping I would become more a part of the Rust community this year, and I think I have. I learned how to open an MCP (Major Change Proposal), which is a process that language and implementation changes not big enough for an RFC go through. I am involved in an effort to create a new plugin lint system similar to Clippy. I made a few more contributions to Clippy, fixing old lints and adding a new one. I am happy to become more involved in Ruma and Rust and plan to finish all my works in progress.

Since I am working with more people than just my mentor, making progress seems to take longer. I am glad for the experience working with multiple people, learning how to follow directions from multiple people. You have to be able to write code in the "voice" of someone else or at least meet them halfway. I know that this is an invaluable skill that will serve me well in my career. Thank you to @jplatte and @iinuwa for all the help and encouragement!

Starting on Automation

Second Half

Progress on Automated Checks for Ruma

I think the work on the linter will be extremely valuable if we can get it working. The ability to write crate-specific lints will be a huge help to library writers and users. I can imagine many such lints for a complex library like Ruma. I am excited to be involved!

Google Summer of Code 2021 Intro

› published on by

Hello all! This year, Ruma has the pleasure of having two students for Google Summer of Code (GSoC) 2021: Devin (who worked on Ruma's proc-macro code during last year's GSoC) and Adam. We'll let them [re]introduce themselves.

Ruma API Coverage GSoC

by Adam Blanchet

This year for GSoC, Ruma has two students! I am the "newer" one of the two, and my name is Adam Blanchet. I've been at work adding support for more of the Matrix Spec in the past few weeks, and will be continuing on doing so for the remainder of GSoC.

Identity Service API

Although I had made progress before the official coding period for GSoC had started, I continued with finishing the remaining unimplemented Identity Service API endpoints:

Apart from the 3PID unbind endpoint which is currently blocked, this concludes the full coverage of the Identity Service API. This means that now projects built upon Ruma will be able to more easily integrate functionality with identity servers. This could also enable easier development of an identity server written in Rust!

Implementing MSCs and adding types

I then moved on to a few other miscellaneous issues, to get accustomed to working on the other parts of the Ruma project:

  • Add client secret and session identifier types: I found this PR to be more interesting than its title may indicate. Not only did I add two identifier types, but I added a new macro to create validated identifier types. The new macro was also used to greatly simplify the declaration of ServerName struct. I really enjoyed working on this PR. It was an occasion to create a new macro to simplify the creation of these and new types. I had never written macro code before, and it was a good occasion for me to learn something new!

  • Add RoomName struct: This one did not involve macro code, though similar in principle to the previous PR. It adds a RoomName struct to be used in an event, which enforces a maximum length.

  • Update endpoints for Blurhash implementation: MSC2488 adds the use of Blurhashes to the Matrix Spec. Although some Blurhash support had been previously added to Ruma, I extended this to all endpoints and events specified in the MSC.

  • Add "knock" feature from MSC2403: This adds support for knocking on rooms, a feature which I'm looking forward to see implemented in clients. It was introduced in MSC2403.

  • Implement reasons for leaving a room: Adding this feature, introduced in MSC1983, seemed rather straightforward. However, I bumped into an issue with a conditionally compiled field with a lifetime annotation. Thankfully, @DevinR528 helped diagnose the problem which was some macro code, for which they think they have found a solution along with @jplatte.

    Once that fix is implemented, we will also be able to merge my PR.

What's next?

I'm looking forward to implementing the MSC for Secure Secret Storage and Sharing. This is the second large part of my project, it will involve adding a number of events to support storing and sharing both keys and secrets. To achieve full support, I would also need to implement crypto routines. I expect that adding these will require more effort than adding the events, though I look forward to the challenge.

I found my first half of GSoC to be very enriching. I really enjoyed working on this project: I got to learn how to read and apply a specification, and I learned the rigor required for a project such as Ruma. I have sometimes been caught off-guard: sometimes I miss extraneous newlines, sometimes (often) I forget to add a changelog entry, and other similar more stylistic rather than functional issues. Ruma is a big project with many crates, yet it maintains consistency throughout, a quality I greatly appreciate. I now know that this consistency comes from the thoroughness with which each change is checked, and I hope to improve my own rigor when it comes to each of my pull requests.

Ruma Tooling GSoC

by Devin Ragotzy

It's been a busy few weeks! I have started the process of checking four todo items off of Ruma's automated checks issue by contributing lints to the Clippy project.

I temporarily broke Clippy, Oh No! Luckily, the fix seems to have worked.

I've spent a lot of time familiarizing myself with the rustc usefulness checker. The is how the Rust compiler determines if you have a match for all possible patterns and checks if any of the patterns are useless.

let x = true;
match x { // `x` is matched exhaustively (all match patterns are useful)
    true => {},
    false => {},
    // true => {}, would be un-useful

This is where I now plan to add a lint to check for missed patterns on structs and enums that are marked non_exhaustive (see issue). This is helpful to Ruma because all of our public types are marked in this way, allowing us to update our types in a non-breaking way while still informing users of the change. Also, for anyone interested, I saw an interesting blog post about usefulness checking.

For the last few days, I have been working with my mentor @jplatte to find and fix a very sneaky issue with conditionally compiled fields and lifetime annotations. This was found by my fellow GSoC participant in this PR. I think we have finally found a solution, fingers crossed.

I have thoroughly enjoyed the first half of this year's GSoC! I'm excited to contribute to back the compiler for the language we all love so much ;). In the second half of GSoC I hope to get ruma-check to a usable state adding the few checks left from Ruma's automated checks.

Devin and Adam have already made it through the first half of GSoC and have made good progress. We're looking forward to the rest of the summer with these two!

Google Summer of Code Wrap-up

› published on by

Wow the summer has flown by, it feels like just yesterday I was learning how to rebase and what exactly it is Ruma does. I exaggerate slightly, but it is a big library with lots of public API surface. I have learned more in the last few months than in two years of school. I have been able to observe and participate in a project with a community growing around it, been a part of discussions about design and best practices, given and received numerous code reviews as well as learned the process of addressing the feedback, and working from a specification. In short, this has been an amazing opportunity to gain experience in all the things that are hard to obtain in a classroom.

My project goal was to improve the existing macros in ruma-events-macros and ruma-api-macros. It became clear early on that this would include some major API changes and that improving the macros as they were was pointless without also moving to a new public API. While improving the durability and readability of the macro code I also rewrote entire sections to accommodate the new design.

A quick overview of the Matrix protocol for reference: a client sends content that is interpreted by the server as events. The server distributes those events out to other clients and other servers (the server case is known as federation). Ruma groups these events by kind Message, State, Ephemeral, ToDevice and Basic which are represented as generic structs (StateEvent<C>). Each event kind needs to be able to hold many different content types, for state events there is room creation, room name, and membership events to name a few. Using the macros, enums are generated to represent all state event possibilities, so a variant for membership, room name, etc. These types exist to support the core API request and response types for each endpoint that is defined by the Matrix specification.

GSoC Starts and I start with ruma-events

More work on ruma-events, now a part of the new mono-repo

Work on ruma-client-api

Back to ruma-events

Continuing maintenance

One of my personal goals was to become more familiar with git. With the help of my mentor I now feel more confident using this tool that is so essential to developers. I became fairly adept at merging, rebasing, and navigating all the headaches that come with that. I learned plenty of new commands. A few highlights: cherry-pick and specific uses of reset to avoid copy-pasting fixes and adding more commits. I used the reset command to craft good commits, splitting work into appropriate chunks. I am glad that I had the opportunity to hone my git skills. I feel like I have accomplished my goal and then some!

I am proud of the work that I have done: Being part of moving ruma-events much closer to the 0.22 release and creating macros to generate types specific to the Matrix specification. Working with the community that has grown around Ruma has been rewarding and I plan on sticking around.

The End is Nigh

› published on by

This week in the ruma/matrix Google Summer of Code project, I worked on refactoring both ruma-api and ruma-events. After moving some of the larger chunks of the ruma_api_macro::api::Api::to_tokens method to helper functions, I spent time removing repetition from the Request/Response code generated by the ruma_api! macro. For ruma-events, the input parsing was changed to only allow valid names for the Any*Event enums. Altering the input parsing had the added benefit of replacing all of the string comparison and manipulation with strongly typed comparison and manipulation.

The final few issues to be resolved before the next release for ruma-events can happen are related to redacted events. Support for redacted events was added to the Any*Event enums, they now have redacted variants of each event kind. A few follow-up PR's have been merged to fully integrate redacted events into ruma-events, fixing specific event deserialization issues and splitting the UnsignedData struct into Unsigned and RedactedUnsigned.