We have been working on porting Rust to the CHERIoT platform for about half a year now. We have been busy with the implementation of fundamental features such as defining the new target, adapting the compiler to respect the nuances of a CHERI platform – starting with making the compiler aware that the size of an address is not necessarily also the size of the memory representation of a pointer – and making the core and alloc libraries compile to the newly-introduced CHERIoT target.

During this time we never found the chance to share some insights on the work. This post inaugurates a (hopefully) stable stream of weekly updates about the project.

News from the past

In this particular occasion the update will be a bit long, because I want to use this chance to exercise our attention span recount important features from weekly status updates that should have been but were not, that is, all the work that we have done but never found the time to put in a blog post.

Let’s start with some numbers. As of today we have 54 commits in the beta branch of the repository which, as the name suggests, builds on top of the beta branch from upstream Rust. Furthermore, git diff compiler/ tells me that 84 files were touched, 805 lines were inserted and 280 were deleted. Now, before you get angry at me: I know that lines of code are a good metric only when you need to store your source code on a floppy disk, and those number could as well mean that we added 805 wrong lines and removed 280 perfectly good ones.

What I am trying to say here is that the fact that we were able to add a new target to rustc - one with the specific requirements of CHERI - and to make it produce functioning (for what we have tested until now) code with few changes tells a lot about the engineering practices of the Rust compiler. I also think we owe much to the strict-provenance effort: I am sure that it made our job measurably easier.

So, I will pick and explain interesting features from those commits, so that you can get an idea of the work we have done: I hope you have your tea and crumpets ready.

First things first (40bff08, 773c56d)

A compiler can’t do a lot without a backend. The first patch updated the LLVM submodule to point to the CHERIoT port of LLVM. The notable bits: some functions from the C interface of LLVM have a different signature from those upstream. In particular, the functions to create calls to memcpy and memmove need to know whether the bits that are to be copied are capabilities and if the metadata contained therein must be preserved. As of now we delegate this decision to LLVM itself. This is, of course, likely to change in the future. That’s it!

Farewell address space zero (f455d94)

An Address space defines the possible values that can be used to reference an entity in memory. Different address spaces can have different properties. By default, if you don’t specify otherwise, LLVM assumes addresses “live” or refer to address space 0. By convention, when programs in LLVM IR need to refer to pointers whose values are actually capabilities - which also entails that pointers in that space also have different semantics - use the address space 200.

CHERIoT, in particular, uses that address space only. This is because it is a purecap architecture, meaning that it can only handle capabilities: other architectures like Morello can also understand and use (without the memory safety features) canonical “bare” addresses.

Long story short, since not too long ago Rust did not have a way to specify that a target uses a default address space different from addrspace(0). Now that it does, we had to make the compiler use it where needed. This specific patch is interesting because, as far as I can tell, it could be useful for other targets as well, such as amdgpu.

To achieve this, the upstreamed PR makes the compiler take into account the relevant bits of the datalayout string that specify the kinds and properties of the address spaces valid on a specific target.

Compiler Wars: Episode MMMMXXVI - A new target (cde4496)

This commit made us able to run rustc --target=riscv32cheriot-unknown-cheriotrtos. Not a lot to say here, because it is actually a fairly small change and we needed a bit more work before we could actually compile any code to CHERIoT, but it was the first step towards that goal.

Play “:%s/pointer_width/pointer_offset/gr”, Sam (8b44ba5, f46ab36, 0e1fc52, 8778ccf)

If you aren’t too familiar with CHERIoT, there are a lot of things you might be interested to know about it. One that is relevant now is that on CHERIoT addresses are 32 bits, but capabilities are 64 bits: 4 bytes contain the proper address you need to refer to something in memory, while the remaining 4 bytes contain metadata which, roughly, tells you what you can do with that specific capability.

This is, of course, pretty different from many currently popular architectures that Rust supports. In general, the compiler often makes the assumption that the bits it takes to store a pointer are exactly the same as the bits of a memory address.

We had to teach the compiler that the two concepts can be different; to do so we used a byproduct of the same PR that introduced the ability to specify different default address spaces for a target. In fact, the data layout string can also contain a value that specifies the size of indices that can be applied to addresses in a given address space. (I know, pointer_offset is not the best name.)

One example of the nature of these changes is this:

fn int_ty_max(&self, int_ty: IntTy) -> u128 {
        match int_ty {
            IntTy::Isize => self.tcx.data_layout.pointer_offset().signed_int_max()
            ...

That function used .data_layout.pointer_size() before. For CHERIoT, that would return u64::MAX instead of u32::MAX: this latter value is effectively what we want here. This change does not impact targets where the default address space is integral. While this is the change we needed and made most sense for this use-case, I expect that this set of changes will (rightfully) require thorough and in-depth discussions with the compiler and language teams.

Diane, 11:30 a.m., February 15th. Entering the town of Twin Peaks (9c89bfe, d99b5fa)

I am skipping a few commits ahead, some of which are actually very interesting: splitting the size method of the internal representation of scalars in two, one to refer to how many bits of data it can fit and one to refer to how many bits it takes to store the scalar in memory (for the same reason as before); adding CHERI-specific intrinsics and using them in core; adapting the generation of discriminants to be aware of the distinctions between the size of a pointer and an address; using non-transmuting casts in MIR passes (this one needs to be fixed to use non-exposing casts!) and much more. I can’t go over all of them now – I think this post is getting pretty long as it is – but if you’d like to ask questions or learn more about the work, feel free to reach out to us in the public CHERIoT chat on Signal or raise an issue on the cheri-rust repository.

Anyways, the commits that give the title to this subsection are those in which we added steps to build core and run the codegen-llvm tests for our new CHERIoT target to our CI. This marked the moment when we paused the efforts to add new features, and focused primarily on verifying that the code we generate for CHERIoT makes sense and matches what Rust thinks it should be, and investigating the bugs we found in the process.

News from a more recent past (last week)

I will tell you, now, what we worked on this last week, and the unlawful imprisonment will be over shortly after.

An actually fun thing I hope we will have the chance to do again (#111)

We have been investigating issue #108 for a couple days. Consider this snippet:

let x = 42;
assert_eq!(alloc::format!("{x:b}"), "101010");

When compiling and executing it on a CHERIoT simulator it worked perfectly fine. This snippet, on the other hand, did not:

let x = 42;
assert_eq!(alloc::format!("{x:#b}"), "0b101010");

The only difference is the use of the “alternate” flag in the format (i.e. {x:b} vs {x:#b}, which in this case means to prefix the integer with 0b), but the latter crashed with this message:

Error handler: PermitExecuteViolation(0x11) error at 0x8000f3fa ...

Looking at the dump of the firmware, we understood that the error was coming from the f.buf.write_str(prefix) call here. Nothing obvious came up when looking at the Rust code or the LLVM IR, so we took a (very) long look at the assembly that was triggering the error:

;             if let Some(prefix) = prefix { f.buf.write_str(prefix) } else { Ok(()) }
80017f68: ce81         	beqz	a3, 0x80017f80 <<core::fmt::Formatter>::pad_integral::write_prefix+0x68>
80017f6a: 6d84         	ct.clc	s1, 0x18(a1)
80017f6c: fea7855b     	ct.cmove	a0, a5
80017f70: fea685db     	ct.cmove	a1, a3
80017f74: 863a         	mv	a2, a4
80017f76: 70a2         	ct.clc	ra, 0x28(sp)
80017f78: 7402         	ct.clc	s0, 0x20(sp)
80017f7a: 64e2         	ct.clc	s1, 0x18(sp)
80017f7c: 6145         	ct.cincoffset	sp, sp, 0x30
80017f7e: 8482         	ct.cjr	s1
80017f80: 4501         	li	a0, 0x0

Notice anything weird? No? Really? Sure? Try again. What about now? Nothing? Alright. So, let’s go line by line, starting with beqz a3, 0x80017f80. The prefix argument is of type Option<&str>, and Rust niche-optimises the representation of this value: null (0) means None, otherwise we are in the Some(prefix) path. In this path execution continues to ct.clc s1, 0x18(a1), which means: load the capability at offset 0x18 from the capability contained in a1 into s1. This is the address of the buf.write_str function. Execution continues, then, loading the arguments it needs to pass to the write_str function. After that we can see the epilogue of a function before a tail call: it restores ra, s0 and s1 and then jumps to the address of buf.write_str. But, wait, we just changed the value of s1!

In short, what we discovered is that there was a bug in CHERIoT-LLVM where the virtual register rewriter assigned a callee-saved register to store the value of the address to jump to for a tail call (which was promptly fixed, by the way). This means that it could occur that the register containing the address to jump to, where execution will continue, would be overwritten with another value, potentially making CHERIoT trap at runtime. This was cool, right?!

I’m out of fun titles - generating the correct e_flags (e058b31)

The object crate does not have definitions for CHERI- or CHERIoT-specific e_flags. We created a new organisation on GitHub where we host patched forks of the third-party crates. We updated our rustc to use the patched version of object and the correct e_flags.

Let there be atomics (15e149e)

As we mentioned before, Rust knows that our pointers are 64 bits wide, although CHERIoT is a 32-bit platform. This means that to have AtomicPtr from core, we need to tell rustc that the platform supports atomic operations on values of the same size as a pointer. We have AtomicPtr now!

Having a runner is not too useful if I can’t see why it fails (d4b3b4d)

We don’t only run codegen-llvm tests (which actually do not execute code on a CHERIoT implementation, but just compare the generated IR), we also have a small selection of tests we build and run on the Sail simulator for CHERIoT. We use a custom runner to execute them, and there was a bug that prevented exceptions from being printed in specific cases. It is now fixed, yay!

Conclusion

If you have made it this far, you definitely deserve a cookie, but I ate ‘em all while writing this, sorry. I’ll try to keep these updates coming on a fairly regular schedule, and the next ones will be shorter and more in-depth. For now, let me conclude saying that we are very happy with the current status of the project, and we know we have a lot more work to do. All the work happens in the public cheri-rust repository. If you want to try it out, here is a one-liner:

git clone https://github.com/CHERIoT-Platform/cheri-rust.git &&\
    cd cheri-rust &&\
    ./cheri/gen_bootstrap.sh --build-clang &&\
    ./x build compiler std --target=riscv32cheriot-unknown-cheriotrtos

You can find here instructions to do something with the compiler after you have built it. Let us know how it went on the public CHERIoT group on Signal or, if something went wrong, file an issue please!

Bye now!