martin/musicmouse
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add a native Slint front-end for the music player

Browse Source 
The React UI is sluggish on the kiosk and the reasons are browser-shaped: a compositor
deciding what gets its own layer, a requestAnimationFrame loop repainting the viewport,
and a JPEG decode per album card per cold start. The last three commits chased that
through the service worker and the cover pipeline. This is the other direction - the
same backend, the same layout, the same German copy, without a browser.

Browse and play only. The typing game, the room-lights page, parent mode and the
IR-remote assignment stay web-only, and nothing here touches python-backend.

Four things carry the performance claim, in rough order of how much they should matter:

Covers are downscaled once, ever. The backend serves one size - 640px on the long edge,
no ?size= - and a grid card is 180. src/covers.rs fetches each cover once on a worker
thread, resizes it, and writes a JPEG thumbnail to ~/.cache/musicmouse-slint/covers/.
Two tiers and only two, which is what makes the cache worth keeping: a per-widget pixel
size would give each album a dozen near-identical files and a fresh decode for each. The
cache is dropped when the websocket announces a rescan, because that is the one moment
the backend rewrites the art behind an unchanged cover URL - the trap sw.js had to learn
about the hard way.

The grid is virtualized. Rust hands the UI the album list pre-chunked into rows and the
view puts those in a ListView, which instantiates only what is on screen. A flat list of
660 cards gives it no rows to skip, hence the chunking. Same purpose as
content-visibility: auto on .grid > .card.

The progress bar animates between the 2 Hz pushes rather than running a clock, so
interpolation costs a property evaluation per frame on the render side and there is no
equivalent of usePlaybackClock. It is suppressed for the frame a track changes on, so a
new track jumps instead of sliding across two unrelated positions.

Nothing on screen animates by itself. The ambient canvas is not ported, for the reason
lib/lowPower.ts already gives: a loop repainting the viewport is a floor you cannot get
under while it runs at all.

The device must not use FemtoVG. On Mesa V3D it draws every runtime-loaded Image as
solid black (slint-ui/slint#11785, open), which here means every album cover; Skia and
the software renderer are unaffected. So `kiosk` is linuxkms + Skia and FemtoVG stays
the default only for desktop development. Both profiles are verified to build; the kiosk
binary links libinput/libgbm/libdrm and no X11 or Wayland at all.

lib/search.ts, lib/keyboard.ts and lib/format.ts were already pure functions with their
own tests, so they port across as pure Rust with theirs: 43 tests, no window required.
The .slint files are layout only - nothing in them formats a number or picks a word.

Verified against the real 660-album library rather than a fixture. tools/headless-shots.sh
renders the UI inside a nested headless compositor and grabs a frame per screen, which
is how that was checked on a machine whose session was locked; a Slint window needs a
real compositor, and under bare Xvfb nothing maps at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Martin Bauer
2026-09-20 22:26:09 +02:00
parent 83d16e0058
commit 6ccb3e458f
21 changed files with 11270 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

406
slint-frontend/src/covers.rs Normal file
View File

@@ -0,0 +1,406 @@
//! Album art: fetched once, downscaled once, then reused forever.
//!
//! The backend serves one size and only one - `MAX_COVER_PX = 640` on the longest edge,
//! JPEG, at `/api/albums/{id}/cover`, with no `?size=` to ask for less. A browse grid
//! draws that at 192 px and the play view at 384, so painting the served file directly
//! means decoding a 640x640 JPEG for every visible card on every cold start. At 343
//! albums on a Pi that is the whole of "the UI feels slow".
//!
//! So this keeps a thumbnail cache on disk, keyed by album id *and* the size that was
//! asked for, and the decode-and-resize cost is paid once ever rather than once per
//! launch. A miss goes to the network on a worker thread; the UI never blocks on one.
//!
//! **Invalidation.** The bytes behind a cover URL do change while the URL does not -
//! the id addresses which album the art belongs to, not which art. Rather than pay a
//! conditional request per cover per start, this cache is cleared wholesale when the
//! websocket announces a rescan (`{"type": "library"}`), which is the one moment the
//! backend reprocesses cover art. See the long comment at
//! `python-backend/musicmouse/services/web/api.py:106` for how the web client learned
//! this the hard way.
use std::collections::{HashMap, HashSet};
use std::path::PathBuf;
use std::sync::mpsc::{Receiver, Sender};
use image::imageops::FilterType;
use image::{Rgba, RgbaImage};
use crate::api::{AlbumKind, Client};
use crate::library::Group;
use crate::oklch::{mix, oklch_to_rgb, parse_hex};
/// The two sizes anything on screen actually asks for. Keeping it to two is what makes
/// the disk cache worth having: a per-widget pixel size would give every album a dozen
/// near-identical files and a fresh decode for each.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Tier {
/// Grid cards, shelf tiles, track rows and the player bar's 56 px thumbnail, which
/// scales down from this rather than earning a tier of its own.
Thumb,
/// The play view's cover, and the album sheet's.
Hero,
}
impl Tier {
pub fn px(self) -> u32 {
match self {
Self::Thumb => 192,
Self::Hero => 384,
}
}
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct Key {
pub album_id: String,
pub tier: Tier,
}
/// Everything the worker needs about an album, flattened so it can cross a thread
/// boundary without the `Rc<Album>` the UI side holds.
#[derive(Debug, Clone)]
pub struct Request {
pub album_id: String,
pub tier: Tier,
pub has_cover: bool,
pub is_book: bool,
pub locked: bool,
pub colors: [String; 3],
}
/// A finished bitmap on its way back to the UI thread. Raw RGBA rather than a
/// `slint::Image`, because the image types are built on the UI thread by whoever
/// receives this.
pub struct Ready {
pub key: Key,
pub width: u32,
pub height: u32,
pub rgba: Vec<u8>,
}
// ------------------------------------------------------------------ generated art --
/// Diagonal two-tone stripes: the stand-in for a music cover, and the thing that shows
/// through underneath one that has not arrived yet.
fn stripes(width: u32, height: u32, primary: [u8; 3], secondary: [u8; 3]) -> RgbaImage {
// `max(6, size / 13)` in `web/src/lib/covers.ts` - the stripe scales with the art so
// a 56 px thumbnail does not turn into a grey smear of forty bands.
let band = (width / 13).max(6) as f32;
RgbaImage::from_fn(width, height, |x, y| {
// 135deg in CSS points the gradient line at the bottom-right corner, so the
// position along it is the projection of (x, y) onto (1, 1)/sqrt(2).
let along = (x + y) as f32 * std::f32::consts::FRAC_1_SQRT_2;
let c = if ((along / band) as u32).is_multiple_of(2) {
primary
} else {
secondary
};
Rgba([c[0], c[1], c[2], 255])
})
}
/// A book: page edges down the right, a darker board down the left, over a diagonal
/// wash. Four layers in CSS, four passes here, in the same order.
fn spine(width: u32, height: u32, primary: [u8; 3], secondary: [u8; 3]) -> RgbaImage {
// The audiobook shelf's own hue - warm amber, not a duller yellow-brown - so a
// generated spine and the shelf it sits on are the same family by construction.
let hue = Group::Audiobooks.hue();
let page_light = oklch_to_rgb(0.97, 0.02, hue);
let page_dark = oklch_to_rgb(0.90, 0.03, hue);
RgbaImage::from_fn(width, height, |x, y| {
let fx = x as f32 / width.max(1) as f32;
let fy = y as f32 / height.max(1) as f32;
// Layer 4 (bottom): linear-gradient(155deg, primary -> secondary).
let t = ((fx * 0.42 + fy * 0.91) / 1.33).clamp(0.0, 1.0);
let mut c = mix(primary, secondary, t);
// Layer 3: the board down the left edge.
if fx < 0.11 {
c = secondary;
} else if fx < 0.13 {
c = primary;
}
// Layer 2: two thin highlight rules over the board.
if (0.035..0.043).contains(&fx) || (0.070..0.078).contains(&fx) {
c = mix(c, [255, 255, 255], 0.35);
}
// Layer 1 (top): the page block on the right.
if fx >= 0.975 {
c = page_dark;
} else if fx >= 0.95 {
c = page_light;
}
Rgba([c[0], c[1], c[2], 255])
})
}
/// What an album looks like before - or instead of - its real art.
pub fn generated(request: &Request) -> RgbaImage {
let primary = parse_hex(&request.colors[0], [0x4a, 0x6f, 0xa5]);
let secondary = parse_hex(&request.colors[1], [0x6a, 0x8f, 0xc5]);
let px = request.tier.px();
let (width, height) = dimensions(px, request.is_book);
if request.is_book {
spine(width, height, primary, secondary)
} else {
stripes(width, height, primary, secondary)
}
}
/// Shape comes from the media type and from nowhere else: albums square, audiobooks
/// taller than wide. `px` fixes the longest edge either way.
fn dimensions(px: u32, is_book: bool) -> (u32, u32) {
let kind = if is_book {
AlbumKind::Book
} else {
AlbumKind::Music
};
(((px as f32) * kind.aspect()).round() as u32, px)
}
// ------------------------------------------------------------------------- worker --
pub struct Worker {
client: Client,
dir: Option<PathBuf>,
}
impl Worker {
pub fn new(client: Client) -> Self {
let dir = dirs::cache_dir().map(|d| d.join("musicmouse-slint").join("covers"));
if let Some(dir) = &dir {
if let Err(e) = std::fs::create_dir_all(dir) {
// Not fatal: without a disk cache every cover is simply fetched and
// resized again next launch, which is the old behaviour, not a failure.
eprintln!("cover cache unavailable at {}: {e}", dir.display());
}
}
Self { client, dir }
}
fn path(&self, key: &Key) -> Option<PathBuf> {
self.dir
.as_ref()
.map(|d| d.join(format!("{}@{}.jpg", key.album_id, key.tier.px())))
}
/// Drop every stored thumbnail. Called when the backend says it has rescanned, and
/// therefore possibly rewritten the art behind these ids.
pub fn clear_disk(&self) {
let Some(dir) = &self.dir else { return };
if let Ok(entries) = std::fs::read_dir(dir) {
for entry in entries.flatten() {
let _ = std::fs::remove_file(entry.path());
}
}
}
fn render(&self, request: &Request) -> RgbaImage {
let px = request.tier.px();
let (want_w, want_h) = dimensions(px, request.is_book);
// A locked album is one the typing game is withholding. It has real art, and
// showing it would give away the reward, so it never gets fetched at all.
if request.locked || !request.has_cover {
return generated(request);
}
if let Some(path) = self.path(&Key {
album_id: request.album_id.clone(),
tier: request.tier,
}) {
if let Ok(image) = image::open(&path) {
return image.to_rgba8();
}
}
let bytes = match self.client.cover(&request.album_id) {
Ok(Some(bytes)) => bytes,
// 404 is a normal answer - `has_cover` can be stale by a rescan - and a
// network error should show the album, not a hole.
Ok(None) | Err(_) => return generated(request),
};
let Ok(decoded) = image::load_from_memory(&bytes) else {
return generated(request);
};
// `resize_to_fill` rather than `resize`: a real cover is square but a book's
// slot is not, and letterboxing one inside the other would break the shelf
// metaphor that the whole browse screen is built on.
let thumb = decoded.resize_to_fill(want_w, want_h, FilterType::CatmullRom);
if let Some(path) = self.path(&Key {
album_id: request.album_id.clone(),
tier: request.tier,
}) {
// Storing JPEG, not PNG: these are photographs, so JPEG is both smaller on
// disk and faster to decode on the next start, which is the whole point.
let _ = thumb
.to_rgb8()
.save_with_format(&path, image::ImageFormat::Jpeg);
}
thumb.to_rgba8()
}
}
/// Runs the fetch/decode/resize loop off the UI thread, calling `deliver` for each
/// finished bitmap. `deliver` is expected to hop back onto the Slint event loop.
pub fn spawn(
client: Client,
jobs: Receiver<Job>,
deliver: impl Fn(Ready) + Send + 'static,
) -> std::thread::JoinHandle<()> {
std::thread::Builder::new()
.name("covers".into())
.spawn(move || {
let worker = Worker::new(client);
// A cover asked for while it is already in flight - a card scrolled past
// twice, a grid rebuilt on a keystroke - must not become a second fetch.
let mut done: HashSet<Key> = HashSet::new();
for job in jobs {
match job {
Job::Clear => {
worker.clear_disk();
done.clear();
}
Job::Fetch(request) => {
let key = Key {
album_id: request.album_id.clone(),
tier: request.tier,
};
if !done.insert(key.clone()) {
continue;
}
let image = worker.render(&request);
deliver(Ready {
key,
width: image.width(),
height: image.height(),
rgba: image.into_raw(),
});
}
}
}
})
.expect("spawning the cover worker")
}
pub enum Job {
Fetch(Request),
Clear,
}
/// The UI thread's half: what is already decoded, and what has been asked for.
pub struct Store {
jobs: Sender<Job>,
ready: HashMap<Key, slint::Image>,
requested: HashSet<Key>,
}
impl Store {
pub fn new(jobs: Sender<Job>) -> Self {
Self {
jobs,
ready: HashMap::new(),
requested: HashSet::new(),
}
}
pub fn insert(&mut self, ready: Ready) {
let mut buffer =
slint::SharedPixelBuffer::<slint::Rgba8Pixel>::new(ready.width, ready.height);
buffer.make_mut_bytes().copy_from_slice(&ready.rgba);
self.ready
.insert(ready.key, slint::Image::from_rgba8(buffer));
}
/// The cover for an album if it is decoded, and a fetch queued if it is not. Callers
/// paint the album's own colours in the meantime, so this returning `None` is a
/// normal frame rather than a missing one.
pub fn get(&mut self, request: Request) -> Option<slint::Image> {
let key = Key {
album_id: request.album_id.clone(),
tier: request.tier,
};
if let Some(image) = self.ready.get(&key) {
return Some(image.clone());
}
if self.requested.insert(key) {
let _ = self.jobs.send(Job::Fetch(request));
}
None
}
pub fn clear(&mut self) {
self.ready.clear();
self.requested.clear();
let _ = self.jobs.send(Job::Clear);
}
}
#[cfg(test)]
mod tests {
use super::*;
fn request(is_book: bool, tier: Tier) -> Request {
Request {
album_id: "abc".into(),
tier,
has_cover: false,
is_book,
locked: false,
colors: ["#4a6fa5".into(), "#6a8fc5".into(), "#a5804a".into()],
}
}
#[test]
fn shape_follows_the_media_type() {
assert_eq!(dimensions(192, false), (192, 192));
assert_eq!(dimensions(192, true), (157, 192));
assert_eq!(dimensions(384, true), (315, 384));
}
#[test]
fn generated_art_fills_the_tier_it_was_asked_for() {
let music = generated(&request(false, Tier::Thumb));
assert_eq!((music.width(), music.height()), (192, 192));
let book = generated(&request(true, Tier::Hero));
assert_eq!((book.width(), book.height()), (315, 384));
}
#[test]
fn stripes_actually_alternate() {
let art = generated(&request(false, Tier::Thumb));
let colours: HashSet<[u8; 3]> = (0..192)
.map(|x| {
let p = art.get_pixel(x, 0).0;
[p[0], p[1], p[2]]
})
.collect();
assert_eq!(colours.len(), 2, "expected exactly the two stripe colours");
}
#[test]
fn a_book_has_bright_page_edges_on_its_right() {
let art = generated(&request(true, Tier::Hero));
let (w, h) = (art.width(), art.height());
let edge = art.get_pixel(w - 2, h / 2).0;
let middle = art.get_pixel(w / 2, h / 2).0;
let brightness = |p: [u8; 4]| p[0] as u32 + p[1] as u32 + p[2] as u32;
assert!(
brightness(edge) > brightness(middle),
"page edges should be lighter than the cover"
);
}
#[test]
fn every_generated_pixel_is_opaque() {
let art = generated(&request(true, Tier::Thumb));
assert!(art.pixels().all(|p| p.0[3] == 255));
}
}