Add a script to seed a device's library and analysis cache
Copying the library to a Pi is one rsync; copying the cache with it is not, because its three parts travel differently. covers/ is keyed by the album folder's path relative to the library root, so it survives the move. analysis/ is keyed by name:size:mtime, so it survives only if mtimes do - hence -a everywhere, and a comment saying why. index.json holds absolute source paths and must not be copied at all: scan_library() reuses a cached album on a matching (name, size, mtime) fingerprint without re-checking the path, so copying it leaves every album pointing at the dev machine and playback fails on files that are present. It is the cheap part, and the device rebuilds it on first scan while reusing everything expensive. Syncs shelf by shelf rather than the whole directory, which is what makes --delete safe: config.yml, tippen-curriculum.yml, tippen-progress.json and the cache all live in the same directory on the device and have no counterpart here. Preflight checks ssh, sizes the transfer and refuses to fill the SD card. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
241
scripts/sync-library.sh
Executable file
241
scripts/sync-library.sh
Executable file
@@ -0,0 +1,241 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Copy a local music library, and the expensive parts of its scan cache, onto a
|
||||
# MusicMouse device.
|
||||
#
|
||||
# scripts/sync-library.sh --host musicdolphin
|
||||
#
|
||||
# The cache is the reason this is a script rather than one rsync line. Its three kinds
|
||||
# of content behave completely differently when they move between machines (see
|
||||
# python-backend/musicmouse/library/cache.py):
|
||||
#
|
||||
# covers/<album_id>.jpg album_id = sha1(folder path relative to the library
|
||||
# root). Portable, as long as the shelf layout under the
|
||||
# target root matches the source - which it does, because
|
||||
# this script syncs the shelves verbatim.
|
||||
#
|
||||
# analysis/<track_key>.json track_key = sha1("<name>:<size>:<int mtime>"). Portable
|
||||
# *only if mtimes survive the copy*. That is why every
|
||||
# rsync here is -a and why you must never reach for scp
|
||||
# without -p: losing mtimes silently invalidates every
|
||||
# analysis file, each of which costs minutes of DSP to
|
||||
# rebuild.
|
||||
#
|
||||
# index.json NOT copied. It holds absolute source-machine paths, and
|
||||
# scan_library() reuses a cached album whenever its
|
||||
# (name, size, mtime) fingerprint matches, without
|
||||
# re-checking the path. Copying it makes every album come
|
||||
# back pointing at /home/<you>/Music/..., and playback
|
||||
# fails on files that are sitting right there. It is the
|
||||
# cheap part of the cache - the device rebuilds it on its
|
||||
# first scan, reusing every cover and analysis file we did
|
||||
# copy.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
HOST="musicdolphin"
|
||||
SSH_USER="root"
|
||||
SRC="${HOME}/Music"
|
||||
CACHE=""
|
||||
DEST="/media/musicmouse"
|
||||
DRY_RUN=0
|
||||
COPY_CACHE=1
|
||||
|
||||
# Shelves the scanner knows about (musicmouse/library/sections.py). Syncing these by
|
||||
# name rather than the whole directory is what makes --delete safe: config.yml,
|
||||
# tippen-curriculum.yml, tippen-progress.json and .musicmouse-cache all live in $DEST
|
||||
# on the device and must survive.
|
||||
SECTIONS=("Figuren" "Musik" "Hörbücher" "Kinderpodcasts")
|
||||
|
||||
# Leave this much room on the target after the copy. These devices are SD cards with
|
||||
# little to spare, and a full root filesystem breaks far more than the music player.
|
||||
HEADROOM_BYTES=$((1024 * 1024 * 1024))
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage: scripts/sync-library.sh [options]
|
||||
|
||||
--host HOST Target device (default: musicdolphin)
|
||||
--user USER SSH user (default: root)
|
||||
--src DIR Local library (default: ~/Music)
|
||||
--dest DIR Library root on target (default: /media/musicmouse)
|
||||
--cache DIR Local cache directory (default: <src>/.musicmouse-cache, falling
|
||||
back to python-backend/.musicmouse-cache next to this script)
|
||||
--no-cache Skip the cache; the device recomputes everything from scratch
|
||||
--dry-run Show what would be transferred, change nothing
|
||||
-h, --help This message
|
||||
USAGE
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--host) HOST="$2"; shift 2 ;;
|
||||
--user) SSH_USER="$2"; shift 2 ;;
|
||||
--src) SRC="$2"; shift 2 ;;
|
||||
--dest) DEST="$2"; shift 2 ;;
|
||||
--cache) CACHE="$2"; shift 2 ;;
|
||||
--no-cache) COPY_CACHE=0; shift ;;
|
||||
--dry-run) DRY_RUN=1; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "unknown option: $1" >&2; usage >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
SSH_TARGET="${SSH_USER}@${HOST}"
|
||||
SRC="${SRC%/}"
|
||||
DEST="${DEST%/}"
|
||||
|
||||
die() { echo "error: $*" >&2; exit 1; }
|
||||
step() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
|
||||
human() {
|
||||
# Bytes -> something a person can judge at a glance.
|
||||
numfmt --to=iec --suffix=B "$1" 2>/dev/null || echo "$1 bytes"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------- cache location
|
||||
|
||||
if [[ -z "$CACHE" ]]; then
|
||||
if [[ -d "${SRC}/.musicmouse-cache" ]]; then
|
||||
CACHE="${SRC}/.musicmouse-cache"
|
||||
else
|
||||
# The dev setup keeps config.yml (and therefore the cache, which resolves
|
||||
# relative to it) in python-backend/ rather than beside the music.
|
||||
CACHE="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/python-backend/.musicmouse-cache"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ------------------------------------------------------------------------- preflight
|
||||
|
||||
step "Preflight"
|
||||
|
||||
[[ -d "$SRC" ]] || die "source library not found: $SRC"
|
||||
|
||||
present_sections=()
|
||||
for section in "${SECTIONS[@]}"; do
|
||||
[[ -d "${SRC}/${section}" ]] && present_sections+=("$section")
|
||||
done
|
||||
[[ ${#present_sections[@]} -gt 0 ]] \
|
||||
|| die "no known shelves under $SRC (looked for: ${SECTIONS[*]})"
|
||||
|
||||
echo "source: $SRC"
|
||||
echo "shelves: ${present_sections[*]}"
|
||||
|
||||
if [[ $COPY_CACHE -eq 1 ]]; then
|
||||
if [[ -d "$CACHE" ]]; then
|
||||
echo "cache: $CACHE"
|
||||
else
|
||||
echo "cache: none found at $CACHE - continuing without it"
|
||||
COPY_CACHE=0
|
||||
fi
|
||||
else
|
||||
echo "cache: skipped (--no-cache)"
|
||||
fi
|
||||
|
||||
ssh -o ConnectTimeout=10 -o BatchMode=yes "$SSH_TARGET" true 2>/dev/null \
|
||||
|| die "cannot ssh to $SSH_TARGET (needs key auth, no password prompt)"
|
||||
echo "target: ${SSH_TARGET}:${DEST}"
|
||||
|
||||
# Size the transfer. This is an upper bound: rsync skips files already present and
|
||||
# identical, so a repeat run moves far less than this.
|
||||
src_bytes=0
|
||||
for section in "${present_sections[@]}"; do
|
||||
section_bytes=$(du -sb "${SRC}/${section}" | cut -f1)
|
||||
src_bytes=$((src_bytes + section_bytes))
|
||||
printf ' %-18s %s\n' "$section" "$(human "$section_bytes")"
|
||||
done
|
||||
if [[ $COPY_CACHE -eq 1 ]]; then
|
||||
for part in covers analysis; do
|
||||
if [[ -d "${CACHE}/${part}" ]]; then
|
||||
part_bytes=$(du -sb "${CACHE}/${part}" | cut -f1)
|
||||
src_bytes=$((src_bytes + part_bytes))
|
||||
printf ' %-18s %s\n' "cache/${part}" "$(human "$part_bytes")"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# df on the parent: $DEST may not exist yet on a first run.
|
||||
avail_kb=$(ssh "$SSH_TARGET" "df -Pk '$(dirname "$DEST")' | awk 'NR==2 {print \$4}'")
|
||||
[[ -n "$avail_kb" ]] || die "could not read free space on $HOST"
|
||||
avail_bytes=$((avail_kb * 1024))
|
||||
|
||||
echo
|
||||
echo "to transfer (upper bound): $(human "$src_bytes")"
|
||||
echo "free on target: $(human "$avail_bytes")"
|
||||
|
||||
if (( src_bytes + HEADROOM_BYTES > avail_bytes )); then
|
||||
# Already-synced content counts against src_bytes but costs nothing, so this is a
|
||||
# warning on a repeat run and a hard stop only when nothing is there yet.
|
||||
remote_used=$(ssh "$SSH_TARGET" "du -sb '$DEST' 2>/dev/null | cut -f1" || echo 0)
|
||||
remote_used=${remote_used:-0}
|
||||
if (( remote_used > 0 )); then
|
||||
echo
|
||||
echo "warning: the full library would not fit in the free space, but $(human "$remote_used")"
|
||||
echo " is already at ${DEST}. rsync only moves the difference; watch the"
|
||||
echo " free space as it runs."
|
||||
else
|
||||
die "not enough space: need $(human $((src_bytes + HEADROOM_BYTES))) including $(human "$HEADROOM_BYTES") headroom, have $(human "$avail_bytes")"
|
||||
fi
|
||||
fi
|
||||
|
||||
# --------------------------------------------------------------------------- transfer
|
||||
|
||||
RSYNC_OPTS=(-aH --info=progress2 --human-readable)
|
||||
[[ $DRY_RUN -eq 1 ]] && RSYNC_OPTS+=(-n)
|
||||
|
||||
if [[ $DRY_RUN -eq 1 ]]; then
|
||||
echo
|
||||
echo "(dry run - nothing will be written)"
|
||||
else
|
||||
step "Creating ${DEST} on ${HOST}"
|
||||
ssh "$SSH_TARGET" "mkdir -p '$DEST'"
|
||||
fi
|
||||
|
||||
step "Library"
|
||||
for section in "${present_sections[@]}"; do
|
||||
echo "--- ${section}"
|
||||
# Per-shelf, with --delete scoped to that shelf. A --delete on $DEST as a whole
|
||||
# would take out config.yml and the cache, which live in the same directory on the
|
||||
# device but have no counterpart here.
|
||||
rsync "${RSYNC_OPTS[@]}" --delete \
|
||||
"${SRC}/${section}/" "${SSH_TARGET}:${DEST}/${section}/"
|
||||
done
|
||||
|
||||
if [[ $COPY_CACHE -eq 1 ]]; then
|
||||
step "Cache (covers and analysis; index.json deliberately not copied)"
|
||||
for part in covers analysis; do
|
||||
[[ -d "${CACHE}/${part}" ]] || continue
|
||||
echo "--- ${part}"
|
||||
# No --delete: the device may have analysed tracks this machine never saw.
|
||||
rsync "${RSYNC_OPTS[@]}" \
|
||||
"${CACHE}/${part}/" "${SSH_TARGET}:${DEST}/.musicmouse-cache/${part}/"
|
||||
done
|
||||
fi
|
||||
|
||||
# ----------------------------------------------------------------------------- report
|
||||
|
||||
step "Done"
|
||||
|
||||
if [[ $DRY_RUN -eq 1 ]]; then
|
||||
echo "Dry run only - nothing was written to ${HOST}."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
ssh "$SSH_TARGET" "
|
||||
echo 'library: '\$(du -sh '$DEST' 2>/dev/null | cut -f1)
|
||||
echo 'free: '\$(df -Ph '$DEST' | awk 'NR==2 {print \$4}')
|
||||
if [ -d '${DEST}/.musicmouse-cache' ]; then
|
||||
echo 'covers: '\$(find '${DEST}/.musicmouse-cache/covers' -type f 2>/dev/null | wc -l)' files'
|
||||
echo 'analysis: '\$(find '${DEST}/.musicmouse-cache/analysis' -type f 2>/dev/null | wc -l)' files'
|
||||
fi
|
||||
"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
The backend rescans on its next start - a few minutes for a large library - and writes
|
||||
a fresh index.json with this device's own paths. Every cover and analysis file copied
|
||||
above is reused, so the expensive work does not happen again.
|
||||
|
||||
ssh ${SSH_TARGET} systemctl restart musicmouse
|
||||
ssh ${SSH_TARGET} journalctl -u musicmouse -f
|
||||
EOF
|
||||
Reference in New Issue
Block a user