log-levels-audit #4

Closed

aureus wants to merge 24 commits from log-levels-audit into main

pull from: log-levels-audit

merge into: aureus:main

aureus:main

aureus:feat/per-card-audience-interest

aureus:fix/remove-redundant-feed-refresh

aureus:feat/browser-ci

Conversation 0 Commits 24 Files changed 30 +271 -54

aureus commented 2026-05-25 14:32:35 +00:00

Owner

Copy link

No description provided.

aureus added 24 commits 2026-05-25 14:32:35 +00:00

fix(logging): demote per-iteration cron logs from info to debug ... 21a14aa936

Top-level cron start/done markers stay at info. Per-user/per-audience
inner events (syncHistory start/done, refreshForUser start/skip/done)
and internal pruneCache row-count stats drop to debug so production
logs at the default level are not flooded during batch operations.

fix(ci): add python3/make/g++ to CI image for native addon compilation ... ffcb0cc2c8

better-sqlite3 uses node-gyp as a fallback when prebuild-install
times out. node-gyp requires Python and build tools which are absent
from node:24-bookworm-slim.

fix(ui): add count badge to curate_body Movies panel ... ccd20eebf8

movieCount was computed and passed to the template but never rendered.
Add a rounded-full badge showing the queue depth at the top of the
Movies panel when movieCount > 0, matching the existing test assertions
in feed-tabs.test.ts.

fix(ci): use Actions services for PG/MariaDB, fix container networking ... 1d1df9a37a

When the job runs inside a container (container: image:), docker compose
spawned databases are on the host — localhost inside the container doesn't
reach them. Switch to Actions services: which run as sidecars on the same
network as the job container, reachable by service name.

Add TEST_MYSQL_HOST/TEST_MYSQL_PORT env var overrides to helpers.ts
(postgres already had TEST_DB_URL). Set both in the test steps so the
correct service hostnames are used in CI without changing dev defaults.

fix(tests): make PG and MariaDB integration tests reliable ... 95ade8393b

Five root causes, all fixed:

1. Parallel file execution on shared databases — vitest.config.ts now
   sets fileParallelism=false when TEST_DB_CLIENT is pg or mysql2,
   serialising test files to prevent concurrent migration races and
   duplicate-key seed collisions.

2. State pollution between files — setupTestDb now truncates all data
   tables before re-seeding for both PG (TRUNCATE … RESTART IDENTITY
   CASCADE) and MySQL (TRUNCATE with FK checks disabled).  For PG,
   setval() advances serial sequences past the explicitly-seeded ids so
   the next auto-generated insert doesn't collide at 1.  For MySQL, the
   first call per file drops schema + knex metadata tables then
   re-applies migrations, guarding against corrupt state left by
   previous parallel runs.

3. const [id] = db.insert() broken on PG (Knex 3.x returns the raw pg
   result object, which is non-iterable) — fixed in auth.ts
   (upsertUser, createUser, initUserAudiences), admin.ts, and
   settings.ts by inserting then immediately re-selecting by unique
   field to obtain the auto-generated id.  Same pattern fixed in
   settings.test.ts test setup.

4. Migration FK type mismatch on MariaDB 11 — t.increments() produces
   INT UNSIGNED but FK reference columns used t.integer() (signed INT).
   MariaDB 11 rejects signed→unsigned FK constraints with errno 150.
   Added .unsigned() to all 8 FK columns in 001_initial.ts and the
   restored column in 002_simplify_ratings.ts down().  PG and SQLite
   silently ignore .unsigned().

5. ISO 8601 timestamps rejected by MySQL — test inserts for
   media_metadata.fetched_at and library.synced_at passed
   new Date().toISOString() which MySQL TIMESTAMP columns refuse.
   Dropped the explicit field from those inserts so the column default
   applies.

chore: pin jellyfin in docker-compose.dev.yml c25e2ad2e8

try security_opt 74a85161da

fix(ci): add apparmor:unconfined to jellyfin in dev compose ... 42d5a5429e

seccomp:unconfined alone didn't prevent the exit 139 (SIGSEGV) in CI.
AppArmor runs independently of seccomp on Ubuntu hosts; the docker-default
profile can still block syscalls even when seccomp is disabled. Adding
apparmor:unconfined removes that restriction for the Jellyfin container.

fix(rate): explicit merge columns to fix MariaDB upsert ... 96e8fb348b

Knex's .merge() with no args includes key columns (user_id, jellyfin_id)
in the ON DUPLICATE KEY UPDATE clause. On MariaDB 11 this causes a 1062
duplicate-key error when a conflict is triggered, because MariaDB checks
uniqueness even when the key columns are being set to their existing values.
The route's catch block then returns 500, leaving the old rating in place.

Fix: pass explicit non-key columns to .merge() so the UPDATE clause only
touches the data columns (rating, tmdb_id, media_type / play_count,
is_favorite). Same pattern applied to both ratings upsert sites and
the watch_history upsert.

fix(rate): explicit merge columns to fix MariaDB upsert ... 17a2ad0a24

Knex's .merge() with no args includes key columns (user_id, jellyfin_id)
in the ON DUPLICATE KEY UPDATE clause. On MariaDB 11 this causes a 1062
duplicate-key error when a conflict is triggered, because MariaDB checks
uniqueness even when the key columns are being set to their existing values.
The route's catch block then returns 500, leaving the old rating in place.

Fix: pass explicit non-key columns to .merge() so the UPDATE clause only
touches the data columns (rating, tmdb_id, media_type / play_count,
is_favorite). Same pattern applied to both ratings upsert sites and
the watch_history upsert.

chore: pin container images to LTS/specific versions ... dd386c6282

mariadb:11 is a rolling tag that resolves to different minor versions
across CI runners and developer machines. mariadb:11.4 is the current
LTS branch; pinning it ensures CI and dev always run the same engine.

Also pin seerr to v3.2.0 (was :latest) in dev compose.

fix(ci): add CI-only compose overlay for Jellyfin .NET JIT flags ... e89565325e

Jellyfin 10.x / .NET 8 JIT emits AVX/AVX2/SSSE3 instructions on startup.
The CI runners live inside virtual machines where those instruction
sets are not reliably exposed, causing an immediate SIGSEGV (exit 139).

Rather than pollute the dev compose with CI-specific workarounds, add a
separate dev/docker-compose.ci.yml override that is merged in only by the
pr-checks workflow via COMPOSE_FILE. Local dev is unaffected.

fix(tests): eliminate MariaDB 11.4 race in rate integration tests ... 9f67833e9f

The two 're-rating' tests were failing on MariaDB 11.4 with either a
wrong rating value (0 instead of 2) or zero rows returned.  Root cause:
triggerRefreshAllAudiences fires fire-and-forget after res.send(), and
on MariaDB 11.4 its concurrent SELECT on the ratings table conflicts with
the second request's INSERT ON DUPLICATE KEY UPDATE or check-then-update.

Two complementary fixes:

1. pool: { min: 0, max: 1 } in the mysql2 test Knex config.
   Serialises all DB access so the background refresh's queries cannot
   run concurrently with the second request's writes.  The same-SQL
   template with different bindings (rating=0 vs rating=2) executes on
   a single connection, eliminating any per-connection prepared-statement
   interleaving that triggered MariaDB 11.4's misbehaviour.

2. drainPool() helper + usage in afterEach and between requests.
   Waits for the pool to go idle (numUsed === 0) before the pool is
   destroyed.  This ensures background reads have released their
   connections (and their MariaDB metadata locks) before the next
   test's TRUNCATE TABLE statements run.

try a jellyfin fix yet again b401596d9b

fix(tests): drain pool before destroy in all integration afterEach hooks ... 8dc7debb2f

settings.test.ts was missing drainPool() before db.destroy(), leaving a
live MariaDB metadata lock when POST /audiences/:id/members fired its
background refreshForUser() call. The next test's beforeEach TRUNCATE
blocked on that lock; under CI load the 10 s hookTimeout was exceeded.

Added drainPool() to all five integration test files that lacked it
(settings, admin, auth-routes, discover, rec-index). rate.test.ts already
had the fix; this brings all files into alignment.

Also added a convention comment in helpers.ts at makeTestDb() documenting
the invariant, raised hookTimeout to 30 s for mysql2 in vitest.config.ts
as a safety net, and added DOTNET_USE_POLLING_FILE_WATCHER=1 to the CI
overlay to address the Jellyfin inotify-limit crash (exit 139) — the
actual error exposed by the new diagnostic step; the existing JIT flags
remain as defense-in-depth.

fix(bootstrap): switch wizard readiness check to /health, increase all timeouts ... 7c21a28001

/Startup/Configuration returns non-200 on Jellyfin 10.10.7 fresh installs in
the CI environment, causing a 2-minute timeout even though
Kestrel is already accepting connections.  /health is reliable from the moment
Kestrel binds and is the correct readiness signal.

All bootstrap timeouts doubled/more to accommodate the slow CI hardware:
- waitFor default: 120s → 300s
- library scan:    300s → 600s
- TMDB metadata:  180s → 600s
- CI snapshot step: 25min → 40min

fix(bootstrap): accept 404 from /Startup/Configuration as a ready state ... 5dd62ba2c3

In Jellyfin 10.10.x, a fresh install may pre-initialise the wizard state and
return 404 from /Startup/Configuration instead of 200.  The previous code only
accepted r.ok (200-299), causing a 2-minute timeout even though Jellyfin was
fully up and ready.

404 is now treated as "ready": the wizard state is determinable, and the
StartupWizardCompleted check that follows handles both cases correctly.

fix(ci): connect CI container to dev network so bootstrap can reach services ... 5bad42ff52

The CI job runs inside the CI container (runner network).  dev:up creates Jellyfin/Seerr on the dev_default network — a
completely separate Docker network.  Every fetch('http://localhost:18096/...')
inside bootstrap.ts got ECONNREFUSED, silently swallowed by .catch(()=>false),
causing the waitFor loop to spin for its entire timeout on every run.

Fix: split dev:reset into dev:reset:services (Docker up only) + bootstrap so
the CI workflow can do `docker network connect dev_default $(cat /etc/hostname)`
in between.  With the CI container on dev_default, bootstrap reaches services
by container name.  JELLYFIN_BOOTSTRAP_HOST and SEERR_BOOTSTRAP_HOST env vars
override the localhost defaults in bootstrap.ts for this path.

fix(ci): reach dev services via Docker host gateway instead of network connect ... a011850026

The previous approach of joining the CI container to dev_default via
`docker network connect $(cat /etc/hostname)` fails because the CI
container's hostname is not its Docker container ID as seen by the daemon.

Instead, compute the Docker host gateway IP at runtime with ip-route and
reach the already host-mapped ports (18096, 15055) directly. This avoids
needing to know the CI container's identity at all and removes the fragile
network-connect step.

fix(ci): read gateway from /proc/net/route — ip command not available in CI image ... a2ffbe85cb

iproute2 is not installed in the CI image. Parse /proc/net/route
directly with awk instead: the default-route row (Destination=00000000) holds
the gateway as a little-endian hex field, no external tools needed.

fix(ci): drop strtonum — use bash hex arithmetic to decode gateway ... 721b9f5810

strtonum is gawk-only; the CI image has mawk/busybox awk. Let awk extract
the raw hex field and convert each byte with bash's $((16#XX)) arithmetic
instead, which works in any POSIX shell without gawk extensions.

fix(ci): connect dev containers to runner network; expand failure diagnostics ... eb6f17dcd7

Network fix: instead of connecting the CI container to dev_default (which
requires knowing its Docker ID — unreliable in CI), connect the dev
containers (watchcraft-jellyfin, watchcraft-seerr) to the CI runner network
after they start. The CI container already lives on that network, so Docker DNS
resolves container names immediately after the connect. No /proc parsing or
host-gateway hacks needed.

Diagnostics: expand the on-failure dump to capture /etc/hosts, the raw routing
table, both Docker networks (CI runner network and dev_default) with full
membership lists, live curl probes to watchcraft-jellyfin, Seerr logs, and the
most recent Jellyfin log file from dev/runtime/jellyfin/log/.

fix(ci): reach dev containers via host-mapped ports instead of network join ... 8ade746af9

The CI runner network lives on the outer runner daemon; the HOST
Docker daemon (used by docker compose) has no knowledge of it. docker network
connect to the CI runner network always fails with "network not found".

Instead, compute the HOST IP from /proc/net/route (default gateway of the CI
container's eth0), then reach Jellyfin/Seerr via their host-mapped ports
(18096, 15055). Docker's iptables DNAT rules on the HOST forward the traffic
to the correct containers on dev_default.

Also update the diagnostics step to test connectivity via HOST_IP:port rather
than the unresolvable container hostnames.

fix(ci): run bootstrap inside dev_default; fix Seerr perms; update snapshot baselines ... 18633c2e3f

Two CI bootstrap failures:

1. Network: the CI container and HOST Docker daemon are on separate daemon
   contexts. The gateway IP approach (172.25.0.1:18096) gives Connection
   refused — Docker's DNAT rules don't fire for traffic on that bridge path.
   Fix: run the bootstrap process inside a transient container on dev_default
   via docker run --network, where it reaches services by DNS name.

2. Seerr EACCES: Seerr starts as a non-root user and can't mkdir
   /app/config/logs/ when the bind-mounted runtime dir is root-owned.
   Fix: add chmod -R a+rwx to dev:up so dirs are world-writable before
   containers start.

Also update the diagnostics connectivity probe to run from a transient
alpine container on dev_default rather than probing a host-gateway IP.

Browser test fixes:
- snapshots.test.ts: reset audience selector to Solo before taking the
  discover-mode screenshot, preventing session leakage from the
  audience-switching tests in discover.test.ts.
- Updated baselines for tiers where the curate badge (added earlier on
  this branch) or audience state caused snapshot mismatches.

aureus closed this pull request 2026-05-25 14:33:10 +00:00

Some checks failed

Hide all checks

PR Checks / test (pull_request) Failing after 5m22s

Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

aureus/watchcraft!4

No description provided.