mirror of
https://github.com/TheoLeeCJ/SemIf-OpenJev.git
synced 2026-09-28 07:02:55 +08:00
DarenandClaude Code
f37a7b68e0
Run direct, serial, and shared scoring on Apple Silicon through two backends that preserve the frozen prompt contracts and uncalibrated probability warnings: - PyTorch/MPS: device resolution (--device auto|cuda|mps) plus device-synchronized timings shared by all torch scorers. The shared scorer loops batch-1 suffixes on MPS, where batched multi-row suffix forwards are pathologically slow, and CUDA keeps its batched path. - Optional MLX backend (--backend mlx): same pinned Qwen3.5-4B BF16 checkpoint, no conversion. The loader validates checkpoint configs before any download, refuses custom model files (mlx-lm executes config-declared model_file as Python), quantized weights, and non-BF16 weights; FP32 is allowed only for linear-attention state parameters, verified by parameter name. - CLI: reranker stays CUDA-only (rejected on MPS before loading); benchmark runners pin CUDA explicitly. - MLX serial rows carry the same audit fields as torch serial rows (answer_token_ids, allowed_token_mass, full_vocab_argmax_id). - Tests: CLI routing with mocked backends, device resolution, MLX loader guards, hybrid-cache branch isolation, MPS shared timing. All run without MLX installed (MLX tests skip) or a GPU. - Docs: docs/APPLE_SILICON.md with install, run, guarantees, and measured M5 numbers (illustrative, not committed benchmarks). Published CUDA results, claims, and checksums are untouched: 50 tests pass, 19/19 SHA256SUMS OK, 69/69 published claims verified. MLX quality comparison vs published CUDA: identical balanced accuracy on authored144 (0.806) and every204 (0.942); wanli256 0.617 vs 0.637 and typesafe102 modal agreement 0.820 vs 0.853 differ within the same backend-independent range as MPS (see APPLE_SILICON.md). Co-Authored-By: Claude Code <noreply@anthropic.com>
2.4 KiB
2.4 KiB
Apple Silicon
SemIf runs on Apple Silicon through two backends. The PyTorch backend executes on the
MPS GPU; the native MLX backend runs the same pinned Qwen3.5-4B checkpoint on Metal.
See MLX.md for MLX install, quantization options, and benchmark evidence.
Published CUDA numbers in results/ are unaffected: both Apple backends are additive.
Install
python -m venv .venv
. .venv/bin/activate
pip install -e '.[test]'
# MLX backend (Apple Silicon only):
pip install -e '.[test,mlx]'
Run
# PyTorch/MPS:
semif-score --mode direct --device mps \
--model Qwen/Qwen3.5-4B --revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a \
--input examples/decisions.jsonl --output results-mps-direct.jsonl
# MLX (direct/serial/shared; reranker is CUDA-only):
semif-score --backend mlx --mode direct \
--model Qwen/Qwen3.5-4B --revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a \
--input examples/decisions.jsonl --output results-mlx-direct.jsonl
Behavior and guarantees
- Both backends preserve the prompt contracts: identical chat template, token boundaries, and prompt hashes; scores carry the same uncalibrated-probability warnings as CUDA output.
- Shared mode on both Apple backends prefills once, then forwards independent batch-1 suffixes rather than a parallel batch. CUDA keeps its batched path.
- Timing fields are device-synchronized on both backends. Shared timing separates prompt encoding, prefix prefill, cache replication/copying, and suffix forwards.
- The reranker and published CUDA benchmark runners remain CUDA-only. Use the
semif-scorecommands above for Apple Silicon.
Measured on an M5 (24 GB), pinned Qwen3.5-4B, BF16
Measurements are illustrative, not committed benchmarks; they were taken 2026-09-17 and are hardware- and version-sensitive.
| Path | PyTorch MPS | MLX |
|---|---|---|
| Warm direct decision (~140-token prompt) | ~0.6 s | ~0.26 s |
| First forward (shader compilation) | ~8 s | ~7 s |
| One group from the 37×21 fixture (one prefill + 21 suffixes) | 16.4 s | 9.1 s |
| Peak allocation during direct scoring | — | ~8.6 GB |
Known limitations: PyTorch MPS falls back to reference kernels for Qwen3.5's
hybrid attention (causal_conv1d, flash-linear-attention are CUDA-only), so MPS
should not be compared directly against the committed RTX 3090 numbers. MLX
quantization and accuracy trade-offs are documented in MLX.md.