nash/corpus/DIVERGENCES.md

# nash ↔ bash divergence log

Live record of behavioral divergences found by the replay harness
(`replay.py`, corpus in `corpus.jsonl`). Per docs/NASH.md §10.3/§11, each entry
stays open until fixed in the fork (or upstream) and re-verified by the corpus.

## Open

(none)

## Closed (recent)

### D2 — non-ASCII bytes re-encoded through `read`/`echo` under C/empty locale (fixed in fork)

- **Found**: narOS N0 rootfs validation (first real-world install run under the
  divert): with `/bin/sh → nash`, `ca-certificates`' postinst
  (`update-ca-certificates`, a `while read`-over-conf loop) mangled the UTF-8
  filename `NetLock_Arany_=Class_Gold=_Főtanúsítvány.crt` and failed the
  whole image build. Worked around structurally in `os/mkimage/build-rootfs.sh`
  (the divert is now always the image's last configure step), but any
  *runtime* `apt install` inside narOS still runs postinsts under nash, so this
  blocks narOS/M3 forcing gates until fixed.
- **Repro** (nash `0.4.0` musl arm64, empty locale):
  `echo 'Főtanúsítvány' | nash -c 'while read x; do echo "$x"; done'` —
  bash emits the input bytes unchanged (`F \305\221 t a n \303\272 …`); nash
  emits each byte Latin-1→UTF-8 double-encoded (`F \303\205 \302\221 …`).
- **Suspected root cause**: brush decodes input bytes to `String` with a lossy/
  Latin-1 assumption on the `read` path (or at word-splitting) instead of
  keeping raw bytes; on output the char sequence is re-encoded as UTF-8.
  POSIX shells treat variable values as byte strings.
- **Severity**: silent data corruption (not a parse failure, so the §4.1
  bash-fallback cannot catch it).
- **Root cause (confirmed)**: `brush-builtins/src/read.rs` `InputReader::read_event`
  read one byte at a time and did `self.buffer[0] as char` — a Latin-1 decode —
  before pushing into the result `String` (upstream had a `TODO(utf-8)` on the
  field). Valid UTF-8 input was thus re-encoded byte-by-byte.
- **Fix**: incremental UTF-8 decoding in `read_event` (`// nash(D2)` markers):
  lead byte classifies the sequence length (0xC2–0xF4), continuation bytes are
  read (blocking, matching `read`'s wait-for-delimiter behavior; a
  non-continuation byte is pushed back), and valid sequences round-trip
  byte-identically. bash's `-n`-counts-bytes semantics are preserved for free
  because the char-limit check measures `line.len()` (UTF-8 byte length).
  Candidate for upstreaming (fixes upstream's own TODO).
- **Residual (accepted)**: *invalid* UTF-8 input bytes still degrade to the
  per-byte Latin-1 mapping (a Rust `String` cannot hold raw invalid bytes;
  byte-fidelity there would need `Vec<u8>` plumbing shell-wide). bash keeps raw
  bytes. Not corpus-gated; revisit only if it bites in practice.
- **Verified**: repro now byte-identical to bash; split-across-writes
  continuation reassembly correct; `read -n 3` byte counting identical to bash
  (C locale); the original real-world failure (`update-ca-certificates` under
  nash) exits 0 with all 141 certs processed; corpus 97/97 = 100% including
  three new cases (`utf8-read-passthru`, `utf8-while-read-file`,
  `utf8-cmdsub-roundtrip`); brush-builtins read tests 20/20; brush compat
  suite failure set **identical to the pristine-HEAD baseline rebuilt in the
  same container** (29 shared environmental failures, ±1 flaky SIGPIPE-timing
  case that passes 10/10 standalone).
- **Regression tests**: the three `utf8-*` corpus cases above.

## Closed

### D1 — IFS word-splitting applied to literal words (fixed in fork)

- **Found**: M0 corpus run (`ifs-split`), brush-shell-v0.4.0.
- **Repro**: `IFS=,; set -- a,b,c; echo $1` → bash `a b c`, nash `a`.
- **Root cause**: brush applied IFS splitting to *literal* words, not just
  expansion results: under `IFS=,`, `echo a,b,c` printed `a b c` (bash:
  `a,b,c`) and `set -- a,b,c` received 3 args (bash: 1). POSIX/bash split only
  the results of parameter/command/arithmetic expansion. Mechanically,
  `brush-core/src/expansion.rs` had a two-state `ExpansionPiece`
  (`Splittable`/`Unsplittable`) conflating "field-splittable" with
  "glob-active", and literal text was marked `Splittable`.
- **Fix**: added a third piece state `LiteralText` (never field-split, glob
  chars active) and produce it for unquoted literal text, the no-expansion-chars
  fast path, and retained-backslash escapes (`// nash:` markers in
  `expansion.rs`). Corpus back to 100% (94/94); candidate for upstreaming.
- **Regression tests**: corpus `ifs-split` (+ `glob-all`, `glob-txt`,
  `cond-pattern`, `case` guard the glob/pattern side); brush compat suite.
- **Fix fallout (caught by the brush compat suite, both fixed)**: (a) text
  substituted by `${v:-word}`/`${v:+word}`/`${v:=word}` *is* an expansion result
  and must split — restored via a boundary conversion in the
  `ParameterExpansion` arm; (b) `compgen -W` splits its word-list string as
  data — restored via `ExpanderOptions.field_split_literal_text`. Final suite:
  1684 succeeded / 6 failed, failure set identical to the pristine-brush
  baseline in this container (environmental only); 3 upstream known-fail IFS
  tests now pass (markers flipped in `ifs.yaml`).