Secret detection
How chat-recall keeps credentials out of the index, and why it is built this way. The short version:
Rules are data and travel from the server. Execution is local and in-process. Binaries are a developer tool, not a product dependency.
This document exists so the design is not re-litigated. Every claim below is a measurement, and each one names what was measured.
1. Where detection runs, and why it has to
Detection runs on the client, in-process, in
packages/engine/src/core/secret-redactor.ts. It runs for every user on every
sync, with no external dependency and no subprocess.
It has to run there. The server only ever receives text that has already been redacted — that is the privacy guarantee — so the server cannot find what the client's redactor missed. There is no arrangement in which detection is server-side and the guarantee still holds.
Consequences we then have to handle, and do:
| Consequence | Where it is handled |
|---|---|
| Rules must reach every device without a CLI release | server-served rule pack (§3) |
| An old client may ship us a secret it did not recognise | server-side re-scan of stored text (§4) |
| Ordinary users have no scanner binaries installed | in-process engine is the primary detector, not a fallback (§2) |
2. The in-process engine is the better detector, measured
Measured on 40 real sessions (53MB) of the maintainer's own transcripts, comparing our in-process engine against gitleaks and trufflehog run over the same raw text:
distinct secrets found — both engines: 7
builtin only: 26 ← our in-process regex
EXTERNAL ONLY: 4 ← all trufflehog: Alchemy x2, Juro x1, TLy x1
gitleaks unique contribution: 0
Our own engine found 33 of 37 (89%). The external binaries added four, all
from trufflehog's niche vendor detectors — Juro/TLy are almost certainly
false positives, Alchemy plausibly real. gitleaks, the one we are legally
free to borrow from, added nothing unique.
Set that against what the binaries cost:
- Licensing. trufflehog is AGPL-3.0. We may invoke it; we may not port its detectors into the product. It is the only one that contributed anything. gitleaks is MIT — portable, and the basis of §3.
- Pre-redaction plaintext. They scan raw text, so using them requires materialising unredacted session text on disk.
- Device-dependent coverage. A laptop with trufflehog and one without produce different findings for the same session.
- Their process behaviour becomes ours. See §5 — this one cost a 31GB tmpfs.
So the external detectors are opt-in and default off
(CHAT_RECALL_EXTERNAL_SCANNERS=1), and on this evidence that is not a
compromise. They are enrichment for a findings dashboard on a developer or CI
machine, never part of the SaaS path.
3. Coverage ships as rules, not binaries
The mechanism (GET /api/secrets/rules → installServerRulePack → in-process
compile at sync time) means a shape discovered on a Tuesday is redacting on
every customer's machine on that Tuesday — no CLI release, no install, no
permissions.
The content is packages/server/src/services/builtin-rulepack.ts: 74 curated
vendor rules, generated by scripts/gen-builtin-rulepack.mjs from the
gitleaks MIT corpus (upstream rule ids preserved, so pack:twitter-api-key
can be looked up against the upstream definition).
Three selection decisions, each with a reason:
Anchored rules only. gitleaks' 130 entropy-gated generic rules were dropped wholesale. Without its Shannon-entropy thresholds and per-rule allowlists they match ordinary tokens — and a false positive in a redactor is not a noisy report, it silently destroys the user's own content. Every shipped rule needs a literal vendor marker to fire.
Public identifiers excluded. Client ids, publishable keys and browser tokens are meant to be seen; redacting them costs context and prevents nothing. The full exclusion list with reasons is in the generator.
The leading [\w.-]{0,50}? is stripped from every pattern. This is a
correctness-of-shipping issue, not cosmetics. gitleaks runs under RE2 (linear);
JavaScript backtracks, and that lazy prefix forces a match attempt at every
position. Measured on a 2.1MB real transcript with the shipped pack:
builtins only (20 rules) ..................... 61 ms 35 MB/s
+ pack, patterns as written upstream ......... 3732 ms 0.6 MB/s 60x
+ pack, keyword prefilter only ............... 323 ms 6.5 MB/s 5.2x
+ pack, leading prefix stripped .............. 123 ms 17 MB/s 2.0x ← shipped
Redaction runs on every chunk of every sync, so 0.6 MB/s was never shippable. The prefix existed only so gitleaks could report the variable name beside the secret; removing it cannot lose a detection, and the match counts over the corpus were identical before and after.
False-positive sweep. The shipped pack was run over 400 real sessions, 1.68GB, and every match compared against what the builtins already cover:
rules that fired: 3 of 74
slack-webhook-url 8 hits across 3 sessions — the services/ form (present in
168 real sessions) is already covered by the builtin rule;
the 2 uncovered hits are the workflows//triggers/ forms,
which the builtin does NOT cover and this rule adds
age-secret-key 6 hits, all in ONE session
heroku-api-key 3 hits, all in the SAME one session
false positives on real content: 0
sweep throughput: 1.68GB in 54s (31 MB/s) with all 74 rules active
The one session those last two fired in is the transcript of the session that
built this pack — they matched the synthetic fixtures written while testing it,
which is a true positive on planted data and not evidence of anything about the
corpus. Attributed by file rather than assumed: an earlier sweep, run before that
transcript existed, showed slack-webhook-url alone.
Two honest readings of that. Against genuine corpus content the pack adds almost nothing — two extra Slack webhook forms — which is the same result gitleaks itself produced in §2 and is consistent rather than surprising: one person's transcripts exercise one person's vendor stack. What the sweep does establish is the property that makes shipping it to other people's machines defensible: 74 vendor rules ran over 1.68GB of ordinary engineering conversation and altered nothing that was not either already being redacted or a real credential shape. The pack is a bet on customers whose stacks include Twilio, Datadog, Fastly or Netlify — made at a measured cost of 2x redaction time and a measured false-positive rate of zero.
Safety properties of the pack, both enforced in code:
- Add-only. A pack can make a client redact more, never less — it cannot remove or weaken a compiled-in builtin. A compromised or fat-fingered server therefore fails in the safe direction, the same way the sync-exclusions union does.
- Validated on both ends. Each rule is checked at serve time and again on the client; a rule that matches the empty string or benign canary text is dropped individually and logged. A silently ignored rule reads as coverage the operator does not have, which is worse than an error.
How that second property earned its keep. The first generated pack shipped
telegram-bot-api-token, which uses(?-i:A)— an RE2 construct for turning a flag off inside a subexpression that JavaScript rejects outright. The translator guarded against(?i)-style groups but not that one. Serve-time validation caught it and logged the rejection, so nothing broke — but the endpoint quietly served 74 rules while the file claimed 75, which is exactly the "coverage you don't have" failure the validation exists to prevent, one layer up. Two fixes: the generator now refuses RE2 flag-scoping groups, and the test asserts against the raw rule list rather than the post-validation one. Comparing the filtered list to itself is a tautology, and that tautology is how it reached production.
Tenant-authored rules from the dashboard join the same pack (labelled
tenant: rather than pack:), so a customer can add their own internal shapes
without waiting on us.
4. The server re-scans what it stores
Client-side detection leaves exactly one gap: a device running an old CLI, or one whose rules simply missed a shape, ships us text with a live credential still in it. That text then sits in Postgres in cleartext and nothing else in the system would ever notice.
So the server re-runs today's rules over text it already holds
(packages/server/src/services/secret-rescan.ts, daily sweep plus an on-demand
POST /api/secrets/rescan). Anything it finds is by definition a client-side
miss, and is worth three things: a finding attributed to the server detector, an
alert because the secret is in our database, and a signal that the device should
upgrade.
It sees only post-redaction text, so it cannot find secrets in raw transcripts — it catches precisely what leaked through redaction, which is the failure mode that would otherwise be invisible.
5. If a binary runs, we own it
The external detectors are resolved from our own pinned, checksum-verified
install (packages/engine/src/core/detector-install.ts,
chat-recall detectors install). PATH is never consulted.
That is not fastidiousness. It is the fix for a real incident:
Two byte-identical trufflehog 3.96.0 binaries (same sha256) behaved differently based only on where they were installed:
trufflehog is wrapped in jpillora/overseer, whose self-update path replaces the binary in place. When the invoking user cannot write it, overseer copies all 34MB to
$TMPDIR/overseer-<hash>, execs the copy, and never removes it. A sync daemon whose PATH resolved the root-owned install accreted 34MB per sync until a 31GB tmpfs was full and every process needing/tmpbroke — 289 copies, 9.4GB, on one dev machine.
"Tell the customer to chown their binary" is not a product. So the machine no
longer gets to decide:
| Property | What it buys |
|---|---|
| Pinned — one version per detector, in the manifest | the same bytes everywhere |
| Verified — release sha256 checked before anything is made executable, binary sha256 re-checked on use | a mirror serving different bytes gets nothing; a binary swapped after install stops resolving |
User-owned by construction — installed under ~/.chat-recall/bin | removes the precondition for the leak rather than mitigating its symptom |
| Absolute-path invocation — PATH never consulted | a stray install elsewhere can neither be picked up nor shadow ours |
Two further defences stay in place, because defence in depth is the point and
they are nearly free: --no-update on trufflehog invocations (verified to
suppress the copy), and a scoped TMPDIR per spawn so anything a future
detector scribbles lands in a directory we delete.
CHAT_RECALL_DETECTOR_DIR lets a self-host operator point at binaries baked
into their image. That is still absolute-path resolution — a different place to
look, not a return to PATH.
6. What a user actually experiences
| SaaS user, nothing installed | Self-host / CI, detectors installed | |
|---|---|---|
| Redaction before sync | ✅ in-process, always | ✅ same |
| Rule coverage | ✅ builtins + server pack, updated server-side | ✅ same |
| Findings dashboard | ✅ from the in-process engine | ✅ plus gitleaks/trufflehog enrichment |
| Server re-scan + alerts | ✅ | ✅ |
| Subprocesses on their machine | none | pinned, verified, user-owned |
7. Open questions
- Are the 4 external-only hits real? A one-off
--only-verifiedpass would say whether any is a live key — the difference between "port some vendor rules" and "there was nothing to port". It sends candidate values to their own issuers, so it is a deliberate, user-initiated action. - Does the vendor pack earn its keep? It added zero detections on 1.68GB of
this corpus at zero false positives. The rules that would pay off are for
stacks this corpus does not contain. §4's server-side re-scan is the
instrument that will answer this from real tenants: a
pack:rule that only ever fires there is a rule that earned its place, and one that never fires anywhere should eventually be dropped rather than carried forever.
File map
| Path | Role |
|---|---|
packages/engine/src/core/secret-redactor.ts | the in-process engine: builtin rules, contextual detection, pack installation |
packages/server/src/services/builtin-rulepack.ts | the curated pack served to every tenant (generated) |
scripts/gen-builtin-rulepack.mjs | regenerates the pack from the gitleaks MIT corpus |
packages/server/src/routes/secrets.ts | GET /rules (pack + tenant rules), findings API, SECURITY_TASKS.md |
packages/server/src/services/secret-rescan.ts | server-side re-scan of stored text |
packages/engine/src/core/detector-install.ts | pinned, verified, PATH-free install of the optional binaries |
packages/engine/src/core/secret-scanner.ts | the opt-in external-detector wrapper |