Repository Public Research preview
1# Procedure in detail23Eight stages. Do not skip stage 4: an unverified finding in a referee report costs more4credibility than a missed one.56## Stage 1: intake78Record, before reading:910- **Files.** Main file, included files, bibliography, supplementary material, code, data.11 List what was provided and what was not.12- **Version.** A commit hash, a file checksum or the modification date of the main file.13 The review is about this version only.14- **Request.** Referee report, pre-submission check by the author, editor's assessment,15 check of a revision. The deliverable differs (see `templates.md`).16- **Venue and field**, when known: they set the bar for contribution, the expected17 reporting standards and the length.18- **Constraints.** Confidentiality (an unpublished manuscript is not sent to external19 services or quoted outside the review), conflicts, deadline.2021If the manuscript is a PDF or a word-processor file, the scripts cannot read it. Convert it22to text for reading and say in the report that the mechanical audit was not run, or run it23on the source if the author can provide it.2425## Stage 2: deterministic audit2627```bash28node scripts/manuscript.mjs audit paper.tex --root . --text29```30311. Check the first lines of the output: how many files were read, and whether any include32 was not read (`outline-include-not-read`). Fix the root and rerun before going further.332. Go through the errors. They are few and usually real: a missing bibliography entry, an34 undefined reference, a percentage that does not match its counts, a leftover TODO.353. Go through the warnings, opening each line. Dismiss a finding when the context answers it36 (the test is named in the previous sentence; the "inconsistent" sample sizes are two37 subsamples). Keep a one-line reason for each dismissal.384. Skim the notes (info level). They are prompts for the role reviews, rarely findings.3940The confirmed audit findings enter the ledger with the prefix A (A-01, A-02 ...). They are41almost always minor comments. A few are not: a number in the abstract that appears nowhere42in the paper, or a sample size that changes between sections, can be major.4344## Stage 3: role reviews4546Five independent readings, defined in `roles.md`. For each role:47481. Read the whole manuscript once without writing.492. Answer the role's questions, writing a finding whenever an answer is "no" or "cannot50 tell".513. End with "Not assessed": what the role could not evaluate and why.5253Budget: a role review that produces more than about fifteen findings has usually mixed54major and minor. Rank and cut.5556When delegating to subagents, the brief for each role contains: the role description from57`roles.md`, the paths of the manuscript files, the finding format, and the rules of58`SKILL.md`. It does not contain the audit findings (except for the role they belong to) or59your opinion of the paper.6061## Stage 4: adversarial verification6263For each finding, the verifier's job is to refute it.64651. **Locate.** Open the location. Is the quote exact? If the quote cannot be found, the66 finding is refuted (and the role that produced it needs a second look).672. **Widen.** Read the paragraph before and after, the table notes, the footnotes, the68 appendix section that the passage points to.693. **Search.** If the finding says something is missing, search the manuscript for it under70 other names (a robustness table may be called "sensitivity"; a data statement may be in a71 title-page footnote).724. **Recompute.** If the finding is about arithmetic or consistency, redo it from the73 numbers in the manuscript and write the calculation down.745. **Classify.** confirmed / not shown / refuted / unverifiable, with one line of reason.7576Typical refutations: the support is in the next sentence; the "undefined" acronym is77defined in a table note; the two "inconsistent" numbers are before and after a stated78filter; the cross-reference is defined in a chapter that was not part of the audit.7980Count and report the outcome: how many findings were raised, confirmed, refuted. A refuted81share near zero means the verification was not adversarial enough.8283## Stage 5: synthesis8485The editor role takes all confirmed and not-shown findings and:86871. **Merges duplicates.** Two roles reporting the same problem is one comment; keep the88 better-evidenced wording and note that it was found independently.892. **Assigns severity.** Major: the paper's main conclusions depend on it, or it would90 mislead a reader about what was done. Minor: everything else.913. **Orders major comments** by how much of the conclusion depends on them.924. **Groups minor comments** by section or by exhibit, so the authors can work through them.935. **Cuts.** Matters of taste that the authors may reasonably decline are dropped or94 clearly marked as suggestions.9596## Stage 6: scoring9798Use `rubric.md`. Score each dimension from its anchors and write one or two sentences that99tie the score to specific findings. Apply the caps. Report the total with the per-dimension100scores, never alone.101102For a revision, score both versions with the same anchors and report the two totals side by103side, with the findings that moved each dimension.104105## Stage 7: the report106107Write from `templates.md`. Principles:108109- Start with a summary of the paper that the authors would accept as fair. If you cannot110 write it, you have not understood the paper well enough to review it.111- Each major comment states: what the problem is, where, why it matters for the112 conclusions, and what would resolve it (an analysis, a clarification, a narrower claim).113- Say what is good, specifically. It tells the authors what to keep.114- Separate what you ask for from what you suggest.115- State what was not reviewed.116- Keep the recommendation and any confidential remarks for the editor's letter, unless the117 venue's format asks otherwise.118119## Stage 8: revision and response letter1201211. Run `compare` on the two versions and the audit on the new one.1222. Build the response table (`templates.md`): one row per point of the previous report.1233. For each row, find the change in the new version. Classify: resolved, partly resolved,124 not resolved, resolved differently than asked (and whether that is acceptable), new125 problem introduced.1264. Check the changed numbers that `compare` lists: is each change explained in the response127 letter? An unexplained change in a headline number is a major comment.1285. Do not raise new comments on unchanged material unless they are serious: say explicitly129 that they are new and why they were missed.130