Triage, Not Certification: Reliability Limits of Automated Triage Signals for Scholarly Knowledge Graphs

Under review. Audits whether automated LLM-judge scores and passive platform traces are reliable enough to route scarce expert source-review effort across short cited contributions in a scholarly knowledge graph — finding they are not, and offering a reusable label-free reliability check instead.

Authors: Iman YeckehZaare · Venue/status: ACM HCOMP

This submitted manuscript performs a pre-validation reliability audit of automated triage signals on thousands of reference-bearing nodes from a public scholarly knowledge graph, testing whether different LLM scoring conditions select the same nodes for review and whether citation-support scores stay stable under perturbation. It argues these signals can triage but not certify, and contributes a reusable label-free reliability check.

Public artifact boundary: this route exposes status, authorship, visual summary, citation metadata, and the contribution boundary; manuscript files are posted only when review and prepublication rules allow it.

The rendered manuscript page adds status, visual summary, review boundary, citation metadata, and contribution notes. Key links: home, systems, papers, manuscripts, Google Scholar, GitHub, LinkedIn, ORCID, MIT profile, and CV PDF.