Analysis Output
Surfacing model limitations is core to what ModEval does.
ModEval will compare model decisions, policy alignment, and disagreement signals here
Interpretation Layer
METHODOLOGY & FRAMEWORK
8 independent models analyze every submission simultaneously. Outputs are normalized, scored against platform-specific AI policy alignment, and resolved into a single verdict.
The technical flow of information from request to high-fidelity safety score.
SECTION 1
Each model returns data in a different format. toxic-bert returns multi-label probabilities. roberta-offensive returns a binary score. The hate speech model returns hate/not-hate. Raw outputs are incompatible for direct comparison.
The Normalizer converts every model's output into a unified schema:
{
model: string,
top_category: string,
confidence: number (0-1),
action: "Allow" | "Review" | "Remove",
flagged: boolean
}SECTION 2
The models themselves are frozen -- thresholds are fixed, not adjusted per platform. All platforms use identical base thresholds. Platform context instead determines how Claude Haiku interprets each model's output against that platform's specific content policy. The same confidence score can mean different things depending on what the platform prohibits.
Platform Context
| Reddit content policy rules | |
| Discord | Discord community guidelines |
| Facebook community standards | |
| Instagram community standards | |
| Custom | User-defined policy text |
SECTION 3
After all eight models return normalized results, ModEval evaluates how each model aligns with the selected platform's content policy. Instead of simple keyword matching, a batched AI call to Claude Haiku assesses each model's output in context of platform-specific rules.
The platform context determines which categories are zero-tolerance, which are deprioritized, and what the expected enforcement rigor is. Each model's category prediction and confidence score are evaluated against these expectations. The AI returns three values per model:
This approach captures nuance that keyword rules cannot. A high-confidence hate speech detection on Reddit aligns perfectly. The same detection on Discord may not align as strongly, since Discord deprioritizes general toxicity in favour of zero-tolerance for CSAM and threatening behaviour. The AI understands context; the numbers follow.
If the Claude call fails, ModEval falls back to keyword-based alignment rules, ensuring the analysis always completes.
SECTION 4
A disagreement is flagged when two or more models produce different final actions after normalization. Three types of disagreement are detected:
| Action Mismatch | Models recommend different actions (Allow vs Remove) |
| Category Mismatch | Models flag different top violation categories |
Disagreements are not errors. They are the most analytically interesting output ModEval produces.
Content that triggers a disagreement is content that sits in an ambiguous zone, which is exactly the content that requires human review in a real T&S pipeline.
SECTION 5
After all eight models have returned results and disagreements have been detected, ModEval passes the full output to a language model for synthesis. Claude Haiku receives each model's action, confidence score, and top category, along with AI alignment assessments per model and the detected disagreement type, and generates an analytical verdict.
This layer exists because structured data alone does not always communicate risk clearly. A Trust & Safety reviewer scanning hundreds of results needs to understand not just what the models said, but what the disagreement means in context. The AI interpretation translates model outputs into an actionable summary.
If the Claude call fails or times out, ModEval falls back to a rule-based consensus summary generated from the same structured data — ensuring the Explainability section always renders.
| Input | Output |
|---|---|
| 8 model results (action, confidence, top category), AI alignment assessments per model | Analytical verdict: CLEAR VIOLATION, CLEAR SAFE, or GENUINE GREY AREA with reasoning |
| Detected disagreement type | Consensus recommendation with semantic highlights |
| Platform context | Fallback rule-based summary if Claude unavailable |
SECTION 6
Each model was chosen to cover a distinct safety dimension with a different architecture and training dataset. Using models that all do the same thing would produce correlated outputs and defeat the purpose of comparison. For detailed model cards including architecture, training data, strengths, and limitations, see the Models tab.
SECTION 7
ModEval is a transparent tool and transparency includes acknowledging what it cannot do.
The eight models cannot be retrained, and novel slang may score incorrectly. Platform context is handled through AI policy alignment, not by adjusting model thresholds.
Guidelines are simplified from official pages. Real enforcement also depends on account history, human judgment, and community rules.
All models were trained primarily on English data. Non-English content produces unreliable scores and multilingual support is planned.
Images, video, audio, and multimodal content are outside the scope of the current system.
HuggingFace free inference and Hive V3 Playground may rate-limit under high traffic. Errors surface transparently rather than silently failing.
Inference Pipeline
Eight independent AI models, each trained on different datasets to detect a distinct safety dimension.
Enterprise APIs
The Hive AI
Detects: Sexual, Violence, Hate, Bullying, Spam
Trained on Purpose-built for T&S pipelines
Strengths
Limitations
Microsoft Azure
Detects: Hate speech, violence, sexual content, self-harm
Trained on Microsoft proprietary enterprise dataset
Strengths
Limitations
Google Cloud
Detects: 8 moderation categories including weapons, drugs, and more
Trained on Google proprietary dataset
Strengths
Limitations
OpenAI
Detects: Multi-category content violations across 13 safety dimensions including harassment, hate, violence, sexual content, and self-harm
Trained on Proprietary dataset curated by OpenAI
Strengths
Limitations
Open Source Models
Unitary AI
Detects: General toxicity, insults, threats, obscene language and severe toxic content
Trained on Jigsaw Toxic Comment Classification dataset (Wikipedia comments)
Strengths
Limitations
Cardiff NLP
Detects: Offensive and harassing language in social media context
Trained on Twitter data (SemEval 2019 OffComEval dataset)
Strengths
Limitations
Facebook AI Research
Detects: Hate speech targeting protected groups and identity-based attacks
Trained on DynaBench R4 — an adversarially collected dataset designed to be harder to game than previous benchmarks
Strengths
Limitations
Valurank
Detects: Biased and non-neutral language in text
Trained on Wikipedia revision history (WNC corpus) — edits where neutral editors removed biased language
Strengths
Limitations