CriteriaBot

criteriabot.io

I needed a classifier for nuanced, subjective buckets that fell outside of typical ML use-cases (e.g., "is this a spoiler?", "is this factually correct?", "is this user being mean?"). I ended up really happy with the architecture I built to solve it, so I rolled it out as a standalone API and service called CriteriaBot. WHAT IT DOES: You give it content and plain-English criteria. It gives you a true/false verdict on whether the content meets those criteria. HOW IT WORKS: In addition to a traditional classifier, the classification request is routed through a pool of small, open-weight LLMs to achieve a consensus verdict. I built a pre-vote factorization machine that selects a sub-pool of LLMs optimized for signal strength based on the embedding of the subject/category. A second factorization machine then reads the votes and the embedding to arrive at a single verdict. That verdict is dynamically modified based on the user's history of agreement/disagreement with the models in semantically similar evaluations. The models are also hooked up to Wikipedia and Wolfram to support edge cases requiring current information or mathematical grounding. FINDINGS: * With the same harn...