TruthinessGPT is an Accuracy Auditor that will help identify statements, facts, assertions that you should double-check.
Coined by Stephen Colbert, Truthiness is the belief or assertion that a particular statement is true based on the intuition or perceptions of some individual or individuals, without regard to evidence, logic, intellectual examination, or facts. Wikipedia
AI responses often seem completely true, causing people to assume they don’t need to fact-check responses. This is a proof-of-concept around having AI reflect on itself as a way to help humans reduce the hallucinations and errors from generative AI. We are working to build this into our clients tools to improve outputs.
Paste in text you would like analyzed. The output will select the top 3 assertions an assign a probability:
High probability = more truthiness
Low probability = less truthiness
Understanding Probability Assignments for Assertions
When evaluating the accuracy of assertions, we assign a probability percentage (ranging from 0% to 100%) to indicate the likelihood that a statement is true. Here’s a breakdown of how we determine low and high probabilities:
High Probability (70% – 100%)
- Definition: A high probability indicates a strong confidence in the truthfulness of the assertion. This means that the claim is well-supported by credible evidence and sources.
- Criteria for High Probability:
- Multiple Reliable Sources: The assertion is confirmed by several reputable organizations, studies, or experts in the field.
- Consensus in the Field: There is a general agreement among professionals or researchers regarding the statement.
- Recent Data: The information is based on the latest research or statistics, ensuring that it reflects current knowledge.
- Example: If a statement claims that “Cigarette smoking is the primary risk factor for lung cancer, linked to about 80%-90% of lung cancer deaths,” and this is supported by numerous studies from the CDC and WHO, it would receive a high probability rating (e.g., 95%).
Low Probability (0% – 30%)
- Definition: A low probability indicates significant doubt about the truthfulness of the assertion. This suggests that the claim is either unverified, misleading, or contradicted by credible evidence.
- Criteria for Low Probability:
- Lack of Evidence: The assertion is not supported by reliable sources or research.
- Contradictory Information: There are credible sources that provide evidence against the claim.
- Outdated Data: The information is based on old studies or statistics that may no longer be relevant.
- Example: If a statement claims that “Lung cancer is less deadly than breast cancer,” but numerous studies show that lung cancer causes more deaths, this would receive a low probability rating (e.g., 20%).
Summary
- High Probability (70% – 100%): Strong evidence and consensus support the assertion.
- Low Probability (0% – 30%): Lack of evidence or contradictory information raises doubts about the assertion.
By using this framework, we can systematically assess the accuracy of claims and provide a clear understanding of their reliability.
