Incremental Pay-as-You-Go Fact Checking with Minimal User Effort
I. Introduction
The open nature of the Web enables users to produce and propagate any content without authentication, which has been exploited to spread thousands of unverified claims via millions of online documents. Maintenance of credible knowledge bases thus has to rely on fact checking that constructs a trusted set of facts through credibility assessment. Due to an inherent lack of ground truth information and language ambiguity, fact checking cannot be done in a purely automated manner without compromising accuracy. However, state-of-the-art fact checking services, rely solely on human validation, which is costly, slow, and non-transparent. This paper presents FactCATCH (inCremental pAy-as-you-go facT CHecking), a human-in-the-loop system to guide users in fact checking that aims at minimisation of the invested effort. It supports incremental quality estimation, mistake mitigation, and pay-as-you-go instantiation of a high-quality fact database.
II. Contributions
1. Incremental quality estimation:
FactCatch features an
efficient probabilistic model to reason on the credibility of claims. It
exploits mutual reinforcing relations between Web sources and claims to
assess the credibility of unchecked claims.
2. Effort minimisation:
FactCatch guides a user in the
fact checking process, while reducing the amount of validation effort needed to
achieve a specific level of result precision.
3. Mistake mitigation:
FactCatch helps to identify
suspicious user input; claims that may have been validated by mistake.
4. Pay-as-you-go instantiation:
FactCatch supports the separation
of credible and non-credible claims at any time, to serve
downstream applications with a high-quality fact database.
5. Early termination:
FactCatch includes means to stop fact
checking to avoid to spend effort on marginal improvements of the quality
of the fact database.
