Finding cracks in content moderation

Abstract

Sales and discussions of illicit drugs have become commonplace online, including on social media. Social media platforms hosting user-generated content must therefore moderate harmful drug content. However, this is a difficult task due to the vast amount of jargon used in drug discussions. Previous works on drug jargon detection were limited to extracting a list of terms. However, systems relying on a banlist of words have limitations. First, they are trivially evaded using word substitutions. Second, they are cannot distinguish whether a drug euphemism (e.g., pot, crack) is used as a drug jargon or not. An effective drug content moderation system must be trained to find drug jargon using contexts, rather than relying on a banlist. Since the language around drugs is difficult and constantly changing, manually annotated datasets for training on this task are not only expensive to create but also prone to becoming obsolete. We present JEDIS, a system that detects illicit drug jargon terms by learning on distantly supervised data. We manually annotate two datasets from two sources, Reddit and Silk Road Forums, to evaluate drug jargon detection. Our experiments show JEDIS outperforms state-of-the-art word-based baselines in drug jargon detection by 26.16 F1-score and by 9.27 F1-score on the two evaluation datasets. We also use JEDIS in extracting a list of drug jargon terms from the corpus, and find it is robust against pitfalls that other systems face.