Shared Task on Fine-grained Propaganda Detection @NLP4IF 2019

Task Description

Background

We refer to propaganda whenever information is purposefully shaped to foster a predetermined agenda. Propaganda uses psychological and rhetorical techniques to reach its purpose. Such techniques include the use of logical fallacies and appealing to the emotions of the audience. Logical fallacies are usually hard to spot since the argumentation, at first sight, might seem correct and objective. However, a careful analysis shows that the conclusion cannot be drawn from the premise without the misuse of logical rules. Another set of techniques makes use of emotional language to induce the audience to agree with the speaker only on the basis of the emotional bond that is being created, provoking the suspension of any rational analysis of the argumentation. All of these techniques are intended to go unnoticed to achieve maximum effect.

Technical Description

The goal of the shared task is to produce models capable of spotting sentences and text fragments in which propaganda techniques are used in a news article.

You will be provided with a corpus of about 500 news articles in which specific propagandistic fragments have been manually spotted and labeled. Two tasks are defined on the corpus:

1. In each article, text fragments that contain any of 18 propaganda techniques have been annotated. Your goal is to develop an intelligent algorithm able to detect the text-fragments in which a propaganda technique is used and to identify the technique. We refer to the task as FLC (Fragment-level Classification).
2. For a light version of the task, a sentence is considered as propagandistic if it contains one or more propagandistic fragments. This is a binary task in which, given a sentence, the correct label, either propaganda or non-propaganda, has to be selected. Notice that, if a propagandistic fragment spans over multiple sentences, all of them are marked as propagandistic. We refer to the task as SLC (Sentence-level Classification).

The competition is divided in 3 phases:

• Phase 1. You compete with the other participants to get the best performance on the development set. A live leaderboard will keep track of all the submissions.
• Phase 2. A test set will be released and you will have few days to submit your final predictions. Only the latest submission will be evaluated and considered to decide the overall winner. In this phase, no immediate feedback on the submission is provided. The winner of the competition will be determined upon the performance on the test set.
• Phase 3. All participants will be invited to submit a paper describing their system and review other partcipants' submissions. Accepted papers will be presented at the "2019 Workshop on NLP4IF: censorship, disinformation, and propaganda", co-located with the EMNLP-IJCNLP 2019 conference.
  We encourage short papers (4 pages), but we accept long papers (8 pages). For submission format see the section "Submission Guidelines" on the NLP4IF website.

Data Description

Input Articles

The input for both tasks will be news articles in plain text format. In the first phase, participants will be provided with two folders, train-articles and dev-articles (in the second phase we will release a third folder for the test set). Each article appears in one .txt file. The title is on the first row, followed by an empty row. The content of the article starts from the third row, one sentence per line. Each article has been retrieved with the newspaper3k library and sentence splitting has been performed automatically with NLTK sentence splitter.

Here is an example article (we assume the article id is 123456):

Notice that line numbers and superscripts are not present in the original article file, we have added them here in order to be able to reference sentences and text spans. The text is noisy, which makes the task trickier: for example in row 1 "Personal Liberty Poll Exercise your right to vote." is clearly not part of the title.

There are several propaganda techniques that were used in the article above:

• Line 1: the fragment “babies” is an instance of both Name_Calling and Labeling
• Line 3: the fragment “the party is more concerned with obstruction than it is with progress” is an instance of Black_and_White_Fallacy
• Line 4: the fragment “stupid and petty” is an instance of Loaded_Language;
• Line 4: “not looking as though Trump killed his grandma” is an instance of Exaggeration and Minimisation
• Line 4: “killed his grandma” is an instance of Loaded_Language

Gold Labels and Submission Format

Task FLC

The format of a tab-separated line of the gold label and the submission files for task FLC is:

 id   technique    begin_offset     end_offset 
	    

where id is the identifier of the article, technique is one out of the 18 techniques, begin_offset is the character where the covered span begins (included) and end_offset is the character where the covered span ends (not included). Therefore, a span ranges from begin_offset to end_offset-1. The first character of an article has index 0. The number of lines in the file corresponds to the number of techniques spotted. This is the gold file for the article above, article123456.txt:

123456    Name_Calling,Labeling      34    40
  123456    Black-and-White_Fallacy    299    368
  123456    Loaded_Language            400    416
  123456    Exaggeration,Minimization  607    653
  123456    Loaded_Language            635    653
	    

Task SLC

The format of a tab-separated line of the gold label and the submission files for task SLC is:

 article_id   sentence_id    label
	    

where article_id and sentence_id are the identifiers of the article and the sentence (the first sentence has id 1) and label={propaganda/non-propaganda}. Gold and submission files must have the same number of rows as the number of sentences, i.e. of lines, in the article. In order to help participants preparing a submission, we provide template prediction files, which have the same format of the gold files where label is replaced with ?. For example, the gold label and template files of task SLC for the article above would look as follows:

123456    1    propaganda
    123456    2    non-propaganda
123456    3    propaganda
123456    4    propaganda
    123456    5    non-propaganda
    123456    6    non-propaganda
	      

123456    1    ?
123456    2    ?
123456    3    ?
123456    4    ?
123456    5    ?
123456    6    ?
	      

Evaluation

Upon registration, participants will have access to their team page, where they can also download scripts for scoring both tasks. Here is a brief description of the evaluation measures the scorers compute.

Task FLC

FLC is a composition of two tasks: the identification of the propagandistic text fragments and the identification of the technique used in the fragment (18-class classification task). While F₁ measure is appropriate for a multi-class classification task, we modify it to account for partial matching between the spans. In addition, an F₁ value is computed for each propaganda technique.

Task SLC

SLC is a binary classification task in which the data is imbalanced. Therefore the official evaluation measure for the task is the standard F₁ measure. In addition, we will report Precision and Recall.

How to Participate

• Register to the challenge (follow the link on top). The task has finished. If you want to get the data, consider registering for the SemEval 2020 - Task 11 shared task.
• You will get an email with your team passcode. In case you do not receive the email, after checking your SPAM folder, then send us an email.
• Use the passcode on the top-right box to enter your team page. There you can download the data and submit your runs.
• Submit your predictions on the development set to check your performance evolution. Users submitting an abnormal number of submissions will be disqualified.
• The test set will be released on the final period of the competition. You will have few days to submit your final predictions and only the latest submission will be evaluated and considered to decide the overall winner.
• All participants will be invited to submit a paper describing their system and review other partcipants' submissions. Accepted papers will be presented at the "2019 Workshop on NLP4IF: censorship, disinformation, and propaganda" workshop.

Rules