Appropriateness (1-5)
Does this paper fit in the event? Both empirical and theoretical results are welcome.
5: Certainly.
4: Probably.
3: Unsure.
2: Probably not.
1: Certainly not.
Clarity (1-5)
For the reasonably well-prepared reader, is it clear what was done and why? Is the paper well-written and well-structured?
5 = Very clear.
4 = Understandable by most readers.
3 = Mostly understandable to me with some effort.
2 = Important questions were hard to resolve even with effort.
1 = Much of the paper is confusing.
Originality / Innovativeness (1-5)
How original is the approach? Does this paper break new ground in topic, methodology, or content? How exciting and innovative is the research it describes?
Note that a paper could score high for originality even if the results do not show a convincing benefit.
5 = Surprising: Noteworthy new problem, technique, methodology, or insight.
4 = Creative: Relatively few people in our community would have put these ideas together.
3 = Somewhat conventional: A number of people could have come up with this if they thought about it for a while.
2 = Rather boring: Obvious, or a minor improvement on familiar techniques.
1 = Significant portions have actually been done before or done better.
Soundness / Correctness (1-5)
First, is the technical approach sound and well-chosen? Second, can one trust the claims of the paper — are they supported by proper experiments, proofs, or other argumentation?
5 = The approach is very apt, and the claims are convincingly supported.
4 = Generally solid work, though I have a few suggestions about how to strengthen the technical approach or evaluation.
3 = Fairly reasonable work. The approach is not bad, and at least the main claims are probably correct, but I am not entirely ready to accept them (based on the material in the paper).
2 = Troublesome. There are some ideas worth salvaging here, but the work should really have been done or evaluated differently, or justified better.
1 = Fatally flawed.
Meaningful Comparison (1-5)
Does the author make clear where the problems and methods sit with respect to existing literature? Are the references adequate? Are any experimental results meaningfully compared with the best prior approaches?
5 = Precise and complete comparison with related work. Good job given the space constraints.
4 = Mostly solid bibliography and comparison, but I have some suggestions.
3 = Bibliography and comparison are somewhat helpful, but it could be hard for a reader to determine exactly how this work relates to previous work.
2 = Only partial awareness and understanding of related work, or a flawed empirical comparison.
1 = Little awareness of related work, or lacks necessary empirical comparison.
Thoroughness (1-5)
Does this paper have enough substance, or would it benefit from more ideas or results?
Note that this question mainly concerns the amount of work; its quality is evaluated in other categories.
5 = Contains more ideas or results than most publications in this conference; goes the extra mile.
4 = Represents an appropriate amount of work for a publication in this conference. (most submissions)
3 = Leaves open one or two natural questions that should have been pursued within the paper.
2 = Work in progress. There are enough good ideas, but perhaps not enough results yet.
1 = Seems thin. Not enough ideas here for a full-length paper.
Impact of Ideas or Results (1-5)
How significant is the work described? If the ideas are novel, will they also be useful or inspirational? If the results are sound, are they also important? Does the analysis in the paper bring new insights into the nature of the problem?
5 = Will affect the field by altering other people’s choice of research topics or basic approach.
4 = Some of the ideas or results will substantially help other people’s ongoing research.
3 = Interesting but not too influential. The work will be cited, but mainly for comparison or as a source of minor contributions.
2 = Marginally interesting. May or may not be cited.
1 = Will have no impact on the field.
Recommendation (1-5)
There are many good submissions competing for slots at this event; how important is it to feature this one? Will people learn a lot by reading this paper or seeing it presented?
In deciding on your ultimate recommendation, please think over all your scores above. But remember that no paper is perfect, and remember that we want a conference full of interesting, diverse, and timely work. If a paper has some weaknesses, but you really got a lot out of it, feel free to fight for it. If a paper is solid but you could live without it, let us know that you’re ambivalent. Remember also that the author has a couple of weeks to address reviewer comments before the camera-ready deadline.
Should the paper be accepted or rejected?
5 = Exciting: I’d fight to get it accepted
4 = Worthy: I would like to see it accepted
3 = Borderline: I’m ambivalent about this one
2 = Mediocre: I’d rather not see it in the conference
1 = Poor: I’d fight to have it rejected
Reviewer Confidence (1-5)
5 = Positive that my evaluation is correct. I read the paper very carefully and am very familiar with related work.
4 = Quite sure. I tried to check the important points carefully, and checked for uncited prior work. It’s unlikely, though conceivable, that I missed something that should affect my ratings.
3 = Pretty sure, but there’s a chance I missed something. Although I have a good feel for this area in general, I did not carefully check the paper’s details, e.g., the math, experimental design, or novelty.
2 = Willing to defend evaluation, but it is fairly likely that I missed some details, didn’t understand some central points, or can’t be sure about the novelty of the work.
1 = Not my area, or paper is very hard to understand. My evaluation is just an educated guess.
Detailed Comments
The paper presents:
- A comparative analysis of how German and Italian press represents each other’s country;
- CorSEGIN, a corpus of 600 expert-annotated news article (300 per country) for document-level target sentiment and emotion recognition.
- A zero-shot evaluation of one proprietary and five open weight LLMs against the expert-annotated standard.
The author finds that a higher Inter-annotator agreement on German texts w.r.t. Italian’s, and argue that LLMs are useful for filtering but can’t replace expert annotation on this task, but could be used to reduce the amount of work for the annotators.
The presented corpus fills a genuine gap as target-based document-level sentiment and emotion recognition tasks, especially on long journalistic texts, as is a considerably harder and less studied task than the headline counterpart. Also, the bilingual design makes it well suited to the stated societal goal. The decision to release non-aggregated annotations alongside the gold standard also enables future perspectivist works. The practical framing of the evaluation (i.e. pre-filtering for the Observatory) is grounded in a real-world application which is appreciated.
However there are some weaknesses that should be addressed:
- Table 4 and 6 shows some inconsistencies regarding the number of examples that have been shown to Chat-GPT w.r.t. the open weight model. On one hand, the total number of ChatGPT examples in the evaluation seems to be 300 per-language, while for Table 6, the total number of examples in the matrix is 1350, which is 270 x 5 per language. This means that while for the open weight models 30 examples have been used as a dev set, ChatGPT has been evaluated on a different split that containts all examples (the 30 from the dev set too). This is an orange-to-apples situation, and the results for ChatGPT tested on the same 270 examples used for the open-weight models should be reported.
- Table 5 and table 7 show no baselines: given the strong class imbalace in the Emotion recognition task, per-class f1 are hard to interpret without some baselines. I would suggest at least to report a random baseline and a “always-most-frequent-class-prediction” baseline. On this, you also implicitly assume that using GPT-style LLMs in zero-shot is the simplest/cheapest option, but I would argue that, to prove this, a fine-tuned supervised baseline would’ve been needed. Some long-context bert-style encoder in a k-fold training/evaluation scenario, or a fine-tuned version of the same model trained on few examples with something like SetFit, would help the claim stand (if true).
- Results obtained with a single prompt template and with a single greedy run per model are a bit shaky. Zero-shot is notoriously very prompt-sensitive, some indication of how the performance varies across prompt variants (even on the dev set) would make the conclusion more solid. Similarly, a lot of per-class counts of the Emotion recognition tasks are very small (i.e. Optimism/trust in Italian has only 13 examples), so the difference between open-source models are likely within noise.
- The 32.8% reduction in workload is a very optimistic view. It assumes that polarized prediction (i.e. negative and positive) are accepted without review, but from Table 4 the precision of those class is not perfect and covers only Sentiment analysis (not Emotion recognition, which I assume the Observatory is also interested into). I would frame the reduction as a precision/recall trade-off rather than a flat free-savings number.
- Also, the Inter-annotator agreement and Model performance correlation (Section 5.3) is not statistically significant, and I wouldn’t report it as it is.
Overall the resource is nice, but the experiments and conclusions drawn from it are somewhat weak.
Questions for Authors
- For Emotion Recognition: is the reported F1 scores the F1 for the “Yes” class? How exactly is the Macro-F1 computed and how are the multi-label confusion matrix in Table 9 constructed?
- Which GPT-5 snapshot/version was used?
- Can researchers without an Infojuice subscription retrieve the corpus texts from the released identifiers, or is platform access effectively required?
Confidential Comments for Committee
You may wish to withhold some comments from the authors, and include them solely for the committee’s internal use. For example, you may want to express a very strong (negative) opinion on the paper, which might offend the authors in some way. Or, perhaps you wish to write something which would expose your identity to the authors. If you wish to share comments of this nature with the committee, this is the place to put them.