Does the paper fit into the scope of the ECML-PKDD research track?
Yes
Please provide a short summary of the paper.
This paper presents a framework for analyzing jailbreak-related signal in the internals of LLMs, in a white box setup.
The core proposal is to take several layers and measure their evolution w.r.t. tokens using entropy traces. The resolution is entirely empirical, on three LLMs, with several benign and harmfull benchmarks.
The proposal functions in this setup, when prompts are not adversarial.
Please rate the technical soundness of the paper: Are the claims well
supported by theoretical analysis and/or experimental results?
medium (e.g., some experiments are missing)
Please rate the contribution to the state of the art of the paper: Is the method
or task introduced in this paper original? Or is it a rather incremental
advancement of the state-of-the-art?
high (e.g., meaningful, novel problem setting and/or innovative solution)
Please rate the quality of the presentation: Is the paper clearly written? Is it
well structured? Is it easy to understand?
medium (e.g., needs some effort to understand but ok after some editing)
Please rate the reproducibility: Does the paper give sufficient information to
reproduce the results? Are code and data open source?
high (information sufficient, code and data accessible)
What are 3 strong points of the paper?
* clear technical proposal
* convincing results, yet on a restricted variety of models
* the general phenomenon that has been uncovered is interesting and must be
deepened
What are 3 weak points of the paper?
* very strict and restrictive setup: white box and frozen LLMs
* hard to be convinced by the generalization claims to all LLMs, due to the narrow evaluation setup
Please enter a detailed review of the paper that explains the answers to the
questions you have given above. Please explain your ratings of technical
soundness, novelty, quality of presentation, reproducibility, strong and weak
points. Please be specific and constructive, e.g., for weak points that can be
improved, add suggestions how they can be improved. If e.g., relevant related
work has not been considered, please include the references to this work.
I appreciated the direction taken by this paper. I think it gives an interesting angle on the internals of LLMs facing harmfull incomoing prompts.
I assume, considering the very narrow and precise technical framing of the proposed solution, that many other variants or failed solutions have been tested before this proposal; it would have been interesting to discuss them all in a larger scope in the current setup, even to compare with less efficient strawmen, to get a larger understanding of the internal dynamics at play.
The setup is restricted, and I am not sure we can be as certain of the generalization capability of the current study to LLMs in general, as only 3, equivalently small sized, non-thinking, LLMs have been tested.
On the runtime side: the authors did not study the stability of their proposal in current scenarios where constant fine-tuning is happening in production on the model; since the weights will constantly shift, how does this affects the continuous monitoring capability mentioned in conclusion of this paper?
In summary, I appreciate the direction, but I think the current experimental part (again, no theory backs the claims) is weak and does not permit a large confidence in the reproducibility of the results in LLMs in general.
Please give an overall rating to the paper into the following categories. The
numerical scores will be used to rank the papers in the CMT.
Weak reject (- 1): borderline, tending to reject
Reviewer 2
Does the paper fit into the scope of the ECML-PKDD research track?
Yes
Please provide a short summary of the paper.
The paper proposes a training-free jailbreak detection method based on token-level predictive entropy trajectories extracted from intermediate layers of frozen LLMs through the logit lens. Rather than relying on static entropy aggregates, the method examines dynamic trend features over token positions, including Kendall's tau, Spearman's rho, and monotonicity. Experiments on Llama-3.1-8B, Qwen3-8B, and Gemma-7B show that these dynamic features separate benign and harmful prompts more clearly than static entropy features. The strongest signal appears in intermediate layers rather than at the final layer. The paper also reports that separability drops substantially when benign prompts are structurally similar to jailbreak prompts, as in the benign split of JailbreakBench.
Please rate the technical soundness of the paper: Are the claims well
supported by theoretical analysis and/or experimental results?
medium (e.g., some experiments are missing)
Please rate the contribution to the state of the art of the paper: Is the method
or task introduced in this paper original? Or is it a rather incremental
advancement of the state-of-the-art?
medium (e.g., interesting extension of established work)
Please rate the quality of the presentation: Is the paper clearly written? Is it
well structured? Is it easy to understand?
high (e.g., a pleasure to read, appropriate formalization of central concepts,good figures, insightful explanations)
Please rate the reproducibility: Does the paper give sufficient information to
reproduce the results? Are code and data open source?
medium (e.g., some information is missing, but that could be fixed)
What are 3 strong points of the paper?
1. The paper identifies a clear and interesting signal: jailbreak prompts are better characterized by the shape of intermediate-layer entropy trajectories than by static entropy magnitude.
2. The method is simple, training-free, and interpretable, using rank-based features that are relatively robust across model families.
3. The experimental analysis covers multiple model families and several harmful/benign datasets, and the depth-wise analysis provides useful evidence that the signal peaks in intermediate layers.
What are 3 weak points of the paper?
1.The empirical comparison lacks important baselines, including perplexity-based
filters, first-token output-distribution detectors, and lightweight internal-representation probes.
2.The method depends on an untuned logit lens and model-specific choices such as
harmful direction and probe-layer depth, but the paper does not sufficiently quantify sensitivity or leakage risk.
3.The evaluation mainly reports AUROC and does not provide operating-point metrics such as TPR at fixed FPR, making deployment relevance harder to assess.
Please enter a detailed review of the paper that explains the answers to the
questions you have given above. Please explain your ratings of technical
soundness, novelty, quality of presentation, reproducibility, strong and weak
points. Please be specific and constructive, e.g., for weak points that can be
improved, add suggestions how they can be improved. If e.g., relevant related
work has not been considered, please include the references to this work.
The paper is technically sound overall and presents a coherent empirical study. The formulation of entropy trajectories and the use of rank-based dynamic features are reasonable, and the experiments support the main claim that dynamic entropy trends are more discriminative than static entropy aggregates. The depth analysis is also useful. The reported results show that the signal is strongest around intermediate layers and often weakens at the final layer, which is consistent with the paper's interpretation that jailbreak-related structure is more visible in mid-network representations.
Please give an overall rating to the paper into the following categories. The
numerical scores will be used to rank the papers in the CMT.
Weak accept (+ 1): borderline, tending to accept
Reviewer 3
Does the paper fit into the scope of the ECML-PKDD research track?
Yes
Please provide a short summary of the paper.
The authors propose to separate benign LLM prompts from jailbreak attempts by
means of dynamic measures of entropy in the prompt itself. They consider increasingly long prefixes of the whole prompt and extract various signals related to the model's predictive entropy. Their evaluation on 3 LLMs (LLama3.1, Qwen3, Gemma) shows that mere "static" aggregates of these signals (such as, trivially, their mean) carry little informative value, whereas dynamic ones allow for much better classification of potentially harmful prompts.
Please rate the technical soundness of the paper: Are the claims well supported by theoretical analysis and/or experimental results?
medium (e.g., some experiments are missing)
Please rate the contribution to the state of the art of the paper: Is the method
or task introduced in this paper original? Or is it a rather incremental advancement of the state-of-the-art?
medium (e.g., interesting extension of established work)
Please rate the quality of the presentation: Is the paper clearly written? Is it
well structured? Is it easy to understand?
high (e.g., a pleasure to read, appropriate formalization of central concepts,good figures, insightful explanations)
Please rate the reproducibility: Does the paper give sufficient information to
reproduce the results? Are code and data open source?
high (information sufficient, code and data accessible)
What are 3 strong points of the paper?
* Training-free jailbreak detection has few other examples in the literature
* Chooses well-established metrics (Kendall's, Spearman's)
* Adequate experimental evaluation to support the central claim (dynamic features
improve detection over static ones)
What are 3 weak points of the paper?
* Lack of comparison with any baseline
* Monotonicity does not sound as compelling as the other metrics
* Probing a full intermediate layer might be hard on large LLMs
Please enter a detailed review of the paper that explains the answers to the
questions you have given above. Please explain your ratings of technical
soundness, novelty, quality of presentation, reproducibility, strong and weak
points. Please be specific and constructive, e.g., for weak points that can be
improved, add suggestions how they can be improved. If e.g., relevant related
work has not been considered, please include the references to this work.
The authors are right: beyond perplexity, to my knowledge (and theirs) there is
surprisingly little research on training-free jailbreak detection techniques, which may effectively complement other guardrails and improve the overall safety of deployed models.
The paper evaluates a reasonable collection of features. Kendall's and Spearman's
coefficients are well-established. Monotonicity seems novel, but has a reasonable
definition (although note my remarks below). The paper's claims about the effectiveness of dynamic features are backed adequately, at least for Kendall's and Spearman's.
I should commend the authors for the high quality of the manuscript in terms of clarity and lack of mistakes/typos.
I still wish the authors had included a comparison with any baseline method, even just perplexity. Even though I can intuitively understand the proposed metrics may outperform it, it would have been interesting to look at actual figures.
Monotonicity does not fare as well as the other metrics, and I have additional
reservations on it. For instance, what if the malicious prompt mixes harmful
instructions with "platitudes" on which the model's uncertainty sharply decreases?
With enough such platitudes, I suspect that the `mono` measure might drop below detection.
Perhaps a windowed variant of monotonicity might be more robust against this.
The way "harmful direction" is determined empirically is also left vague, which may hinder other author's attempt to evaluate the role of monotonicity in other models. (In this sense, I appreciate that authors attached code to their supplemental material, but an explanation in the Appendix would have been easier to read).
Lastly, another practical detail left out of the presentation is: how invasive/expensive is it to probe a full layer of a real-world LLM? All models considered here are in the 7-8Bparameter "weight class". How much more effort does the method require if we move to, say, a 14B-parameter model?
All things considered, and although monotonicity shows room for improvement, I am in favour of accepting the paper, given the relevance to the venue, the real-world importance of harmful prompt detection, and the relative lack of comparable
approaches in the literature.
Please give an overall rating to the paper into the following categories. The
numerical scores will be used to rank the papers in the CMT.