ICML Workshop Reviews

Interesting internal-entropy signal for harmful prompt separation, but overclaims and weak controls undermine the jailbreak-detection claim

Official Reviewby Reviewer 3E8S11 May 2026, 06:28 (modified: 18 May 2026, 14:09)Program Chairs, Area Chairs, Reviewers.
Summary:

Summary of the paper

The paper proposes a training-free method for detecting jailbreak or harmful prompts by analyzing token-level predictive entropy trajectories at intermediate layers of frozen LLMs. For each token and layer, the authors project hidden states through the model’s unembedding matrix using a logit lens, compute vocabulary entropy, and summarize the resulting entropy trace with static features such as mean/max/std and dynamic features such as Kendall’s (\tau), Spearman’s (\rho), and a monotonicity score.

The main empirical claim is that dynamic entropy-trend features separate harmful from benign prompts better than static prompt-level entropy aggregates, that the signal is strongest at intermediate layers, and that this pattern appears across Llama-3.1-8B, Qwen3-8B, and Gemma-7B on combinations of AdvBench, HarmBench, StrongREJECT, UltraChat, WildJailbreak benign, and JailbreakBench benign. The claimed relevance to epistemic intelligence is that harmful or jailbreak-like behavior is reflected in structured uncertainty dynamics inside the model rather than only in prompt text or output behavior.

Relevance to EIML@ICML 2026

The paper has a plausible but somewhat indirect fit to EIML. It addresses uncertainty-aware foundation-model analysis and AI safety under adversarial prompting. The use of token-level entropy trajectories is relevant to uncertainty-aware generative AI, and the focus on jailbreak detection connects to epistemic blind spots and unsafe overconfident behavior.

However, the paper does not clearly operationalize epistemic uncertainty in the usual sense. The entropy being measured is single-model next-token predictive entropy under a logit-lens projection; this quantity can reflect linguistic ambiguity, tokenization effects, prompt length, model calibration, or representation-misalignment artifacts, not necessarily epistemic uncertainty or ignorance. The work is therefore better described as an internal-signal study for harmful prompt detection than as a foundational contribution to epistemic uncertainty, unknown unknowns, imprecise probability, or higher-order uncertainty.

The strongest EIML fit is in AI safety as uncertainty-aware monitoring for foundation models. The fit is weaker for unknown unknowns, abstention/refusal, value uncertainty, continual/open-world learning, and decision-theoretic foundations, none of which are substantially developed.

Strengths:

The central idea is simple and potentially useful: rather than only inspecting prompt text or generated responses, the paper probes internal entropy dynamics over token positions and layers. This is a meaningful direction for safety monitoring because many deployed defenses are external to the model or output-based, while the paper attempts to characterize an internal computational signal.

The paper evaluates more than one model family and more than one harmful-prompt benchmark. Testing Llama, Qwen, and Gemma on several benign/harmful dataset pairings is a reasonable first step toward assessing whether the phenomenon is architecture-specific.

The feature design is interpretable. Kendall’s (\tau), Spearman’s (\rho), and monotonicity are transparent scalar summaries of entropy trajectories. This makes the method easier to inspect than a learned hidden-state classifier and supports the paper’s stated goal of understanding which entropy-derived properties carry signal.

The authors include a valuable hard-negative control using JailbreakBench benign prompts. The large performance collapse on this setting is important evidence about the method’s limitations, even though I disagree with the paper’s relatively benign interpretation of that collapse.

The appendix provides more layer-wise feature results than the main paper, which helps reveal where the main claims are robust and where they are overstated.

Weaknesses:

The most serious concern is that the evaluation may be detecting dataset/style structure rather than jailbreak intent or semantic harmfulness. The primary experiments pair broad benign datasets such as UltraChat or WildJailbreak benign with harmful-prompt datasets. The hard-negative evaluation with JailbreakBench benign collapses to below-random or near-random AUROC in many cases; for example, Table 6 reports mean monotonicity AUROC of 0.348 for Llama, 0.347 for Qwen3, and 0.436 for Gemma under JailbreakBench benign. This is not a minor robustness boundary: it directly challenges the claim that the method detects harmful intent or jailbreaks rather than prompt structure, topic, or safety-adjacent wording.

The paper overstates the “intermediate-layer peak” claim. The main text says the signal peaks consistently around 69% depth and degrades at the final layer, but the appendix shows a more mixed picture. For Llama, monotonicity is higher at L26 than L22, and Kendall/Spearman are also higher at L26 than L22. For Gemma, Kendall/Spearman appear strongest around L15 rather than L19. For Qwen, some dynamic and static features remain very strong at later layers. The intermediate-layer observation is interesting, but the current wording is too categorical.

The claim that static features are weak is not fully supported. The main comparison in Table 3 uses only a small subset of static features on one dataset pair, while the appendix shows several static features performing strongly in some settings, including final-layer mean entropy or early/max-type statistics. The paper should narrow the claim to “some static features at the chosen focal layer are less stable than rank-based trend features” rather than asserting that static aggregates generally carry little signal.

The term “jailbreak detection” is not adequately validated. The datasets mostly contain harmful requests or red-teaming prompts, but the paper does not evaluate whether the prompt actually succeeds in jailbreaking an aligned model, whether the target model refuses, or whether the model produces policy-violating content. The paper studies prompt labels, not jailbreak success. This matters because a detector of harmful requests is not the same as a detector of successful jailbreaks.

There is possible model-choice mismatch. The paper frames the problem around aligned LLMs and jailbreaks, but the Hugging Face handles listed include base-style model names such as meta-llama/Llama-3.1-8B and google/gemma-7b, not clearly instruction-tuned/aligned variants. If these are not aligned chat models, then the connection to jailbreak behavior is substantially weakened.

The evaluation lacks strong baselines. The paper compares entropy features against other entropy features, but not against standard or recent jailbreak detectors, perplexity-based detectors, hidden-state linear probes, refusal-direction methods, prompt classifiers, output-distribution detectors, or simple lexical/length/topic baselines. Without these comparisons, it is difficult to know whether the proposed signal is useful beyond known prompt-distribution artifacts.

The direction and focal-layer choices are insufficiently specified. The paper states that the harmful direction is fixed from held-out data, but does not explain the held-out split, whether it is disjoint by dataset, whether the focal layer was selected on validation data, or whether the same datasets influenced the reported choices. This creates a risk of implicit tuning on the evaluation distribution.

There is limited statistical reliability. The paper reports many AUROC values but no confidence intervals, no multiple random subsampling runs, no variance over seeds, and no thresholded operational metrics such as TPR at low FPR. Since larger datasets are randomly subsampled to enforce 1:1 balance, the lack of seeds and repeated subsampling is a reproducibility and reliability issue.

The paper’s interpretation of WildJailbreak benign is confusing. It states that performance is higher when the safe set is WildJailbreak than UltraChat and presents this as evidence under “harder negative” conditions, but higher AUROC usually suggests an easier separation. This should be clarified.

Limitations:

Technical quality

The empirical pipeline is understandable, but the technical support for the main claims is incomplete. The logit-lens entropy definition is mathematically straightforward, yet the paper does not specify several important implementation details: whether final layer normalization/RMSNorm is applied before unembedding intermediate states, whether entropy is computed over the full vocabulary or an approximation, how prompts are truncated or templated, how special tokens are handled, and whether the first token positions are excluded consistently. These choices can materially change entropy trajectories.

The dynamic features are reasonable for measuring monotonic trends, but their connection to epistemic uncertainty is not established. A single deterministic model’s token entropy is not sufficient to distinguish epistemic uncertainty from aleatoric ambiguity, prompt-distribution artifacts, or calibration effects. The paper should either avoid epistemic claims or add experiments that more directly test epistemic uncertainty, such as distribution-shift calibration, uncertainty under paraphrase/topic controls, ensemble disagreement, or abstention/failure prediction.

The hard-negative results substantially weaken the detector interpretation. AUROC values below 0.5 under the prescribed direction indicate not merely “overlap” but systematic failure or inversion in some cases. Treating this primarily as a diagnostic rather than a major limitation is not technically convincing.

The layer-wise evidence is interesting but not decisive. The paper should avoid strong claims about a universal 69%-depth focal layer unless it pre-registers layer selection or evaluates on held-out datasets after selecting the layer. The appendix suggests that the location and strength of the best feature vary by model and feature family.

Originality and relation to prior work

The combination of logit-lens entropy trajectories with rank-based trend features is a nice and interpretable contribution. It appears more original than simply training a classifier on hidden states, and it may provide a useful diagnostic perspective on internal safety-relevant dynamics.

That said, the novelty is somewhat incremental relative to existing work on internal representations for safety, refusal directions, perplexity/entropy-based detection, and output-distribution jailbreak detection. The paper cites several of these areas but does not empirically compare against them. As a result, it is difficult to tell whether the contribution is a genuinely stronger detection method, a new mechanistic observation, or a repackaging of known prompt-distribution separability using entropy-trend terminology.

The related-work section would also benefit from pruning or strengthening some citations. A few cited works appear only loosely related to jailbreak defense or LLM safety, while highly relevant detector baselines are not implemented. The paper’s positioning would be more convincing if it focused less on broad coverage and more on direct comparisons.

Significance and impact

If the main claim were validated, the work could be useful for white-box safety monitoring: a single-forward-pass internal score that flags jailbreak-like prompts before generation could complement output-based filters and refusal mechanisms. The approach is interpretable, lightweight relative to multi-sampling defenses, and potentially useful for model debugging.

The impact actually supported by the paper is narrower. The evidence currently supports the existence of entropy-trajectory differences between certain harmful-prompt datasets and certain benign-prompt datasets. It does not yet establish robust jailbreak detection, successful-attack prediction, semantic harmfulness detection, epistemic uncertainty estimation, or safe deployment behavior under realistic hard negatives. The JailbreakBench benign failure is especially limiting for practical impact because real users often ask legitimate questions involving policy-sensitive topics.

Thus, I see the work as a promising diagnostic observation, not yet as a reliable safety method.

Clarity and presentation

The paper is generally readable and well organized. The problem statement, feature definitions, and main experimental tables are easy to follow. Figure 1 and Figure 2 help illustrate the intuition that harmful and benign prompts can have different entropy-trend distributions.

However, the presentation repeatedly overstates what the experiments show. Terms such as “harmful intent,” “jailbreak behavior,” “epistemic uncertainty,” and “intermediate-layer peak” are used more strongly than the evidence justifies. The paper should more carefully distinguish:

• harmful request classification vs. successful jailbreak detection;
• predictive entropy vs. epistemic uncertainty;
• dataset separability vs. semantic intent detection;
• exploratory layer analysis vs. validated focal-layer selection.

The appendix is useful but also reveals tensions with the main narrative, especially regarding static features and layer peaks. Those tensions should be addressed in the main text rather than left implicit.

Evaluation, reproducibility, and resources

Reproducibility is currently insufficient. The paper gives model names, probe layers, datasets, and feature definitions, but omits many details needed to reproduce the reported numbers:

• exact dataset sizes after filtering and subsampling;
• random seeds and number of subsampling runs;
• confidence intervals or standard errors;
• prompt formatting/chat templates;
• truncation length and tokenization details;
• whether instruction-tuned or base checkpoints were used;
• whether final normalization is applied in the logit lens;
• hardware, runtime, and memory overhead;
• code availability;
• exact held-out split used to choose harmful direction and possibly focal depth.

The evaluation should also include baselines beyond entropy-feature variants. At minimum, I would expect comparisons with prompt perplexity, length/topic lexical baselines, a simple text classifier, a hidden-state linear probe, and at least one recent jailbreak-detection method. The paper cites several alternatives but does not benchmark against them.

The use of AUROC is acceptable for exploratory ranking, but deployment claims require thresholded metrics, especially false-positive rates on benign but safety-sensitive prompts. The JailbreakBench benign result shows that this is not a cosmetic issue.

Ethics, safety, and broader impacts

The work has potential safety benefits if used as one component of a white-box monitoring system. It may help identify internal signatures of adversarial prompting and could support safer fallback, abstention, or additional scrutiny before generation.

However, the current method also poses safety risks if overtrusted. The hard-negative failure suggests that it may flag benign policy-sensitive queries as harmful or fail to distinguish safe educational, fictional, or compliance-oriented prompts from genuinely malicious ones. This could cause unfair or overly broad refusal behavior, especially for users discussing sensitive topics in legitimate contexts.

Conversely, a detector based on a single simple entropy-trend score may be easy to evade once known, particularly for white-box or adaptive attackers. The paper does not evaluate adaptive attacks against the entropy feature.

The paper should include a more explicit limitations and misuse discussion. In particular, it should warn against deploying the method as a standalone jailbreak detector, discuss false positives on safety-adjacent benign content, and clarify that the reported scores do not establish robust safety under distribution shift.

Questions for the authors

1. Were the evaluated checkpoints base models or instruction-tuned/aligned chat models? If base models were used, in what sense are the prompts “jailbreaks” rather than harmful requests?

2. How exactly was the harmful entropy direction selected? What held-out data was used, and was it disjoint from all reported dataset pairs?

3. Was the (\sim 69%) focal layer selected before seeing the evaluation results, or chosen after exploratory layer-wise analysis?

4. Can the method distinguish harmful intent from benign safety-sensitive prompts after controlling for topic, length, and prompt style? The JailbreakBench benign results suggest this is currently not the case.

5. Why does the paper characterize static features as weak when appendix tables show some static features with strong AUROC in certain layers and models?

6. Are entropy traces computed using the final model normalization before unembedding intermediate hidden states? If not, how sensitive are the results to this logit-lens implementation choice?

Suggestions for improvement

Essential changes:

• Reframe the contribution as an exploratory internal-signal analysis unless additional evidence supports robust jailbreak detection.
• Evaluate on aligned instruction-tuned models and label prompts by actual jailbreak success or refusal failure, not only by harmful-prompt dataset membership.
• Add topic-, length-, and style-matched benign controls, including benign prompts about sensitive topics.
• Use a clean validation/test protocol for selecting direction, layer, and feature, then report held-out performance.
• Add strong baselines: perplexity, prompt classifiers, hidden-state probes, refusal-direction baselines, output-distribution methods, and simple lexical/length controls.
• Report confidence intervals, multiple seeds, exact dataset sizes, and thresholded metrics such as TPR at fixed low FPR.
• Clarify that logit-lens predictive entropy is not necessarily epistemic uncertainty.

Optional improvements:

• Compare vanilla logit lens with tuned lens or normalization-aware projections.
• Analyze by harm category and prompt template.
• Test adaptive attacks that optimize prompts to evade entropy-trend detection.
• Study whether entropy dynamics predict generation-time refusal or unsafe completion.
• Add calibration or abstention experiments if the paper wants to make a stronger EIML claim.

Presentation improvements:

• Move the JailbreakBench benign discussion into the main evaluation rather than treating it as secondary.
• Tone down claims about “harmful intent,” “structured semantic representations,” and a universal intermediate-layer peak.
• Clearly separate empirical observations from mechanistic interpretations.

Overall assessment

The paper has an interesting and interpretable idea: intermediate-layer entropy trajectories may contain useful safety-relevant signal. The main reason for my rating is that the current experiments suggest a real phenomenon worth investigating.

The main reason the rating is not higher is that the evidence does not yet establish robust jailbreak detection or epistemic uncertainty modeling. The method appears highly sensitive to prompt distribution, hard negatives collapse performance, the layer/static-feature claims are overstated, and key baselines and reproducibility details are missing.

The main reason the rating is not lower is that the paper is readable, the method is clearly defined at a high level, the authors test multiple model families, and the hard-negative failure is at least reported rather than hidden. For a workshop, this could become a useful poster after substantial revision, but in its current form I would lean weak reject rather than acceptance.

Rating: 4: Borderline reject

Confidence: 5: You are absolutely certain about your assessment. You are very familiar with therelated work and checked the math/other details carefully.

---

Interesting contribution on entropy-based jailbreak detection, with some presentational concerns

Official Reviewby Reviewer TooF08 May 2026, 11:04 (modified: 18 May 2026, 14:09)Program Chairs, Area Chairs, Reviewers

Summary:

The paper analyzes token-level predictive entropies in large language models (LLMs) to identify discriminative signals for detecting malicious prompts. The central argument is that dynamical features, specifically how entropy evolves across token positions, and corresponding rank-based trend scores carry information that distinguishes benign from harmful prompts This is opposed to static prompt-level summary statistics, which show much poorer discriminative ability. The authors show that this dynamical signal is consistent across three architectures (Llama-3.1-8B, Qwen3-8B, Gemma-7b) and that it is most pronounced in intermediate layers of the network, before degrading at the output head. The work is framed around three research questions: (Q1) whether dynamic trajectory features outperform static aggregates, (Q2) at which depth the signal peaks, and (Q3) whether it generalizes across architectures and benchmarks.

Strengths:

• The proposed analysis is intuitive and conceptually simple, yet yields good discriminative ability for detecting harmful prompts without any additional training or threshold tuning.
• The rank-based dynamical measures display remarkably stable performance across architectures, with very small cross-model standard deviations. This suggests that the signal reflects a transferable structural property of how transformer LLMs process adversarial prompts and is not architecture-specific.
• The layer-wise analysis convincingly demonstrates that the most relevant information is concentrated in intermediate layers, providing a useful starting point for future internal-signal-based detection methods and for interpretability work more broadly.

Weaknesses:

Overall, I enjoyed the paper and believe it constitutes a good contribution to the workshop. The weaknesses below are relatively minor, and several point toward natural directions for follow-up work rather than fundamental issues.

• Undefined notation. In Eq. (2), the symbol  is not defined and the space over which the sum ranges is not made explicit (presumably the vocabulary). This should be clarified.
• Figure 2 readability and framing. The fonts in Figure 2 are too small to read comfortably and the statistical tests (MWU, KS) are reported without being introduced in the main text. More importantly, the framing in lines 265–267 around the “peak” at 69% depth is, in my opinion, slightly misleading. The relevant property of these distributions is not that they peak, but that the safe and harmful distributions become well separated at intermediate layers. In fact, both could in principle have a sharp peak at the same value while still being indistinguishable, leading to poor AUROC. I would recommend rewording this passage in terms of distributional separation or distance, rather than peakedness.
• Table 3 vs. Tables 2 and 4. Table 3 reports static-vs-dynamic results on a single dataset pair (UltraChat × AdvBench), while Tables 2 and 4 are averaged over the six primary pairs. Reporting only one pair here leaves the impression that the chosen pair could be cherry-picked. I would suggest either presenting averaged results across all primary pairs (matching Tables 2/4) or moving per-pair breakdowns into the appendix for full transparency.
• Choice and bolding of the focal layer. The 69% layer is presented as the focal layer across all three models, but the layer-wise tables in the appendix do not support this. For Llama (Table 8), L26 (~81% depth) appears at least on par with L22. The current bolding convention emphasizes the focal layer rather than the best-performing one, which I find unhelpful. I would recommend (i) bolding the actual best value per row, and (ii) softening the framing around a single 69% layer. The data look more consistent with a range of well-performing intermediate layers (roughly 60–80% depth), which would be a more defensible and arguably more interesting claim.
• Across-layer features as a natural extension. Following the previous point, an obvious next step is to combine information across multiple intermediate layers (e.g., averaging trend scores or using them jointly as a multivariate feature). Given that several layers carry comparable signal, this is likely to improve discriminative performance and would fit naturally within the current framework.

Limitations:

As the authors note, the method requires access to intermediate activations and is therefore not directly applicable to black-box API models.

• The use of an adversarially-crafted “benign” control set (JailbreakBench benign) shows that the method captures structural prompt composition rather than semantic harmfulness, which is in principle expected. However, it does highlight a practical limitation: it is unclear how one would deploy this approach as a real detector without first specifying a reference safe set. In other words, the method is currently a comparative tool rather than an absolute detector of malicious content, which could be limiting in applications

Rating: 7: Accept

Confidence: 3: You are fairly confident in your assessment. It is possible that you did notunderstand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/otherdetails were not carefully checked.

---

Okay for a workshop paper

Official Reviewby Reviewer rnwF29 Apr 2026, 17:08 (modified: 18 May 2026, 14:09)Program Chairs, Area Chairs, Reviewers, AuthorsRevisions

Summary:

The paper introduces a method to detect prompts trying to circumvent safety mechanisms in LLMs and evaluates it. It shows that the progression of entropy through the layers is more informative than single entropy scores.

Strengths:

The paper introduces a novel method for an important problem and evaluates it on three different models as well as different datasets. It is generally well written.

Weaknesses:

The paper mentions alternative approaches from the literature but does not compare against them. It arguably overclaims saying that its preliminary analyses ‘highlight that exploring intermediate-layer uncertainty dynamics provides a clear and insightful way to understand jailbreak behavior’.

Limitations:

There is only one sentence about actual limitations. There is nothing on potential negative societal impact.

Rating: 6: Weak Accept

Confidence: 2: You are willing to defend your assessment, but it is quite likely that you didnot understand the central parts of the submission or that you are unfamiliar with some pieces of related work.