Qwen3-30B Reproduction: Troubleshooting & Discussion

Hello everyone,

I'm writing to discuss some challenges and questions I've encountered while trying to reproduce the REAP pruning experiments for Qwen3-30B-A3B, specifically at 25% and 50% compression rates. I truly appreciate the team's outstanding work on this model and am eager to delve deeper into its capabilities. However, I've noticed some discrepancies between my results and those presented in the research paper, and I'm hoping to gain some insights and clarification from the community.

1. EvalPlus: Addressing the Performance Gap and Understanding Thinking Handling

My primary concern revolves around a significant performance gap observed in the EvalPlus benchmark. At a 25% compression rate, I'm seeing a performance difference of approximately 5% compared to the paper's findings. This gap narrows slightly to 3% at a 50% compression rate, but it's still a noteworthy deviation. It’s important to address these gaps to ensure accurate reproduction of results and a thorough understanding of the model's behavior under different compression levels.

To provide context, here's a breakdown of my experimental setup:

• Greedy Decoding: Set to True with a temperature of 0.0 to ensure deterministic output.
• Maximum New Tokens: max_new_tokens is set to 16384 to accommodate lengthy code generation tasks.
• Number of Samples: num_samples is set to 1024 to obtain a robust estimate of the model's performance.
• Seed: A fixed seed of 42 is used for reproducibility.
• Backend: vLLM serve is used as the backend for efficient inference.
• Attention Implementation: attn_implementation is set to "flash_attention_2" to leverage optimized attention mechanisms.
• Dataset: The theblackcat102/evol-codealpaca-v1 dataset is used for evaluation.

One specific area of interest is the handling of Qwen3's "Thinking" mode. Since there isn't a direct parameter to disable this mode, I've kept enable_thinking=True and relied on the evaluation framework's parsing capabilities to manage the output. This reliance on parsing introduces a level of complexity, and it's possible that inconsistencies in how the "Thinking" output is interpreted could contribute to the observed performance differences. Therefore, understanding the handling of the 'Thinking' output is crucial for achieving accurate and reproducible results.

To further investigate this, I have a few key questions:

• How was the Qwen3 Thinking output handled in your experiments? Understanding the precise methods used to process and interpret this output is essential for aligning my approach.
• Could you provide a set of reproducible experimental settings? Sharing specific configurations and hyperparameters would greatly assist in replicating the reported results.

By addressing these questions, we can collectively work towards a clearer understanding of the model's performance and ensure the reproducibility of research findings. This collaborative approach is vital for advancing the field and fostering trust in the results.

2. LiveCodeBench (LCB): Resolving Implementation Issues for Enhanced Comparison

Another challenge I've encountered pertains to the LiveCodeBench (LCB) evaluation. The current evaluation scripts available in the repository don't seem to function seamlessly out-of-the-box. This is primarily due to missing dependencies or configuration issues. These technical hurdles make it difficult to directly compare my results with those obtained using LCB.

LiveCodeBench is a crucial benchmark for evaluating code generation models in a realistic setting, and its accurate implementation is paramount for comprehensive model assessment. The inability to run the LCB scripts directly hinders the evaluation process and limits the ability to draw meaningful comparisons. Addressing these implementation issues is essential for promoting fair and thorough evaluation of Qwen3-30B and other code generation models.

Therefore, I would greatly appreciate it if you could release an updated or runnable version of the LCB code. This would significantly enhance the ability to compare results and contribute to a more comprehensive understanding of the model's strengths and weaknesses. A fully functional LCB implementation would also benefit the broader research community, enabling more consistent and reliable evaluations across different studies.

By providing a readily usable version of the LCB code, we can ensure that researchers have the necessary tools to conduct rigorous evaluations and contribute to the advancement of code generation technology. This collaborative effort will foster a more transparent and reproducible research landscape.

3. Math: Bridging the Gap at 50% Compression Through Hyperparameter Tuning and Calibration

In the domain of mathematical problem-solving, I've successfully reproduced the reported results at a 25% compression rate. However, a 3% gap persists at a 50% compression rate, even when using the same settings. This discrepancy raises questions about potential factors that might influence performance at higher compression levels. Understanding these factors is crucial for optimizing the model's performance and ensuring its effectiveness across a range of compression scenarios. This requires a detailed understanding of the intricacies of hyperparameter tuning and the selection of appropriate calibration data.

This observation suggests that specific hyperparameter adjustments, calibration data choices, or other settings might be necessary to optimize performance at higher compression rates. It's possible that the model's sensitivity to certain parameters increases as the compression level rises, necessitating a more fine-grained approach to configuration. Delving into the specifics of these settings is key to unlocking the model's full potential at various compression levels.

To better understand this behavior, I'm wondering if there are any specific strategies employed for higher compression rates. This might include:

• Hyperparameter Optimization: Are there specific hyperparameters that require careful tuning at higher compression levels?
• Calibration Data Selection: Does the choice of calibration data play a more significant role at higher compression rates?
• Other Settings: Are there any other settings or techniques that are particularly relevant for achieving optimal performance at 50% compression?

By exploring these questions, we can gain valuable insights into the nuances of model optimization at different compression levels. This knowledge is essential for deploying the model effectively in resource-constrained environments and maximizing its performance across a variety of tasks. A deeper understanding of these factors will ultimately lead to more robust and reliable applications of the Qwen3-30B model.

I eagerly anticipate your responses and thank you in advance for your time and expertise. Your insights will be invaluable in my ongoing research and contribute to a deeper understanding of Qwen3-30B.

Thank you again for your outstanding work!

I hope this discussion will foster a collaborative environment where we can collectively address these challenges and advance the field of large language models.

For further reading on large language models and pruning techniques, consider exploring resources like the Hugging Face Blog. They often publish insightful articles and tutorials on these topics.