Hi there,
First of all, thank you so much for providing such well-designed program understanding tasks! We find them truly valuable for evaluating our models’ reasoning and comprehension capabilities in code-related scenarios.
However, one thing we noticed is that the ground-truth answers are not currently available, which makes it a bit difficult for us to conduct accurate evaluations in our experiments.
We were wondering:
Would you consider releasing the answers?
Having access to them would greatly help us benchmark and analyze performance more effectively.
Thanks again for your amazing work and openness — we’re looking forward to seeing more progress on this project!
Warm regards,
Email: gaohongwan@bytedance.com
Hi there,
First of all, thank you so much for providing such well-designed program understanding tasks! We find them truly valuable for evaluating our models’ reasoning and comprehension capabilities in code-related scenarios.
However, one thing we noticed is that the ground-truth answers are not currently available, which makes it a bit difficult for us to conduct accurate evaluations in our experiments.
We were wondering:
Would you consider releasing the answers?
Having access to them would greatly help us benchmark and analyze performance more effectively.
Thanks again for your amazing work and openness — we’re looking forward to seeing more progress on this project!
Warm regards,
Email: gaohongwan@bytedance.com