Sifat Hossain
← all posts

LLM-ProS: Benchmarking GPT-4o, Mistral, Llama, and OpenAI o1 on ICPC World Finals Problems

Can large language models solve ICPC World Finals problems?

Competitive programming has quietly become one of the hardest, most honest benchmarks for reasoning-capable large language models. Unlike leetcode-style tasks, ICPC World Finals problems demand multi-step algorithmic reasoning, exact I/O adherence, and tight resource budgets. If a model gets an edge case wrong, the judge simply says Wrong Answer — there is nowhere for a plausible-looking but incorrect solution to hide.

Our paper LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving (Hossain et al., LLM4Code workshop at ICSE 2025) is, to our knowledge, the first systematic evaluation of frontier LLMs on the full ICPC World Finals corpus. This post summarizes the motivation, the setup, and the finding that most surprised us: reasoning models like OpenAI o1-mini and o1-preview dominate general-purpose models by a wide, consistent margin — and the why is more interesting than the how much.

Why ICPC World Finals as an LLM benchmark?

Most published LLM code benchmarks (HumanEval, MBPP, APPS, SWE-bench) suffer from one or more of three problems:

  1. Training-data contamination. These datasets are years old, widely mirrored, and almost certainly memorized by frontier models. IBM's own researchers have flagged that Python SWE-bench scores of ~80% are likely contaminated.
  2. Task difficulty ceiling. HumanEval and MBPP were designed as undergraduate coursework-style tasks. Modern models saturate them.
  3. Weak evaluation signal. Many benchmarks use unit tests generated by the same LLM being evaluated, or use LLM-as-judge scoring that rewards plausible-looking patches rather than correct ones.

The ICPC World Finals avoids all three:

  • Each year's problems are curated by expert setters to be hard even for elite human teams; contamination is limited because full editorial solutions are not typically indexed in the way LeetCode discussion threads are.
  • Problems require deep algorithmic reasoning: DP over trees, computational geometry, flow networks, number-theoretic constructions.
  • Evaluation is executable and adversarial: the judge is the Codeforces Gym archive, which runs your submitted code against the official test suite and returns Accepted, Wrong Answer, Time Limit Exceeded, Compilation Error, or Runtime Error.

The LLM-ProS benchmark: what we built

LLM-ProS is a reproducible evaluation pipeline:

  • Dataset. 166 ICPC World Finals problems spanning 2011-2024, scraped from the Codeforces Gym mirror. We normalize LaTeX and HTML content into a consistent structured form (statement, input spec, output spec, constraints, sample I/O) suitable for automated prompt construction.
  • Models evaluated. GPT-4o, Mistral Large, Llama-3.1-405B, o1-mini, and o1-preview — a mix of dense general-purpose models and OpenAI's o-series reasoning models.
  • Prompting protocol. Each model receives an identical prompt template containing the full problem statement and samples. We report Pass@1 with a single-attempt setting to keep the signal clean.
  • Automated verdict capture. A Selenium- and Playwright-driven submission pipeline files each candidate solution to the Codeforces Gym judge and records the exact verdict, runtime, and memory usage in SQLite for reproducible analysis.

Code and dataset are open-source: github.com/sifat-hossain-niloy/Hybrid-Feedback-Loop---LLM.

The headline finding: o-series reasoning models dominate

Across all 166 problems, the two OpenAI o1 models substantially outperformed GPT-4o, Mistral Large, and Llama-3.1-405B. The gap is not marginal — it is a qualitative shift in which classes of problems become solvable.

More importantly, the pattern of errors changes:

  • General-purpose models (GPT-4o, Mistral, Llama) fail predominantly through plausible-but-wrong algorithms: they generate compilable code that runs on the samples but misses edge cases, off-by-one boundaries, or overflow.
  • o-series models shift the failure mode toward Time Limit Exceeded and correctness-adjacent errors — they usually pick the right algorithm but occasionally miss a constant-factor optimization. When they fail on correctness, it is more often on genuinely difficult problems where the algorithmic insight itself is hard.

The reason is not that o1 is a bigger dense model. It is that o1 is trained to think longer before answering, and competitive programming rewards exactly this: internal chain-of-thought reasoning, verification of intermediate claims, and iterative refinement inside a single response.

What this means for LLM evaluation

Three takeaways for anyone benchmarking LLMs on code:

  1. Reasoning-time compute is the new axis. Comparing raw parameter counts across generations misses the point. o1-mini beats far larger dense models on hard reasoning because of its inference-time thinking budget.
  2. Executable, expert-curated benchmarks matter. LLM-as-judge scoring on contaminated datasets is not measuring what you think it is measuring. Codeforces / ICPC-style verdicts are boring, mechanical, and correct.
  3. Failure taxonomy is more informative than aggregate accuracy. Two models at the same Pass@1 can fail in completely different ways, which matters for how you build agents on top of them.

How to cite LLM-ProS

If this work helps your research on LLM code generation, competitive programming benchmarks, or reasoning-model evaluation, please cite:

@inproceedings{hossain2025llmpros,
  title     = {LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving},
  author    = {Hossain, Md Sifat and Tabassum, Anika and Arefin, Md. Fahim and Zaman, Tarannum Shaila},
  booktitle = {Proceedings of the 1st ACM International Workshop on Large Language Models for Code (LLM4Code), ICSE},
  pages     = {80--87},
  year      = {2025},
  address   = {Ottawa, Canada},
  eprint    = {2502.04355},
  archivePrefix = {arXiv}
}

Read more