# ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS

DevFeed: [ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS](<https://devfeed.tech/articles/arkeval-benchmarking-and-evaluating-automated-coderepair-for-arkts-129136.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.08866>)

Author: Bang Xie, Senjian Zhang, Zhiyuan Peng, Wei Chen, Xin Yin, Chenhao Ying, Yuan Luo

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [bugfixes](<https://devfeed.tech/topics/bugfixes.md>), [Claude](<https://devfeed.tech/topics/claude.md>)

Tags: [automated](<https://devfeed.tech/tags/automated.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [claude](<https://devfeed.tech/tags/claude.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [extension](<https://devfeed.tech/tags/extension.md>), [framework](<https://devfeed.tech/tags/framework.md>), [harmonyos](<https://devfeed.tech/tags/harmonyos.md>), [huawei](<https://devfeed.tech/tags/huawei.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mining](<https://devfeed.tech/tags/mining.md>), [mobile](<https://devfeed.tech/tags/mobile.md>), [repair](<https://devfeed.tech/tags/repair.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [software](<https://devfeed.tech/tags/software.md>), [test](<https://devfeed.tech/tags/test.md>), [typescript](<https://devfeed.tech/tags/typescript.md>), [voting](<https://devfeed.tech/tags/voting.md>)

## AI overview

ArkEval provides a benchmark for evaluating automated repair of ArkTS, a statically typed TypeScript extension used in HarmonyOS development. The authors mined issues from an official Huawei repository with over 400 ArkTS applications and, after multi-stage filtering, curated 502 reproducible issues. They used an LLM-based test-generation and voting process involving Claude and other models, and standardized problem statements to support fair evaluation. The paper evaluates four LLMs through a retrieval-augmented repair workflow and reports their capabilities and limitations. The supplied text does not include comparative results or specific repair success rates.

## Source excerpt

arXiv:2602.08866v3 Announce Type: replace Abstract: Large language models have transformed code generation, enabling unprecedented automation in software development. As mobile ecosystems evolve, HarmonyOS has emerged as a critical platform requiring robust development tools. Software development for the HarmonyOS ecosystem relies heavily on ArkTS, a statically typed extension of TypeScript. Despite its growing importance, the ecosystem lacks robust tools for automated code repair, primarily due to the absence of a high-quality benchmark for evaluation. To address this gap, we present ArkEval, a unified framework for ArkTS automated repair workflow evaluation and benchmark construction. It provides the first comprehensive benchmark specifically designed for ArkTS automated program repair. We constructed this benchmark by mining issues from a large-scale official Huawei repository containing over 400 independent ArkTS applications. Through a rigorous multi-stage filtering process, we curated 502 reproducible issues. To ensure testability, we employed a novel LLM-based test generation and voting mechanism involving Claude and other models. Furthermore, we standardized problem statements to facilitate fair evaluation. Finally, we evaluated four state-of-the-art Large Language Models (LLMs) on our benchmark using a retrieval-augmented repair workflow. Our results highlight the current capabilities and limitations of LLMs in repairing ArkTS code, paving the way for future research in this low-resource language domain.