Lost in Translation: Evaluating AI Across Languages

Author: Denis Avetisyan

A new study reveals significant inconsistencies in the safety and performance of large language models when tested beyond English, highlighting critical gaps in current evaluation methods.

Researchers conducted a multi-lingual testing exercise to assess cross-lingual consistency, harmful content detection, and the challenges of culturally-aware AI development.

Despite growing deployment of large language models globally, ensuring consistent safety and reliability across diverse linguistic contexts remains a significant challenge. This is addressed in ‘Improving Methodologies for LLM Evaluations Across Global Languages’, which details a collaborative, multilingual evaluation exercise involving ten languages-spanning both high and low resource settings-to assess model performance across five key harm categories. The study revealed substantial variations in safeguard robustness and evaluator reliability-both between languages and harm types-highlighting the critical need for culturally contextualized evaluation methodologies. How can the research community build upon these insights to develop a shared framework for truly comprehensive and equitable AI safety testing?

The Imperative of Rigorous LLM Assessment

The proliferation of Large Language Models extends far beyond simple text generation, now underpinning critical infrastructure in sectors like healthcare, finance, and legal services. This rapid integration necessitates a paradigm shift towards robust safety assessments; previously acceptable performance metrics are insufficient when these models inform high-stakes decisions. As LanguageModel applications become more pervasive – assisting in medical diagnoses, automating financial transactions, or providing legal counsel – the potential for harm resulting from biased outputs, factual inaccuracies, or malicious exploitation increases exponentially. Consequently, a proactive and comprehensive evaluation framework is no longer merely desirable, but essential to ensure responsible deployment and maintain public trust in these increasingly powerful technologies.

Current methods for assessing Large Language Models frequently struggle to detect nuanced weaknesses and ingrained biases that can manifest as problematic outputs. While benchmark tests often gauge performance on standardized tasks, they frequently fail to probe the complex reasoning and contextual understanding necessary to identify subtle harms-such as the generation of discriminatory content, the amplification of misinformation, or the vulnerability to adversarial prompts. This limitation stems from a reliance on surface-level analysis and a difficulty in replicating the unpredictable nature of real-world interactions. Consequently, a model might achieve high scores on conventional evaluations yet still exhibit dangerous behaviors when deployed in practical applications, highlighting the critical need for more sophisticated and comprehensive testing strategies.

The increasing prevalence of Large Language Models in sensitive applications necessitates a move beyond superficial safety checks towards comprehensive AIModelSafetyEvaluation. Simply identifying obvious flaws is insufficient; a robust evaluation must proactively uncover subtle vulnerabilities and biases that could manifest as harmful outputs in real-world scenarios. This detailed assessment requires a multifaceted approach, encompassing adversarial testing, bias detection across diverse datasets, and the analysis of model behavior under unexpected inputs. Prioritizing such thorough evaluation isn’t merely about mitigating risk; it’s fundamental to fostering public trust and ensuring these powerful technologies are deployed responsibly, preventing potential misuse and maximizing their beneficial impact on society.

A Structured Methodology for LLM Performance Verification

The LLMTestingMethodology is a structured process designed to assess Language Model performance across identified risk areas. This framework employs a tiered approach, beginning with automated tests for common failure modes and progressing to evaluations of model outputs for harmful content, bias, and factual accuracy. Systematic evaluation involves defining specific test cases, establishing clear acceptance criteria, and utilizing both quantitative metrics-such as precision and recall-and qualitative analysis from human reviewers. The methodology aims to provide a comprehensive and repeatable process for identifying vulnerabilities and tracking improvements in Language Model safety and reliability, facilitating responsible development and deployment.

The LLMTestingMethodology employs automated techniques, including HarmfulContentDetection, to identify potentially problematic outputs generated by language models. A key component of this assessment is evaluating the effectiveness of each model’s RefusalMechanism – its ability to decline to respond to prompts designed to elicit harmful content. Quantitative analysis reveals significant variation in refusal rates across different models, ranging from a low of 23% to a high of 73%. This indicates substantial differences in how effectively various language models are configured to avoid generating undesirable or unsafe responses, necessitating thorough comparative testing.

Human evaluation remains a critical component of the LLMTestingMethodology, providing nuanced assessments of model outputs that are difficult for automated systems to replicate. However, to address the limitations of human evaluation in terms of scalability and cost, the methodology increasingly incorporates LLMJudge – utilizing other large language models to assess the responses of the primary model under test. LLMJudge functions by applying predefined criteria and rubrics, allowing for higher throughput and consistency in evaluations, though the results are continually validated against human benchmarks to ensure alignment and minimize potential biases inherent in automated scoring.

Multilingual Testing: A Necessary Expansion of Verification Scope

The MultilingualJointTesting initiative implemented a standardized safety evaluation framework applied to leading artificial intelligence models across ten languages: English, Chinese, Spanish, French, German, Japanese, Russian, Arabic, Hindi, and Telugu. This initiative established a unified methodology for assessing potential risks and vulnerabilities in multilingual AI systems, facilitating comparative analysis of model performance across different linguistic contexts. The testing process involved consistent prompt engineering and evaluation criteria, allowing for quantifiable measurements of safety characteristics such as the potential for harmful content generation and susceptibility to adversarial attacks. The successful implementation of this common approach represents a significant step towards ensuring equitable safety standards for AI technologies used globally.

Data translation accuracy is paramount in multilingual AI safety testing because inaccuracies can introduce unintended biases and misinterpretations that compromise evaluation results. The TranslationProcess involves converting test prompts and expected responses across languages, and any errors during this conversion directly impact the validity of vulnerability assessments, such as those targeting cybersecurity prompt injection. Maintaining high fidelity during translation requires careful consideration of linguistic nuances, cultural context, and potential ambiguities to ensure that the translated content accurately reflects the original intent and does not inadvertently alter the meaning or introduce new vulnerabilities. Rigorous quality control measures, including back-translation and expert review, are essential to mitigate these risks and ensure the reliability of multilingual safety evaluations.

The MultilingualJointTesting initiative prioritizes evaluation in Low Resource Languages (LRLs) to address disparities in AI safety research and deployment. These languages, characterized by limited available data for model training and evaluation, often receive less attention than high-resource languages like English. This focus ensures equitable access to safe and reliable AI technologies for a broader global population, preventing the disproportionate benefit of AI advancements towards users of dominant languages. Specifically, the initiative aims to identify vulnerabilities and biases that may be more pronounced or unique to LRLs due to data scarcity and linguistic differences, ultimately promoting inclusivity in AI development and deployment.

The MultilingualJointTesting initiative’s methodology included evaluations for cybersecurity vulnerabilities, specifically prompt injection attacks, across all tested AI models and languages. Results indicate a success rate of 65-67% in identifying these vulnerabilities, meaning the models were successfully exploited in approximately 65 to 67% of test cases. While demonstrating a degree of robustness, this success rate also highlights a significant area for improvement in model security. Further investigation and mitigation strategies are required to address the remaining 33-35% vulnerability rate and enhance the overall security posture of frontier AI models against prompt injection attacks.

Evaluations conducted as part of the MultilingualJointTesting initiative revealed statistically significant discrepancies between model performance in English and Telugu. Specifically, content generated by tested AI models, and the subsequent responses to prompts, demonstrated variations in quality when assessed in Telugu compared to English. This suggests that while models may achieve comparable success rates in English-language safety testing – around 65-67% for Cybersecurity Prompt Injection vulnerabilities – performance is not consistently maintained across all languages. These differences indicate potential biases in model training data or architectural limitations impacting its ability to reliably process and generate content in LowResourceLanguages like Telugu, necessitating language-specific evaluation and refinement.

The Pursuit of True AI Alignment and Responsible Innovation

AI model safety evaluation transcends simple flaw detection; its core purpose is to ensure AI alignment with complex human intentions and values. This necessitates a shift in focus from merely identifying what an AI does wrong to understanding why it deviates from desired behavior. Researchers are increasingly recognizing that evaluating an AI’s performance requires defining and measuring alignment – the degree to which its goals and actions are consistent with human objectives, even in unforeseen circumstances. This approach demands nuanced methodologies capable of assessing not just demonstrable errors, but also subtle misinterpretations or unintended consequences stemming from the AI’s learning process, ultimately fostering AI systems that are not only technically proficient but also ethically and socially beneficial.

The proactive identification and correction of model bias represents a critical safeguard against potential harms arising from artificial intelligence. Researchers are developing systematic methods to detect these biases – stemming from skewed training data or algorithmic flaws – across a range of AI applications. Implementing robust safety mechanisms, such as adversarial training and differential privacy, then works to mitigate these identified issues. This approach doesn’t simply address errors after they occur; it actively shapes AI development to align with ethical principles and societal values, reducing the risk of discriminatory outcomes or unintended negative consequences and fostering greater public confidence in these increasingly powerful technologies.

A commitment to meticulous AI safety evaluation isn’t simply a technical exercise, but a foundational element in cultivating public confidence and encouraging beneficial advancement. When AI systems consistently demonstrate reliability and alignment with human values through rigorous testing, it establishes a positive feedback loop: increased trust encourages wider adoption, which in turn fuels further innovation. This cycle necessitates a proactive stance, where potential harms are systematically identified and addressed before deployment, moving beyond reactive measures. By prioritizing safety and transparency, developers can unlock the full potential of artificial intelligence while simultaneously safeguarding against unintended consequences and fostering a future where AI serves as a powerful tool for progress, rather than a source of concern.

Recent assessments of AI model safety evaluations revealed a noteworthy inconsistency: roughly one in ten evaluations produced discrepant results. This suggests that current automated processes, while valuable, are not yet entirely reliable and may require human oversight to ensure accuracy. The presence of these inconsistencies doesn’t necessarily indicate systemic failure, but rather underscores the critical need for continuous refinement of evaluation methodologies. Addressing these discrepancies is paramount for building robust and trustworthy AI systems, demanding ongoing research into improved testing protocols and a commitment to minimizing potential errors in automated assessments. Ultimately, bolstering the precision of these evaluations is essential for responsible AI development and deployment.

The development of consistently reliable large language models hinges on the establishment of a standardized testing methodology. Currently, the absence of universal benchmarks hinders objective comparisons between different models and complicates the process of identifying potential risks. A unified approach to LLM testing would not only facilitate more accurate performance evaluations, but also dramatically increase transparency within the AI community. By providing a common framework for assessing capabilities and limitations, researchers and developers can collaboratively address safety concerns and foster responsible innovation. This standardized methodology would enable a clearer understanding of model behavior, allowing for the detection of biases, vulnerabilities, and unintended consequences, ultimately building greater trust in these increasingly powerful technologies.

The pursuit of robust large language model evaluation, as detailed in this study, demands a focus on provable consistency rather than merely functional outputs. The findings regarding cross-lingual inconsistencies underscore the need for metrics that transcend superficial performance. G.H. Hardy aptly stated, “The essence of mathematics is its economy.” This principle applies directly to LLM evaluation; a truly elegant and reliable system will achieve consistent, predictable results across languages-a concise, mathematically sound approach to identifying harmful content and ensuring safety, rather than relying on complex, language-specific heuristics. The paper’s emphasis on cultural context is paramount; a solution proven in one linguistic sphere must be demonstrably valid in others, mirroring the universal truths sought in mathematical proofs.

What’s Next?

The exercise detailed within reveals, predictably, that equivalence across languages remains an illusion. The consistency of a model’s boundaries-its ability to reliably delineate safe from harmful content-proves stubbornly resistant to simple translation. One observes not merely a failure of linguistic transfer, but a fundamental disparity in the application of safety protocols, suggesting these are often surface-level implementations rather than deeply integrated principles. The notion of a universally ‘safe’ LLM, therefore, appears less a matter of engineering and more a question of philosophical coherence.

Future work must move beyond identifying that inconsistencies exist, and focus on formalizing the very concept of ‘harm’ itself. The current reliance on human annotation, while pragmatic, introduces a subjective element that obscures any genuine attempt at objective measurement. A truly elegant solution would involve a mathematically rigorous definition of undesirable outputs, allowing for provable guarantees of model safety – a concept currently residing more in the realm of aspiration than demonstrable reality.

The challenge, ultimately, is not to build models that appear to understand cultural nuances, but to create systems whose internal logic is demonstrably independent of such arbitrary constructs. Until evaluation metrics reflect this aspiration – until they prioritize formal verifiability over superficial performance – the pursuit of genuinely reliable, multilingual LLMs will remain, at best, a well-intentioned approximation.

Original article: https://arxiv.org/pdf/2601.15706.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

The Imperative of Rigorous LLM Assessment

A Structured Methodology for LLM Performance Verification

Multilingual Testing: A Necessary Expansion of Verification Scope

The Pursuit of True AI Alignment and Responsible Innovation

What’s Next?

See also: