Introduction
In the complex and continually evolving LLM market, organizations have many choices, including both frontier models hosted by providers such as OpenAI and Anthropic, and open-weight models that can be run locally. Chinese open-weight models are becoming more widely available, easier to run, cheaper and increasingly capable, particularly in cybersecurity use cases. However, F5 Labs testing shows capability does not translate into consistent resistance to attack. Across the models tested, security performance varied significantly by model, version, and month.
These models are becoming of interest to many. Their capabilities in code analysis, vulnerability discovery, exploit development, and autonomous action make them very attractive to offensive and defensive security teams, especially when compared to models with greater safeguards, which can impede legitimate cybersecurity use-cases.
But these very qualities make them also attractive to criminal actors, who want models they can run without any guardrails. These models are also risky to internal use or in legitimate applications, since they frequently show much less resistance to attack themselves. This raises issues that prospective users of these models must find a way to balance.
Open-Weight vs Closed-Weight Models
The differences between open-weight models and proprietary models are important to understand. Proprietary, closed-weight models, such as OpenAI’s GPT-5 series and Anthropic’s Claude Sonnet and Opus models are highly performant, backed as they are with massive compute resources, as well as being more controlled, with system prompts, guard rails, and other protections in an attempt to reduce misuse.
They are hosted on the provider’s servers, which raises the issue of data privacy. Many organizations are concerned about the location of their proprietary data, being uploaded into a black box cloud, outside their control, and the possible use of that data to train models. Cost is also a concern, since the cost of tokens is variable and can change rapidly if the provider decides to change them.
Open-weight models, on the other hand, require that the user provision their own compute resources. This could be within a hyper-scalar cloud, a private cloud, or locally on prem. In public cloud environments, data sovereignty can be a concern. Regulations may prevent public cloud from being an option, depending on industry or the rules of the user’s country.
Running open-weight models locally or in a user-controlled cloud alleviates data privacy and sovereignty concerns but does require provisioning of adequate resources to run the model efficiently.
Open-weight models also allow users to add, tune, and even remove controls to suit their use cases. They can also be additionally trained to perform better in specific use cases. In many cases, quantized models, with much less compute required, can be tuned in these ways to provide equivalent performance to API-only models in specific use cases.
There also exist open-source models, such as Falcon from the Technology Innovation Institute (TII) in Abu Dhabi, UAE, which not only provide weights, but give access to their entire training pipeline design, including training data, allowing users to even further customize and even train their own models.1
The Rewards of Open-Weight Models
Besides the general benefits of open-weight models mentioned above in terms of cost and privacy, several Chinese models have emerging capabilities that make them especially worthy of attention. Some are truly excellent at cybersecurity tasks. They have been proved to be able to find security bugs in even well-audited applications, such as Kimi K3 finding 19 vulnerabilities in Redis2 and allegedly finding RCE in Telegram3. In a recent incident, OpenAI was evaluating a combination of GPT-5.6 Sol and a highly capable pre-release model on offensive cyber capabilities. Despite operating in an isolated environment, the models exploited a zero-day vulnerability, obtained internet access and compromised Hugging Face.4
OpenAI’s models are not open weight. Hugging Face reported that the hosted models it initially used refused legitimate forensic requests, so it instead used GLM 5.2 on its own infrastructure to analyze logs related to the incident.5 This demonstrates how self-hosted models can support sensitive tasks, including forensic analysis, while keeping incident data within an organization-controlled environment.
They can help smaller security teams perform work that previously required larger analyst capacity. They also improve resilience by reducing dependence on a small number of heavily restricted commercial systems.
The Risks of Open Weight Models
Chinese open-weight models are a dual-use, where the same capability that helps defenders can help attackers. Open access means fewer usage controls, fewer monitoring points, and as previously noted, more room to remove safeguards.
Trust is a serious issue, since despite being open-weight, they are not open-source, and these models were created outside of the control of users. In most cases, the training data is not disclosed, there are no guarantees of the integrity of the training pipeline, and there exists a risk that specific fine-tuning has been included for reasons that may be opaque. These concerns exist for API-only US models as well, at least in theory. Country of origin may form part of an organization’s broader risk assessment, particularly where requirements around model provenance, supply-chain assurance, data sovereignty and regulatory compliance apply. Those considerations should be evaluated separately from the model’s measured security resilience.
A separate supply-chain risk applies across the open-weight ecosystem. Many model artifacts have historically been distributed using Pickle-based serialization, which can allow arbitrary code execution when untrusted files are deserialized. Attackers have used malicious model files to conceal executable payloads, meaning repository-level scanning should not be treated as a guarantee of safety. Security teams should verify model provenance, pin trusted versions, scan all artifacts, test models in isolated environments, avoid loading untrusted Pickle files, and prefer data-only formats such as Safetensors where supported. In practice, model artifacts should be governed as software supply chain inputs rather than assumed to be passive data.
By running these models locally, the user assumes the risk of using them and securing them, which can add significant barriers to deploying them. These risks are not insignificant.
CASI Security Scores Evaluation
F5 Labs CASI (Comprehensive AI Security Index) measures how vulnerable a model is to common prompt injection and jailbreak attacks. It evaluates how easily an LLM can be manipulated into producing harmful or policy-violating outputs.
July 2026 CASI results show substantial variation across Chinese open-weight models. Qwen3.5-397B-A17B scored 81.13 and Xiaomi’s MiMo-V2.5 scored 73.80, while Z.ai’s GLM-5.2 scored 46.58. Claude Sonnet 5 led the overall leaderboard with a CASI score of 93.08. The range reinforces that provider, model family and capability are not reliable proxies for resistance to prompt injection and jailbreak attacks.
Other models tested include MiniMax-M2, scoring high at about 80, MoonshotAI Kimi-K2-Instruct which scores similar to the more robust Deepseek models above, and various Qwen models which all score in the 60-70s range. CASI should be interpreted comparatively and alongside the exact model version, assessment date, intended use and compensating controls. It is one measure of adversarial resilience, rather than a standalone determination that any model is secure or insecure.
These results reinforce that capability does not reliably predict resistance to attack. Deployment model is not a security proxy either: hosted and open-weight models must be evaluated independently and protected according to their intended use, access and authority.
Figure 1: Median CASI scores of major Chinese model families, 6 months, forward filled.
When evaluating risks, many factors need to be considered. Where is the workload running? What isolation or access to networks or the internet is allowed? How are actions logged and how are they allowed or denied? Where and in what capacity are humans involved?
Visibility into agentic actions seems to us to be one of the most important places to focus. So far, these models, while very fast and capable of scaling actions massively, are not coming up with novel activities, so in that sense, defense in depth, with a focus on observability and an assume breach posture, seems the most sensible approach to both resisting AI based attacks from outside the organization, as well as to control and monitor AI activities within.
The following table attempts to summarize the pros and cons.
| Evaluation Factor | Closed-Weight Hosted Services (API/Cloud) | Privately Hosted Open-Weight Models |
|---|---|---|
| Effectiveness & Quality |
Pros: Immediate access to bleeding-edge "frontier" capabilities, superior reasoning, and massive context windows without engineering effort Cons: Subject to silent model updates (behaviour drift) and sudden deprecations by the provider. |
Pros: Can be deeply customized via full weight fine-tuning or distillation to specialize in niche domains Cons: Slightly trails the absolute peak logical reasoning of the latest closed-weight frontier models; requires custom MLOps to match latency/throughput optimizations |
| Hosting & Operational Cost |
Pros: Pay-as-you-go per token is highly cost-effective for low-to-medium or unpredictable traffic Zero hardware capex Cons: Becomes highly expensive at extreme, continuous production scale Vulnerable to future API price hikes. |
Pros: Highly cost-effective for massive, continuous, predictable workloads where you can maximize hardware utilization No per-token markup and predictable costs. Cons: High upfront cost (hardware/cloud GPU leases) You pay for idle GPU time during traffic lulls, which can vastly increase the "effective" cost. High engineering salary overhead. |
| Data Privacy & Sovereignty |
Pros: Enterprise agreements (BAAs, zero-retention policies) can secure standard corporate data Cons: Highly sensitive data (healthcare, defence, finance) must still leave your network perimeter, violating strict air-gapped or localized sovereignty mandates |
Pros: Absolute data control. Data and model weights reside strictly inside your private network/cloud Zero external exposure Meets strict GDPR, HIPAA, and national sovereignty laws Cons: You shoulder 100% of the burden for securing the physical/virtual servers where the data is processed |
| Security & Abuse Resistance |
Pros: Turnkey, constantly updated, multi-layered security and automated safety filters managed entirely by the provider Cons: Highly vulnerable to "jailbreaks" and prompt injection attacks that target the universal provider APIs. System outages disrupt your entire service |
Pros: You control the exact security guardrails and can implement highly customized network isolation, firewalls, and hard boundaries Complete immunity to third-party outages. Cons: The "Do-It-Yourself" tax. You are fully responsible for server security, patching, log auditing, and preventing model theft or intrusion |
| Vendor Lock-in & Customization |
Pros: Zero-setup maintenance; updates are handled automatically Cons: Deep vendor lock-in. Switching providers requires completely rewriting system prompts, custom integrations, and application logic. |
Pros: Zero lock-in. You own a permanent copy of the runnable model weights Full flexibility to modify model architecture and runtimes Cons: High operational complexity. Requires a highly skilled, specialized team of MLOps/platform engineers to maintain |
Table 1: Pros and Cons of Open-Weight vs Closed-Weight models across several categories.
Agentic Resistance Score (ARS)
F5 Lab's Agentic Resistance Score (ARS) provides a complementary view by measuring how well a model holds up in real-world, multi-step and autonomous agent scenarios, over time. In July testing, three Qwen3.5 models ranked in the ARS top 10: Qwen3.5-4B scored 91.04, Qwen3.5-122B-A10B scored 89.81 and Qwen3.5-27B scored 88.23. However, their CASI scores were 58, 31 and 53 respectively, while average performance ranged from 16.0% to 29.3%.
This divergence illustrates why no single score captures a model’s overall security posture. A model can demonstrate strong resistance during complex, sustained agentic attacks while remaining more vulnerable to common prompt injection and jailbreak techniques, or delivering lower general task performance. Organizations should therefore evaluate CASI, ARS and performance separately and in the context of the system in which the model will operate.
Security Research and Cybercrime
Open-source models are critical for legitimate cybersecurity research. They can materially improve the speed and depth of investigation, find subtle bugs, and accelerate the speed from vulnerability to proven exploit, to the issuance of a patch.
Likewise, they are incredibly useful for the analysis of logs, for triaging reports, for enabling AI powered defenses, and ideally, for discovering gaps and vulnerabilities before attackers can. Given the speed with which new vulnerabilities are being discovered, and the amount being disclosed every day, organizations must take on at least some responsibility for defending their infrastructures, even as their vendors issue patches at a much faster rate.
For illegitimate use, for cybercrime or espionage, for ransomware, these models are equally useful. Here too, they will increase the speed and scope of attacks and allow small groups of attackers to massively expand their reach.
Given the proven effectiveness of Chinese models in exactly these tasks, for both good or ill, it seems clear that the capabilities are needed by defenders, if they are going to be able to keep up with attackers. This is especially true for smaller organizations who have to be more cost-conscious than larger enterprises, and for those without access to close commercial models with similar capabilities.
Of course, open-weight models are not limited to Chinese sources. Notable models from other regions include Falcon, mentioned above, Meta’s Llama series, Google’s Gemma models, and France’s Mistral. All have different capabilities, sizes available, and hosting requirements. None of them are without security limitations.
Much of the current discussion centers around the performance of specific models, such as Kimi-K3, and their astonishing capabilities around cybersecurity. Both Anthropic5 and Microsoft6 have released position papers on the topic in recent days, with other big players likely to follow. It is likely that debate on these issues will continue in the near future, regardless of any regulatory action that may be taken.
Conclusion
Open models, both from Chinese companies and other sources, offer genuine value for defenders and researchers, but that value comes with serious strategic and operational risk, especially as model capability in bug discovery, exploit reasoning, and autonomous attack support improves. Organizations must include plans for how to properly control and secure such models, as useful as they are, as they deploy them, and understand the risks of using them.
Learn More
Interested in evaluating additional AI models or learning more about how F5 assesses model resilience, security, and deployment risk?
- CASI and ARS Leaderboards
Compare the security resilience of leading AI models against prompt injection, jailbreak and adaptive agentic attacks, with updated results published monthly.
Explore the latest CASI & ARS Leaderboards - F5 AI Red Team demo
See how adversarial testing can uncover model- and application-specific weaknesses and produce actionable findings for security and development teams.
Request an F5 AI Red Team Demo - F5 AI Red Team
Explore how F5 evaluates models, applications and agents using automated testing, current attack intelligence, and customizable adversarial scenarios.
Learn more about F5 AI Red Team - F5 AI Security Platform
Learn how F5 helps organizations evaluate AI risk, define governance and enforce tailored protections across models, applications and agents.
Explore the F5 AI Security Platform
