One year of AI pentesting on your own hardware 

Over the past year, Warpnet built an autonomous pentest solution that we now use daily in our projects: PenLocal.

Can an LLM do what an ethical hacker does, within the same frameworks and using the same evidence? That research began last year with our security specialists. PenLocal has now started the steps of a pentest itself, performing it in an isolated environment and verifying its own findings. Our pentesters are in control. They take the lead, approve risky steps and do what humans are still better at: determining what really matters to your organization.

In September, we wanted to know if that would also work with a model we were running ourselves. We tested four models with the same four applications. A local model on our own graphics cards came up with three concatenated runs at 87.4%, the same score as the most expensive cloud model in one run, for $7.74 instead of $132.19.

Known vulnerabilities are exploited within hours

Attackers nowadays turn a published vulnerability into a working attack within hours. AI accelerates and amplifies the abuse of these threats. On September 24, 2026, NCTV, NCSC, AIVD, MIVD, OM, CIO Rijk, and the police warned in a joint statement that vulnerabilities are being detected and exploited by AI ever more quickly. According to them, attackers do not need a frontier model for that. Existing, easily accessible models suffice.

Yet many organizations still test their security only once a year. That gives one day of insight and 364 days in which new code remains untested.

Testing with source code becomes the standard

In a manual pentest, reading the source code is a cost factor. Each additional document and each additional line of source code costs a pentester extra hours, which are reflected in the quote. For an AI agent, that is different. Reading is the cheapest thing an LLM does, and everything it reads can be used to attack more specifically.

In the final of DARPA’s AI Cyber Challenge in August 2025, AI systems scanned over 54 million lines of code. They found 54 of the 63 built-in vulnerabilities and 18 real ones, For an average of about $152 per task. Therefore, PenLocal tests the standard with the source code included, if you make it available. In the same time frame, we will thus examine a larger part of your application and try to detect more types of attacks on each component.

The hourly price also varies. Researchers from Stanford, among others, unleashed an AI agent and ten professional pentesters on a university network with about 8,000 hosts. The agent finished second out of eleven participants and cost in some configurations less than a third of the hourly rate of a pentester.

An LLM is just the brain

An LLM can actually be surprisingly little on its own. It reads and writes text. It does not execute commands; in a long conversation it forgets what it saw earlier and cannot check whether its own answer is correct. To perform a pentest with it, you need a system around the model that determines what the model sees, what it can execute, and what happens to its answer. That system is called the harness.
 
Finding vulnerabilities is largely a search process. A powerful model still misses a flaw if no one directs it in the right direction, if previous evidence is lost, or if no one verifies the findings. Moreover, an LLM is non-deterministic: the same task yields a different outcome in every run, so what one run misses, the other sometimes finds. Therefore, our specialists did not spend most of last year working on the LLM, but on the harness of PenLocal.

The harness of PenLocal has five parts:

1. Planner: transforms a proposal for the scope into a threat model and divides it into tasks per component and per attack class. This way, you know exactly what we have tested, and what we haven’t.
2. SandboxEach task runs in its own, secure environment, separate from the rest of our platform. The only network connection to your systems runs through a single port. That forces the scope and a list of prohibited activities and first requires a pentester to take those steps. PenLocal does not have another route to your systems.
3. Auditlog: records every command with the full output. Every finding can be traced back to what actually happened.
4. Verifier: a different model than the one that made the finding tests every finding against the audit log. A vulnerability that the model has created does not pull your report.
5. Evaluator: Filters out duplicate findings, weighs the severity and looks for chains: loose weak spots that together pose a major risk.

Models and harnesses are separate with us. This allows us to connect each model to PenLocal: a cloud-based frontier model or a smaller model on our own graphical cards. We choose the model that suits your requirements for price, speed, and confidentiality per assignment.

Thirty years of harness testing experience

Pairing a model with a terminal nowadays takes an afternoon. The work is elsewhere: in the thousands of small choices in the harness. Which attacks do you try where? What counts as evidence? What step never happens without the agreement of a human? When do individual small findings together become a critical risk? Those decisions do not come from a model. They come from thirty years of experience.

Warpnet is an independent Dutch security company and has been doing this work since 1996. Our team of ethical hackers discovered tens of thousands of vulnerabilities during those years. This experience is reflected in the way the agents break down and address an application and in the way the verifier assesses findings.

An LLM is weakest in two phases, and we deliberately do not leave that to the machine: the consultation with you and determining the threat model. It finds technically correct but innocent vulnerabilities just as easily as vulnerabilities that really matter; it doesn’t know which of the two will affect your organization. In The Stanford study The agent also produced more false positives than the human testers. Therefore, our pentesters assess what appears in your report.

Sensitive data remains in Groningen

A pentest inherently contains the most sensitive material an organization possesses: source code, test accounts, internal systems, and the vulnerabilities that lie within them. Sending that material to an external AI vendor is not an option for many companies, and it’s not an option for us either (We already wrote an article about this).  

That’s why we invested last year in our own AI server, with plenty of working memory and NVIDIA RTX PRO 6000 Blackwell cards, each with 96 GB of VRAM. The server is located in our own part of a data center in Groningen.

We run language models on that server in-house. All AI applications from Warpnet use this local AI: the autonomous pentests of PenLocal, the code reviews, the control of findings, and help with writing reports. If you want us to deploy a frontier model in the cloud for your assignment, that’s still possible. But by default, everything runs locally.

We force data retention in four ways technically:

• No direct line to a supplier. The pentester's workplace always talks to our own platform, never directly with a vendor.
• A new sandbox per task. Each task runs in a protected sandbox that only the systems in scope can access. After the task is completed, we destroy that sandbox.
• An emergency brake. If an agent touches something he is not supposed to touch, the test stops immediately.
• A complete audit log. Every action of the AI can be read.

Where the data is to be sent, we will determine together with you the exact destination for each assignment. We will then choose the appropriate model for that purpose.

Local models versus frontier models

The large cloud-based frontier models are impressive, and the assumption is that only they are good enough for serious pentesting work. We wanted to measure that. We deployed four models against four realistic pentesting targets from Warpnet: applications with vulnerabilities from the real world.

Our pentesters previously determined the answer key based on the source code, with 35 vulnerabilities, weighted by severity. A critical vulnerability counts five times as much as carelessly set file permissions in this. Each model received the same task, the same tools and the same harness. The local models ran three times per application on our own server in Groningen, the cloud models once.

---

---

The local qwen3.8 came with three combined runs of 87.4%, the same score as claude-opus-5 in one run. The twelve runs of qwen3.8 cost a total of $7.74. The four runs of claude-opus-5 cost $132.19, more than seventeen times as much. The older qwen3.6 came together for 81.6%.

Three runs equal more than one

The outcome of an LLM varies per run, and that difference is significant. Ten vulnerabilities were found in only one of the three runs. The most striking example: qwen3.6 found 16% of the vulnerabilities in one run and 95% in another run.

Anyone who runs each model once counts such vulnerabilities as missed. A local run costs us about $0.65. This allows us to run each test multiple times and combine the findings.

What the figures do not say

The comparison is not entirely fair. The local models received three attempts and the cloud models one. In a single run, claude-opus-5 was clearly better than qwen3.8, with 87.4% against an average of 71.3%. What three combined runs of claude-opus-5 had yielded, we did not measure.

The benchmark shows that a local model with repetition at the same level corresponds to one run of claude opus 5, for a seventeenth at no cost and without data leaving the server. The sample size is small: four applications and 35 vulnerabilities, with a response key from our own pentesters. The benchmark measures the detection of known vulnerabilities and not the phases that we leave to people.

What follows here

Currently, only our own security team is working with PenLocal. In the coming months, we will increase the proportion of vulnerabilities found by PenLocal using both frontier and local models. You can read about how this will happen here on the blog.

Then it's your turn. We want to open PenLocal up to customers soon:

• Test it yourself. You create tests, start them, and follow them yourself.
• With our pentesters on board. They assess findings and make the phases where human judgment makes the difference.
• On your own infrastructure. In the long run, you will also run PenLocal entirely in-house, with your own models.

One pentest per year was always a budget choice, not a security choice. With AI on our own hardware, that choice is no longer necessary.

Place PenLocal next to your latest pentest

Have we recently tested an application for you? Then we run PenLocal on the same application and compare the two reports side by side. You can then see what the agent found extra, what he missed, and what our pentesters filtered out of it.

Contact us and mention PenLocal; we will then plan the test with you. Not yet a customer of Warpnet? Fill in the form below and we will discuss whether PenLocal is suitable for your organization.

Sources consulted on October 9, 2026:

• NCTV, AI accelerates and increases the threat: action is now necessary, September 24, 2026
• Lin et al., Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing, arXiv, December 2025
• DARPA, AI Cyber Challenge marks pivotal inflection point for cyber defense, August 8, 2025