Abliterated DeepSeek Finds a Bug a Frontier Model Missed
In @cybersec’s VulnHunter eval, DS4.1-1017 turned up one new vulnerability in a production Go repo.
An open-weight model found a vulnerability the frontier baseline did not.
@MiaAI_lab shared VulnHunter results from @cybersec for an abliterated DeepSeek V4.1 Flash 1017. The headline: one new, unique vulnerability that was not discovered by a frontier model.
@cybersec’s verdict, as Mia quotes it: “DS4.1F-1017 is the most thorough and most precise local VulnHunter run so far.”
How the eval works.
@cybersec says every cyber eval has two parts. The “easy” one is the benchmark on capability_matrix.
The second is an operational pass against an oracle repo, pinned to a version with known vulnerabilities. Here that is a Go repo for a production app. @cybersec uses Grok as the frontier baseline, and all open-weight evals are measured against it.
On the trace files, @cybersec writes: “DS’s trace files show it examined every one of these. The pattern isn’t missed coverage. It saw the mechanism and dismissed the impact.”
Sources
- @MiaAI_lab on X · Oct 19, 12:16 PM PT