The line between assisted and autonomous
Most platforms described as autonomous are assisted. The distinction is not how much AI is involved but where the decision sits. If a human reviews and approves each action, the platform is an assistant regardless of how sophisticated its analysis is. If the platform decides and acts within a boundary, and a human reviews afterwards, it is autonomous.
This matters because the two carry different risk. An assistant that reaches a wrong conclusion produces a bad recommendation, which a competent analyst catches. An autonomous platform that reaches a wrong conclusion produces a wrong action, at machine speed, possibly across many hosts before anyone is awake.
So the useful evaluation question is not whether the platform is intelligent. It is whether the platform's architecture assumes it will sometimes be wrong, and what it does about that assumption.
What autonomous has to demonstrate
Speed is the claim most often offered and the least informative. A rules engine is also fast. The bar for autonomy is a complete loop, demonstrated end to end under production-like conditions: the platform decides, acts within an explicit envelope, records evidence that survives independent scrutiny, and changes its behaviour when corrected.
Each of those four is separately testable, and a platform can be strong on one and absent on another. Most weak platforms in this category are strong on deciding and weak on the other three.
- Decides, meaning it reaches a conclusion about a situation it was not specifically configured for.
- Acts within an envelope, meaning something other than the deciding component enforces what it is permitted to do.
- Records evidence, meaning the decision can be reconstructed months later by someone who was not there.
- Learns from correction, meaning a human marking a conclusion wrong changes future behaviour without a vendor release.
Architectural tests
Architecture is where the difference between platforms is most durable, because it is the hardest thing to change later. Four tests are worth running.
Agent specialisation. Ask what the agents are and what each one is responsible for. Genuine specialisation shows as division of labour, with investigation, containment decisions, evidence assembly and stakeholder communication handled by different components with different objectives. If the agents are one model prompted several ways, they share a single failure mode and the architecture provides no redundancy of judgement.
Model provenance. A platform built on a general-purpose commercial model API inherits that provider's terms, pricing, latency, rate limits and data handling. A cybersecurity-specific model is trained on security telemetry and can usually run inside your boundary. The question is not which is better in the abstract but whether the vendor can answer it precisely, because a vendor who treats the model as an implementation detail has not thought about the risk it carries.
Separation of decision and permission. This is the architectural test that matters most for autonomy. If the component that decides an action is also the component that permits it, there is no check at all. Look for a distinct governance layer that validates each action against policy before execution and is capable of refusing.
Deployment topology. For regulated organisations this often decides the shortlist. Establish whether the platform supports in-boundary, on-premises or hybrid deployment, and whether autonomous reasoning still functions in that mode or quietly degrades to something simpler.
Operational tests
Operational testing is where vendor demonstrations are least useful, because the scenario has been rehearsed. Everything below should run on your telemetry, with your analysts, against incidents the vendor did not choose.
Novel incident handling. Take an incident from your own history that does not resemble a standard playbook and ask the platform to work it. Watch the intermediate reasoning rather than the conclusion. A platform reasoning genuinely will form hypotheses, discard some, and be able to say what it does not know. A platform pattern-matching will produce a confident narrative of uniform shape regardless of the input.
Time to conclusion, not time to alert. Alerting fast is a solved problem and not what you are buying. The measure that matters is how long until a analyst has something they will act on without redoing the work.
Analyst experience. Have analysts drive during the evaluation, not the buying team, and count the surfaces they visit to close one incident. Watch specifically for re-verification: if analysts habitually redo the platform's investigation before acting, the platform has added a step, and any time-saved figure on the dashboard is fictional.
Behaviour under load. Ask what happens during a genuine surge, when incident volume is several times normal. Platforms degrade differently, and queueing, throttling, shedding and simply billing more are very different operational outcomes.
Governance tests
Governance is the part of the evaluation most often deferred to legal and most often decisive. Autonomy without governance is not a more advanced product; it is an unbounded one.
Evidence chain. Ask to see the complete record for one automated action: the telemetry that triggered it, the reasoning that justified it, the policy that permitted it, the action taken and the outcome. Then ask how long it is retained and how it exports. A log entry stating that a host was contained is not evidence, because it does not explain why containment was correct given what was known at the time.
Policy-bounded autonomy with graduated approval. Disabling an identity, isolating a production server, blocking a network segment and quarantining a mailbox are not equivalent risks. A single autonomy toggle covering all of them means the boundary has not been thought through. Look for the ability to set different thresholds for different action classes, and for an explicit approval path on high-impact actions.
Reversibility. For each class of autonomous action, ask what undoing it looks like and how long it takes. Some actions are cheap to reverse and some are not, and an autonomy envelope should reflect that asymmetry rather than treating all actions as equally safe.
Learning loop ownership. If correcting the platform requires a vendor release, it is not a learning loop from your side, it is a support ticket. Ask how quickly a correction takes effect, and what its blast radius is: whether marking one conclusion wrong changes behaviour only for you or for other customers as well.
Questions worth asking that vendors rarely volunteer
The following tend to produce the most informative answers, precisely because they are not on the standard demonstration path.
- Describe the worst incorrect action the platform has taken at a customer, and what changed afterwards.
- What happens if your upstream model provider deprecates the model you depend on?
- Show me an incident where the platform concluded it did not have enough information to act.
- Which decisions does the platform deliberately refuse to make autonomously, and who decided that list?
- If we disable the autonomous actions entirely, what value remains?
- What does an auditor typically ask for, and what do you hand them?
How this differs from an AI SOC evaluation
The two evaluations overlap substantially and are often run together, but their centre of gravity differs. An AI SOC evaluation concentrates on the operational surface: detection quality, investigation, analyst workflow and the economics of running the SOC day to day.
An autonomous cyber defence evaluation extends into architecture and governance across the wider security programme, because the thing being assessed is not only whether the platform works but whether the organisation can defend what it did. If you are running one evaluation, run the AI SOC questions on operations and the questions here on architecture and governance, and score them separately rather than averaging them into a single number.
Scoring and common traps
Weight the categories according to your own constraints rather than to a generic template. An organisation that cannot grant meaningful autonomy in year one should weight governance and evidence heavily and treat response speed as secondary, because the speed is not available to them yet. An organisation whose primary problem is alert volume should weight reasoning quality and analyst experience above architectural elegance.
Three traps recur. The first is scoring vendor answers rather than observed behaviour; written answers are a record of what was promised, not evidence. The second is evaluating against rehearsed scenarios, which measures preparation rather than capability. The third is accepting a platform that scores evenly well across everything, which in a category this young usually indicates the evaluation was not adversarial enough rather than that the product is uniformly excellent.
Where Spharaka fits these tests
Spharaka Sphere is composed to make the architectural distinctions above visible rather than asserted. AuraXP is the agentic engine, a fabric of more than forty specialised agents rather than a single model prompted several ways. SAGE is a cybersecurity-specific model rather than a general language model reached through an API. AirWatch is a separate governance layer that validates each action against policy before it executes, records an evidence chain for what was done and why, and lets administrators review or roll back what was done.
That separation is the point: the component that decides and the component that permits are not the same component. Deployment supports cloud, on-premises and hybrid topologies, so organisations with data localisation obligations can keep security telemetry inside the jurisdiction their regulator requires. We would rather these tests were run against us and against every alternative than accepted on description.
Frequently asked questions
How is this different from an AI SOC evaluation?
AI SOC is the operational surface: detection, investigation, analyst workflow and the economics of daily operations. Autonomous cyber defence is the broader category covering reasoning, action and governance across the security programme. The evaluations overlap, and this one goes considerably further into architecture and governance. Score them separately rather than averaging them.
What is the single clearest test of genuine autonomy?
Whether the component that decides an action is distinct from the component that permits it. If they are the same, there is no check on a wrong decision, and the platform is autonomous only in the sense that nothing is stopping it.
Should we start with full autonomy or with approval workflows?
Almost every organisation should start with approval on high-impact actions and widen the envelope as evidence accumulates. The important thing is that widening is a decision you make deliberately, on the basis of the evidence chain the platform produced, rather than a setting that was permissive by default.
How do we test reasoning rather than pattern matching?
Give the platform an incident from your own environment that does not fit any playbook, and inspect the intermediate steps rather than the conclusion. Genuine reasoning forms hypotheses, discards some, and can state what remains uncertain. Pattern matching produces a confident narrative with the same shape for every incident.
What should the evidence chain contain?
Enough to reconstruct the decision without the people who were present: the telemetry that triggered it, the reasoning that justified it, the policy that permitted it, the action taken and its outcome. It should be retained for the length of your audit obligation and be exportable.
How long should this evaluation take?
Longer than a functional evaluation of a conventional tool, because the architecture and governance questions involve people beyond the SOC. Four to eight weeks of technical testing is typical, with the governance review running in parallel rather than afterwards.
Who should own this evaluation internally?
The SOC leads on operational testing, but the CISO office has to own the autonomy envelope, and legal, privacy or audit should be involved from the start rather than consulted at contract stage. The scope of autonomy you grant is a risk decision, not a technical configuration.
Is a platform still worth buying if we never enable autonomous action?
It can be, but ask the vendor directly what value remains with autonomy disabled. The answer tells you how much of the platform is reasoning and how much is execution, which is useful whether or not you intend to turn the actions on.