Today is an inflection point for cyber security with Anthropic’s announcement of Project Glasswing marking one of the most significant developments in cyber security this year. At the heart of the initiative is Claude Mythos Preview, a frontier AI model that has already identified thousands of previously unknown zero-day vulnerabilities across every major operating system and web browser. The oldest bug it surfaced had been sitting quietly in OpenBSD for 27 years.
The coalition behind the project reads like a who’s who of the technology industry: Apple, Microsoft, Google, AWS, CrowdStrike, Palo Alto Networks, Cisco, Broadcom, NVIDIA, JPMorganChase, and the Linux Foundation. These are organisations that have built entire security empires on proprietary AI and threat intelligence, and they are now publicly acknowledging the need for a collaborative, model driven approach to finding what their own tools have missed.
It is an inflection point for cyber security.
What Project Glasswing Means in Practice
Project Glasswing uses Claude Mythos Preview to systematically hunt for vulnerabilities across critical infrastructure before adversaries can find them. The model operates agentically, reading source code, forming hypotheses about potential flaws, running live tests to confirm or reject those hypotheses, and producing detailed bug reports with proof-of-concept exploits and reproduction steps.
The results so far have been striking. Thousands of zero-day vulnerabilities, many of them critical, have been discovered in software that had already undergone extensive human led security review. That tells us something important: even well-resourced security programmes have blind spots, and AI driven analysis is now capable of seeing what human reviewers have not.
Why This Matters for the Organisations HOOP Cyber Works With
At HOOP Cyber, we work with organisations across sectors to strengthen their cyber security posture, from strategy and governance through to technical assurance and threat intelligence. Project Glasswing reinforces a message we have been delivering to our clients for some time: the threat landscape is shifting faster than traditional security approaches can keep pace with, and AI is now central to both sides of the equation.
For the organisations we support, the implications of Glasswing are threefold. First, the volume and severity of newly disclosed vulnerabilities is about to increase significantly. Patching cycles, vulnerability management programmes, and risk assessment processes will need to adapt accordingly. Second, the sophistication of AI augmented attacks will continue to grow. The same capabilities that allow Mythos Preview to find and fix vulnerabilities at scale could, in the wrong hands, be used to exploit them. Anthropic themselves have been candid about this, warning that frontier AI capabilities are likely to advance substantially in the coming months. Third, and perhaps most critically, this development highlights the importance of foundational cyber hygiene. AI powered vulnerability discovery is only as effective as the visibility and asset management that underpins it.
The Visibility Gap: You Cannot Patch What You Cannot See
This is where Project Glasswing becomes particularly relevant for the organisations HOOP Cyber serves. The enterprises most at risk from what comes next are not necessarily those without mature security operations centres or enterprise patching pipelines. They are the organisations running operational technology networks where firmware has not been updated since the equipment was installed. Clinical environments where a connected infusion pump or imaging system sits outside every mobile device management policy ever written. Industrial floors where the programmable logic controller communicating with a SCADA system was never designed with a security model at all, because when it was built, nobody imagined it would one day be networked.
For those environments, the question was never just whether we can find the vulnerability. It has always been whether the asset register or security architecture even knows the device exists. You cannot patch what you cannot see. You cannot segment what you have not inventoried. You cannot respond to a compromise in an asset that is not visible.
How HOOP Cyber Is Helping Clients Prepare
At HOOP Cyber, we help organisations build the foundations that make initiatives like Glasswing actionable. That means working with our clients to achieve continuous, real-time visibility across IT, OT and IoT environments. It means ensuring asset registers are accurate and current, that network diagrams reflect reality rather than aspiration, and that vulnerability management processes are mature enough to act on the intelligence that AI powered discovery will generate.
Project Glasswing is genuinely impressive, and the early results are real. But AI powered vulnerability discovery at scale only closes the gap if defenders already know where to look. Our role at HOOP Cyber is to make sure they do.
Looking Ahead
The window of advantage that Glasswing provides is measured in months, not years. Anthropic have been transparent about this. As frontier AI capabilities proliferate, the organisations that will be best positioned are those that have already invested in the fundamentals: asset visibility, robust vulnerability management, and a security architecture that is designed for the reality of their environments rather than the idealised version of them.
HOOP Cyber stands ready to help organisations navigate this new chapter. If you would like to discuss what Project Glasswing means for your organisation and how to prepare, get in touch with our team via
Attackers are using AI to craft highly personalised, convincing phishing campaigns at scale. This article examines what has changed, why conventional detection is struggling to keep up, and what a stronger response looks like.
Phishing has always been the path of least resistance for attackers. It requires no sophisticated exploit, no zero-day vulnerability and no costly infrastructure. It requires only that a human being be deceived into taking an action they should not. For decades, security professionals have responded with a combination of technical controls and awareness training, teaching people to spot the telltale signs of a phishing attempt: the odd sender address, the generic greeting, the clumsy grammar, the slightly-off branding.
Those signals are becoming unreliable. The arrival of accessible, powerful artificial intelligence has fundamentally changed what phishing looks like and how it is produced. Attacks that once required significant skill, time and manual effort can now be generated at scale, personalised to individual targets, written in flawless prose and timed with a precision that would previously have been impossible. The assumptions that underpin much of our current defence posture no longer hold.
Understanding what has changed, and what that means for both technical controls and human awareness programmes, is now an urgent priority for every security team.
What Has Actually Changed
To appreciate the scale of the shift, it helps to understand what producing a convincing phishing campaign used to require. A well-crafted spear phishing attack against a senior executive, for example, would have demanded considerable investment: researching the target, understanding their professional relationships, mimicking the writing style of a trusted colleague, getting the cultural and linguistic register exactly right. This kind of attack was effective but time-consuming, limiting how many targets a threat actor could realistically pursue.
Large language models (LLMs) have collapsed that time investment close to zero. An attacker with access to a commercially available AI tool, or one of the growing number of purpose-built malicious variants emerging in underground markets, can now:
Generate highly personalised phishing emails from nothing more than a name, job title and organisation scraped from LinkedIn or a company website.
Produce content in multiple languages, with native-level fluency, removing the linguistic errors that once served as a reliable detection signal.
Replicate the writing style, tone and vocabulary of a specific individual by training on their publicly available communications.
Rapidly iterate across thousands of variations of the same campaign, allowing attackers to evade signature-based detection systems that rely on identifying known patterns.
Generate convincing pretexts grounded in real, current events, pulling in timely context that makes the message appear plausible and relevant.
The practical result is that the volume, quality and personalisation of phishing attacks are all increasing simultaneously. This is not a marginal improvement for attackers. It is a step change.
Why Conventional Defences Are Struggling
Traditional anti-phishing defences operate on a model that AI-generated content is increasingly designed to circumvent. Understanding the specific limitations of each layer helps clarify where the gaps now lie.
Email filtering and signature-based detection
Conventional email security gateways rely heavily on known indicators of compromise: blacklisted domains, recognised malicious URLs, known sender patterns and, increasingly, the linguistic fingerprints of previously identified phishing templates. AI-generated phishing content is novel by design. Because each campaign can be generated fresh, it does not match existing signatures. Polymorphic content that varies automatically across sends is particularly effective at evading filters trained on historical samples.
Security awareness training based on visual cues
The advice most employees have received about identifying phishing centres on observable signals: check the sender address, look for spelling mistakes, hover over links before clicking, be suspicious of urgent requests. These remain valid principles, but they are insufficient when the attack is grammatically perfect, sent from a convincingly spoofed or legitimately compromised account, and references accurate details about the recipient and their organisation. Training that focuses primarily on spotting imperfection will not prepare people for attacks that have no obvious imperfections.
Frequency-based anomaly detection
Some security tools look for anomalous volumes of similar messages as an indicator of a phishing campaign in progress. AI-generated campaigns can be deliberately low-volume and highly targeted, sending small numbers of uniquely crafted messages to specific individuals rather than blasting a single template to thousands of recipients. This targeted approach not only evades volume-based detection but also increases the likelihood of success, since the message is specifically tailored to its recipient.
DMARC, DKIM and SPF
Email authentication protocols remain an important foundational control, but they address sender authentication, not content. A phishing email sent from a legitimately compromised account, or from a domain that closely resembles a trusted one, can pass authentication checks while still being entirely malicious. Authentication controls are necessary but not sufficient on their own.
The Business Email Compromise Dimension
Nowhere is the impact of AI-generated phishing more acutely felt than in business email compromise (BEC). BEC attacks, where an attacker impersonates a trusted internal or external party to authorise fraudulent financial transactions or extract sensitive information, have long been among the most financially damaging forms of cybercrime.
AI amplifies the threat in several ways. Voice cloning technology now allows attackers to supplement email-based deception with convincing audio messages, creating multi-channel attacks that are substantially more persuasive than written communication alone. The combination of a realistic email from a spoofed executive account, followed by a voicemail that sounds genuinely like that executive, represents a level of social engineering sophistication that was effectively inaccessible to most attackers only a few years ago.
AI also enables attackers to conduct far more thorough research on their targets before making contact. Organisational hierarchies, ongoing projects, financial processes, supplier relationships and individual communication styles can all be inferred from public information sources. The resulting attacks are contextually rich in a way that makes them extremely difficult for recipients to identify as fraudulent.
What a Stronger Response Looks Like
None of this means that phishing defence is a lost cause. It means that the approach needs to evolve, across both technical controls and human-centred programmes, to match the changed threat landscape.
AI-powered detection to counter AI-powered attacks
The most immediate technical priority is deploying detection capabilities that can keep pace with AI-generated content. Machine learning models trained specifically to identify AI-generated text, and to detect subtle semantic and behavioural anomalies that escape signature-based filters, are now available and maturing quickly. These tools do not rely on matching known patterns; they assess the statistical properties of content and the behavioural context in which it arrives. Layering AI-driven detection over existing email security infrastructure is increasingly straightforward and represents one of the highest-value investments a security team can make.
Behavioural and contextual signals over content-only analysis
Because content alone is now an unreliable indicator, detection needs to incorporate a wider range of signals. Who is the sender in relation to the recipient? Is this communication pattern consistent with past behaviour? Is the request being made consistent with established processes? Does the timing correlate with other unusual activity in the environment? This kind of contextual, behavioural analysis, particularly when applied at the point of data ingestion and enrichment rather than retrospectively, allows security teams to surface suspicious communications even when the content itself appears entirely legitimate.
Rethinking awareness training for the AI era
Security awareness training needs to shift its emphasis from spotting imperfection to developing scepticism and process discipline. The most resilient defence against AI-generated phishing is not a human who can identify a poorly written email; it is a human who knows never to bypass the established verification process for a financial transaction, regardless of how convincing the request appears. Training programmes that build genuine security culture, reinforce procedural controls and create psychological safety around questioning unusual requests will outperform those focused primarily on identifying attack indicators.
Simulations should also evolve. Running phishing simulations using AI-generated content, rather than the same recognisable templates that employees have seen many times before, provides a more accurate picture of actual susceptibility and a more realistic learning experience.
Zero trust principles applied to high-risk actions
For the categories of action that are most targeted by phishing, particularly financial authorisations, credential changes and sensitive data access, zero trust principles provide an important structural safeguard. Requiring out-of-band verification for significant financial instructions, enforcing multi-party approval for high-value transactions and applying step-up authentication for sensitive system access all reduce the potential impact of a successful phishing attack, even when the initial deception succeeds.
Rapid detection and response capability
Because no combination of preventive controls will eliminate phishing entirely, the speed and effectiveness of detection and response becomes a critical variable. Integrating phishing reporting workflows with security orchestration and response tooling, ensuring that reported emails are triaged and investigated quickly, and that indicators of compromise are actioned across the environment in near real time, all contribute to minimising the dwell time and blast radius of a successful attack.
The Strategic Imperative
AI-generated phishing is not a future threat to be monitored and prepared for at some later date. It is a present and growing reality that is already affecting organisations across every sector and geography. The defenders who are best positioned to manage it are those who recognise that the threat has qualitatively changed, not just grown in volume, and who are updating their defences accordingly.
That means investing in detection capabilities that match the sophistication of the attacks. It means building awareness programmes that develop judgement and culture rather than just pattern recognition. And it means ensuring that the data infrastructure underpinning security operations can provide the contextual, behavioural signals that make the difference between detecting a sophisticated attack early and discovering it far too late.
The attackers have adopted AI. The defence needs to do the same, thoughtfully, systematically and with a clear understanding of where human judgement remains irreplaceable.
HOOP Cyber specialises in specialises in data-centric security operations, helping organisations build the foundations for AI-ready SOC environments through Amazon Security Lake, SIEM modernisation and data normalisation services.Contact us via to book a discovery call today.
How Amazon Security Lake moves beyond being a passive repository to becoming the active data backbone for agentic AI workflows in security operations, enabling autonomous investigation, enrichment and response at scale.
For most of its history, the security data lake has been a place where data goes to be stored. Logs arrive, are normalised and compressed, are indexed and retained, and wait patiently for the moment when an analyst or a scheduled query arrives to retrieve them. The value proposition has been clear and genuine: cheaper storage, longer retention, better cross-source correlation than a traditional SIEM can provide. But in architectural terms, the data lake has largely been a passive participant in security operations. Data flows in; queries flow out; humans do the thinking in between.
That model is changing. The emergence of agentic AI, systems that do not merely respond to queries but autonomously plan, reason, act and adapt across multi-step workflows, is beginning to reframe what a security data lake is for. Amazon Security Lake, built on the Open Cybersecurity Schema Framework (OCSF) and deeply integrated with the broader AWS ecosystem, is particularly well positioned to make this transition.
Understanding what that means in practice, and what it requires to get right, is one of the more important conversations in security architecture today.
What Agentic AI Actually Means in a Security Context
The term agentic AI is used with varying degrees of precision, so it is worth being clear about what it means and, just as importantly, what it does not mean in a security operations context.
A conventional AI model in security is a classifier or a scorer. It receives an input, applies a trained model to it and produces an output: a risk score, a category, a recommendation. It does this one event at a time, and it does only what it is directly asked to do. A human analyst reviews the output and decides what to do next.
An agentic AI system operates differently. Given a goal rather than a single input, it plans a sequence of actions to achieve that goal, executes those actions using available tools and data sources, evaluates the results, adjusts its approach based on what it finds and continues until the goal is achieved or it determines that human escalation is required. It is, in essence, an autonomous investigator that can run in parallel across many threads simultaneously, without waiting for human direction at each step.
In a security operations context, this might look like an agent that is triggered by an initial alert and then autonomously: queries the relevant log data in Amazon Security Lake to gather the full event timeline; enriches the findings with threat intelligence and asset context; correlates the activity with other events across the estate; checks whether similar patterns have been seen before; assesses the severity and likely intent; and either closes the alert with a documented rationale, escalates to a human analyst with a fully assembled evidence package, or initiates a predefined containment action.
The analyst who receives an escalation from an agentic system is not starting an investigation from scratch. They are reviewing the findings of an investigation that has already been substantially completed and deciding whether to act on it. That shift, from initiating to adjudicating, is where the operational leverage of agentic AI lies.
Why the Data Layer Is the Agent Layer
Agentic AI systems are only as capable as the data they can access and reason over. This is not a peripheral consideration; it is the central architectural constraint that determines whether an agentic security operations capability delivers its potential or falls short of it.
An AI agent tasked with investigating a suspicious authentication event needs to be able to query login history across identity systems, network flows, endpoint telemetry and cloud API activity, correlate those events across time, enrich them with user context and threat intelligence, and do all of this rapidly enough to be operationally useful. If the underlying data is fragmented across siloed stores, inconsistently formatted, sparsely enriched or slow to query, the agent cannot perform the investigation effectively. Its conclusions will be incomplete; its false positive rate will be higher than necessary and its value to the security team will be diminished.
Amazon Security Lake addresses this constraint directly. By centralising security data in a single, queryable store, normalising it to the OCSF schema so that events from different sources can be compared and correlated consistently, and making it accessible through standard interfaces including Amazon Athena and Amazon OpenSearch, it provides the data foundation that agentic AI workflows require.
Three properties of Amazon Security Lake are particularly significant for agentic use cases.
Schema consistency through OCSF
OCSF provides a common language for security events. When authentication events, network connections, process executions and file operations all conform to the same schema, an AI agent can reason across them without needing to translate between source-specific formats at query time. This consistency is what makes cross-source correlation tractable for an autonomous system operating at speed. An agent that has to navigate inconsistent field names and varying data structures will make mistakes and take longer; one working with consistently structured data can move with confidence and precision.
Query performance at scale
Agentic workflows are query intensive. An agent investigating a single alert may issue dozens of queries across multiple data sources in the course of a single investigation thread. If each query is slow, the cumulative latency makes real-time agentic response impractical. Amazon Security Lake’s use of Apache Parquet columnar storage, combined with partitioning strategies optimised for security query patterns, ensures that the query performance required to support agentic workflows is achievable without prohibitive cost.
Integration with the AWS AI ecosystem
Amazon Security Lake does not exist in isolation. Its native integration with Amazon Bedrock, AWS’s managed foundation model service, creates a direct pathway from the data layer to the AI reasoning layer. Bedrock agents can be configured to query Security Lake as a tool, pull relevant data as part of an investigation workflow, reason over the results using a foundation model and take further actions based on what they find. The same integration extends to Amazon GuardDuty, Amazon Detective and AWS Security Hub, creating a coherent, AWS-native agentic security architecture that does not require complex custom integration work to stand up.
The Architecture of an Agentic SOC on AWS
Translating the concept of an agentic SOC into a concrete architecture requires thinking about four interconnected layers: the data layer, the enrichment layer, the agent layer and the governance layer. Each one is necessary; none is sufficient alone.
The Data Layer: Amazon Security Lake
The data layer is the foundation. Amazon Security Lake aggregates security data from AWS-native sources including CloudTrail, VPC Flow Logs, Route 53 query logs and AWS Security Hub findings, as well as from third-party sources via the OCSF-compatible subscriber model. Data arrives, is normalised to OCSF, is stored in Parquet format in Amazon S3 and is made queryable via Athena or OpenSearch.
Getting this layer right means investing in comprehensive source coverage, consistent normalisation and enrichment at ingestion. Every gap in source coverage is a potential blind spot for the agents operating above it. Every inconsistency in normalisation is a potential source of error in agent reasoning. The quality of agentic security operations is determined here, at the data layer, before the agents begin their work.
The Enrichment Layer: Context at Ingestion
Enrichment transforms raw normalised events into contextually meaningful data. In an agentic architecture, enrichment at ingestion is substantially more valuable than enrichment at query time, because it means that every event arriving in the data store already carries the context an agent needs to reason about it, without requiring an additional lookup that adds latency and complexity to the investigation workflow.
Useful enrichment for agentic security operations includes threat intelligence tagging against known malicious indicators, asset classification that identifies what role a particular system plays and what data it holds, user behavioural context that establishes normal patterns against which anomalies can be assessed, and MITRE ATT&CK tactic and technique classification that situates individual events within the broader framework of attacker behaviour.
The Agent Layer: Amazon Bedrock Agents
Amazon Bedrock Agents provides the orchestration framework for building agentic workflows on AWS. Agents built on Bedrock can be given access to tools, which in a security operations context typically means API access to query Amazon Security Lake, retrieve threat intelligence, interact with ticketing and case management systems, call AWS Systems Manager for remediation actions and communicate findings to analysts via notification services.
Agent design in a security context requires careful thought about goal specification, tool access scope and decision boundaries. An agent tasked with investigating suspicious network activity needs to know what data sources to query, in what order, and what conditions should trigger escalation versus autonomous closure. These workflows can be built incrementally, starting with fully supervised agents that recommend actions for human approval, and progressively extending autonomy as confidence in the agent’s judgement is established.
The Governance Layer: Human Oversight by Design
The governance layer is not an afterthought. It is the set of controls, boundaries and audit mechanisms that determine what agents can do autonomously, what requires human authorisation and how every agent action is logged and attributable. In a regulatory environment where accountability for security decisions matters, the ability to demonstrate that autonomous actions were taken within defined, approved parameters, and to reconstruct the reasoning behind every agent decision, is not optional.
Amazon CloudTrail provides the audit backbone, logging every API call made by an agent across the AWS environment. Defined approval workflows ensure that high-impact actions, such as isolating a workload or revoking credentials, require explicit human authorisation regardless of the agent’s confidence in its assessment. And regular review of agent decision logs allows security teams to identify where agent judgement is consistently sound and where human oversight remains essential.
What Agentic Security Operations Changes in Practice
The practical impact of a well-implemented agentic SOC architecture on Amazon Security Lake is measurable across several dimensions.
Investigation throughput increases substantially. Where a human analyst can actively investigate one alert at a time, an agentic system can run parallel investigations across hundreds of alerts simultaneously, triage them, close the clear negatives with documented rationale and surface the genuine threats for human review. The volume of alerts that receive thorough investigation, rather than a superficial check driven by time pressure, rises dramatically.
Mean time to detect and mean time to respond both improve, because the investigation that previously began when an analyst picked up an alert has already been substantially completed by the time the alert reaches human review. The analyst is not starting cold; they are reviewing a detailed findings package and making a decision.
Coverage consistency improves. Human analysts, under pressure and working long shifts, inevitably apply varying levels of thoroughness to different alerts. An agentic system applies the same investigative rigour to every alert it handles, regardless of volume, time of day or analyst workload. The quality floor of investigation rises.
And the nature of analyst work changes. The proportion of time spent on mechanical data gathering and repetitive triage falls; the proportion spent on genuine judgement, complex investigation and adversary understanding rises. For many security professionals, this represents a more satisfying and professionally developmental working experience, with positive implications for retention in a market where experienced analysts are consistently in short supply.
The Realistic Starting Point
An agentic SOC built on Amazon Security Lake is not a single deployment event. It is an architectural evolution that can and should be approached incrementally, with each stage delivering operational value before the next is undertaken.
For organisations with Amazon Security Lake already in place, the natural starting point is identifying the investigation workflows that are most repetitive, most time-consuming and most clearly defined. These are the workflows most amenable to initial agentic automation: the processes where the steps are known, the data sources are clear and the decision criteria are explicit. Building an agent for a well-understood workflow, running it in supervised mode alongside human investigators, and validating its outputs against human judgement is how confidence in agentic systems is established.
For organisations that have not yet deployed Amazon Security Lake, the data foundation work and the agentic capability work can be planned together from the outset. Designing the enrichment pipeline, the OCSF normalisation strategy and the source coverage plan with agentic use cases in mind from the beginning avoids the rework that comes from retrofitting an agentic layer onto a data architecture that was not designed to support it.
In either case, the progression from passive data store to active decision engine does not happen by accident. It requires deliberate investment in the data layer, thoughtful agent design and a governance framework that earns organisational trust in autonomous security operations over time.
The Direction of Travel
The security operations centre of the next five years will look substantially different from the one of today. Not because human analysts will have been replaced, but because the proportion of investigative work that requires human initiation and human execution at every step will have fallen significantly. The most capable security teams will be those that have built the data foundations to support autonomous investigation, deployed agentic systems that can operate reliably within defined boundaries, and redirected human expertise towards the judgement calls, adversary understanding and strategic security work that AI cannot replicate.
Amazon Security Lake, as the normalised, enriched, centrally queryable data backbone of an AWS-native security architecture, is the logical starting point for that evolution. The organisations investing in it now, not merely as a data store but as the foundation of an agentic security capability, are building an operational advantage that will compound over time.
The data store is becoming a decision engine. The question for security leaders is not whether to make that transition, but how to make it with the rigour, the governance and the data quality that the opportunity demands.
HOOP Cyber specialises in specialises in data-centric security operations, helping organisations build the foundations for AI-ready SOC environments through Amazon Security Lake, SIEM modernisation and data normalisation services.Contact us via to book a discovery call today.
A clear, accessible introduction to agentic AI for security audiences: what distinguishes an AI agent from a conventional automation or ML tool, what it can do autonomously in a security context, and what the realistic near-term use cases look like.
Every significant technology shift in cybersecurity arrives with its own vocabulary, and that vocabulary tends to travel faster than the understanding it is supposed to convey. Cloud-native, zero trust, AI-powered: each of these terms became ubiquitous before most security professionals had a clear, shared sense of what they meant in practice, what they required to implement well, and where the gap between marketing language and operational reality actually lay.
Agentic AI is at exactly that stage right now. The term is appearing in vendor materials, conference agendas and analyst reports at an accelerating rate. Some of what is being described under that label represents a genuine and significant shift in how security operations can be conducted. Some of it is rebranded automation with a new coat of paint. And the distance between the two is consequential for any security leader trying to make informed decisions about where to invest.
This article is an attempt to cut through the noise. What does agentic AI actually mean, technically and operationally? How is it genuinely different from the automation and machine learning tools that security teams are already using? And, perhaps most importantly, what does it realistically look like when it is applied in a security operations context today, as opposed to in the aspirational future that vendors tend to describe?
Three Things That Are Not Agentic AI
The clearest way to define what agentic AI is may be to start with what it is not. Three categories of existing technology are frequently conflated with it, and the distinctions matter.
Automation and SOAR
Security Orchestration, Automation and Response (SOAR) platforms have been a fixture of mature SOC environments for several years. They are excellent at executing predefined workflows: if this alert type fires, run this playbook, query this source, send this notification, create this ticket. The key word is predefined. A SOAR playbook does what its author wrote it to do, in the order it was written, without deviation. It cannot reason about a situation it has not encountered before, adapt its approach based on intermediate findings, or decide that a different sequence of actions would be more appropriate given what it has discovered. It is fast, reliable and consistent, but it is not intelligent. It executes; it does not think.
Machine Learning Detection Models
ML-based detection tools, whether they are detecting anomalous network behaviour, scoring alerts for risk, or classifying email content, are pattern-recognition systems. They are trained on data, they learn to identify signals associated with particular outcomes, and they apply that learning to new inputs. They are genuinely valuable and represent a meaningful advance over purely rule-based detection. But an ML model, however sophisticated, produces an output and stops. It does not then go and gather more information based on that output, reason about what the information means, decide what to do next and then do it. It classifies; it does not investigate.
Chatbots and Conversational AI Assistants
The natural language interfaces now being built into many security platforms, allowing analysts to query data or summarise alerts in plain English, are a useful and increasingly capable category of tooling. But responding to a question, however intelligently, is a single-step interaction. The user asks, the system answers, and control returns to the user. An agentic system does not wait to be asked what to do next. It decides for itself, acts on that decision, evaluates the result and continues towards its goal. Conversational AI assists; it does not act autonomously.
What Actually Defines an AI Agent
An AI agent, properly defined, is a system that can pursue a goal through a sequence of self-directed actions, using available tools and information, adjusting its approach based on what it discovers along the way, without requiring human instruction at each step.
Four properties distinguish a genuine AI agent from the tools described above.
Goal-directed reasoning
An agent is given an objective, not a script. Rather than executing a fixed sequence of steps, it reasons about what actions are most likely to achieve its goal given its current state of knowledge, takes those actions, observes the results and updates its reasoning accordingly. This ability to plan under uncertainty and adapt as new information becomes available is what separates agentic behaviour from rule-based or playbook-driven automation.
Tool use
Agents are not limited to reasoning in the abstract. They can use tools: querying databases, calling APIs, retrieving documents, sending notifications, triggering actions in other systems. In a security context, the tools available to an agent might include the ability to query a security data lake, retrieve threat intelligence, look up asset information, create or update a case management ticket, run a containment action through a cloud management API, or page an analyst through an alerting system. The richness of an agent’s tool access determines the scope of what it can accomplish autonomously.
Multi-step operation
A single query and response is not agentic behaviour. What makes a system agentic is its ability to chain actions across multiple steps, where each step is informed by the results of the previous one. An agent investigating a suspicious process execution on an endpoint does not simply retrieve the process log and stop. It then queries what else that process communicated with, checks whether those destinations are associated with known threat infrastructure, looks at what other endpoints may have run the same process, assesses the timeline of activity and builds a picture of potential scope, all within a single autonomous investigation thread.
Autonomous decision-making within defined boundaries
An agent makes decisions. Not all possible decisions, and not without constraints, but within its defined operating parameters it chooses what to do next without being told. This is the property that creates both the operational leverage and the governance requirements that come with agentic AI. An agent that can decide to escalate, to close, to gather more data, or to initiate a containment action is doing something qualitatively different from a tool that presents options for a human to choose between.
Why the SOC Is Particularly Well Suited to Agentic AI
Security operations is not an obvious first choice for autonomous AI systems. It involves high-stakes decisions, ambiguous information, adversarial conditions and significant consequences for errors in either direction. Missing a genuine threat is costly; acting incorrectly on a false positive can disrupt legitimate operations and erode trust in security tooling.
And yet the SOC is in many respects an excellent environment for agentic AI, precisely because its core challenge is one that agentic systems are well suited to address.
The fundamental problem in a security operations centre is the gap between the volume of events that need to be investigated and the human capacity available to investigate them. That gap is structural and growing. The attack surface expands continuously; the volume of security telemetry grows with it; the supply of experienced security analysts does not keep pace. The result is a sustained, systemic shortfall in investigative capacity that manifests as alert backlogs, superficial triage, delayed detection and analyst burnout.
Agentic AI addresses this gap directly by extending investigative capacity without the constraints of human bandwidth. An agent can run continuously, investigate in parallel across many alerts simultaneously, apply consistent rigour regardless of alert volume, and do so at a speed that human investigators cannot match. It does not get tired, does not become desensitised to repeated alert types and does not cut corners under time pressure.
It also operates in an environment that is, compared to many other domains, relatively well structured. Security data, particularly when normalised to a common schema such as OCSF, is amenable to systematic querying and correlation. The logic of a security investigation, while it requires judgement, follows recognisable patterns. And the tools available to security agents, data lakes, threat intelligence APIs, case management systems, cloud management interfaces, are increasingly well-documented and accessible. These structural features make security operations a more tractable domain for agentic AI than many others.
Realistic Near-Term Use Cases
The most useful question for security leaders assessing agentic AI is not what is theoretically possible but what is actually deliverable today, in real operational environments, with the tools and data infrastructure that exist now. The answer is more substantial than the hype sometimes suggests, and more bounded than the most ambitious vendor claims imply.
Autonomous alert triage and investigation
This is the most mature and immediately deployable agentic use case in security operations. An agent receives an alert, queries the relevant security data to gather the full event context, enriches the findings with threat intelligence and asset information, correlates the activity with related events across the environment, assesses severity and likely intent, and produces a documented investigation finding. Alerts assessed as low risk are closed with a rationale; alerts assessed as significant are escalated to a human analyst with a complete evidence package already assembled.
The operational impact is immediate and measurable. Analysts receive fewer alerts that require their attention, and the alerts they do receive come with substantially more context than they would have had if the analyst had initiated the investigation themselves. Mean time to investigate falls; coverage consistency improves; the quality of human attention is directed where it adds most value.
Threat hunting on a continuous basis
Traditional threat hunting is a periodic, analyst-driven activity: a skilled analyst forms a hypothesis about attacker behaviour, constructs queries to test that hypothesis against historical data and pursues the investigation until they either find something or exhaust the hypothesis. It is valuable but resource-intensive, and in most organisations it happens far less frequently than security teams would like.
Agentic AI can transform threat hunting from a periodic activity into a continuous one. Agents can be tasked with running defined hunting hypotheses against the security data lake on an ongoing basis, pursuing any promising findings, escalating confirmed or probable discoveries and logging negative results for future reference. The breadth of hypothesis coverage that a single human hunter can achieve in a week, an agentic system can cover continuously, across a much larger dataset.
Incident scoping and evidence gathering
When a significant security incident is confirmed, one of the most time-consuming early tasks is establishing its scope: which systems are affected, what data may have been accessed or exfiltrated, how the attacker moved through the environment and what the timeline of activity looks like. This work is critical for effective containment and recovery, but it is largely mechanical, involving systematic querying across multiple data sources to build a complete picture.
Agentic AI systems can run this scoping work in parallel with the human incident response team, covering a broader range of data sources more quickly than human investigators working sequentially. The result is a more complete and more rapidly assembled picture of incident scope, which directly improves the quality and speed of containment decisions.
Vulnerability context and prioritisation
Organisations routinely face vulnerability backlogs containing thousands of identified issues that cannot all be remediated immediately. Prioritising which vulnerabilities to address first requires combining the raw vulnerability data with contextual information about the affected assets: their exposure, their criticality, whether there is active exploitation of the vulnerability in the wild and whether there are compensating controls in place that reduce the effective risk.
Agents can automate the assembly of this contextual picture for each vulnerability, querying asset inventory systems, threat intelligence feeds and internal security controls data to produce a risk-adjusted prioritisation that reflects actual organisational exposure rather than generic severity scores alone. Security and IT teams receive a prioritised remediation list that is genuinely actionable rather than mechanically generated from CVSS scores.
Post-incident reporting and documentation
The documentation work that follows a security incident, assembling timelines, recording actions taken, producing reports for internal governance and external obligations, is both important and time-consuming. It is also largely a matter of organising and presenting information that already exists in logs, tickets and communications, rather than generating genuinely new analysis. Agentic systems can draft initial post-incident reports from the accumulated evidence and action records of an investigation, substantially reducing the time analysts spend on documentation and freeing them to focus on the lessons learned analysis that genuinely benefits from human judgement.
The Governance Question That Cannot Be Avoided
Any serious introduction to agentic AI in security must address the governance dimension, because it is the dimension that most directly determines whether autonomous security operations create value or create risk.
An agent that can take actions has the potential to take wrong actions. It can close an alert that warranted investigation. It can escalate in a way that disrupts legitimate operations. In more autonomous configurations, it can trigger containment actions based on a misassessment of the situation. These are not hypothetical risks; they are the predictable failure modes of any system that makes decisions under uncertainty.
Managing them requires deliberate design rather than hopeful assumption. Specifically, it requires clear definition of the boundary between what an agent can do autonomously and what requires human authorisation. In practice, most organisations beginning their agentic AI journey set this boundary conservatively: agents can query, correlate, enrich and recommend, but any action that affects systems, users or data requires explicit human approval. As confidence in agent judgement is established through operational experience, that boundary can be extended thoughtfully.
It also requires comprehensive audit capability. Every action taken by an agent, every query issued, every decision made and every rationale recorded should be logged in a way that allows it to be reviewed, understood and, if necessary, challenged. In a regulated environment, the ability to demonstrate that autonomous security actions were taken within approved parameters and for documented reasons is not optional.
Governance is not a constraint on agentic AI. It is the condition that makes agentic AI trustworthy enough to be genuinely useful.
Why It Matters Now
The question of timing is legitimate. Agentic AI has been discussed as a future capability for long enough that healthy scepticism about whether it has truly arrived is reasonable. The answer in 2026 is that it has, in a qualified but meaningful sense.
The foundation model capabilities that underpin agentic reasoning have matured substantially over the past two years. The infrastructure for connecting agents to tools and data sources, including platforms such as Amazon Bedrock Agents, has moved from experimental to production ready. The security data platforms needed to give agents something useful to work with, including Amazon Security Lake and its OCSF-normalised data model, are deployed and operating in real enterprise environments. And a growing body of early-adopter experience is producing the operational knowledge needed to implement agentic security systems with realistic expectations and appropriate governance.
None of this means that every organisation should deploy fully autonomous agentic security operations immediately. It means that the building blocks are sufficiently mature that security leaders who begin the journey now, starting with well-defined, well-governed use cases and expanding incrementally, will be substantially better positioned in two or three years than those who wait for the technology to mature further before engaging with it.
The SOC of the near future will be one in which autonomous agents handle the systematic, repeatable, data-intensive work of security investigation, and human analysts focus their irreplaceable capabilities on the judgement, creativity and adversary understanding that no AI system can replicate. Getting there requires starting somewhere. And the starting point is understanding, clearly and practically, what agentic AI is and what it can realistically do.
That understanding is now available. The next step is deciding what to do with it.
HOOP Cyber specialises in specialises in data-centric security operations, helping organisations build the foundations for AI-ready SOC environments through Amazon Security Lake, SIEM modernisation and data normalisation services.Contact us via to book a discovery call today.
In an industry that is never short of announcements, genuinely transformative moments can be easy to miss. The news that HOOP Cyber, one of the UK’s most innovative specialist security data operations firms, has joined the FSP Consulting Services Group, is one that deserves to be heard clearly.
To explore what it means, Lisa Ventura MBE FCIIS sat down with Simon Johnson, CEO and Founder of HOOP Cyber, for a wide-ranging conversation about the partnership, the rationale behind it, and what lies ahead for customers and the broader industry.
The Problem HOOP Cyber Was Built to Solve
When Simon Johnson founded HOOP Cyber three years ago, he had a clear and pressing problem in his sights. Having spent years working across security operations, SIEM, and threat intelligence, he had seen the same challenge play out time and again at organisations of all sizes: security data volumes had grown exponentially, but the tools and architectures used to manage them had not kept pace.
“My SIEM was fantastic and it was great ten years ago when I had 100 gigabytes of data, but now I’ve got two or three terabytes of data a day and I’m struggling.”- Simon Johnson, CEO and Founder, HOOP Cyber.
For many security teams, what had once been a quick and efficient way to interrogate data had become, in Simon’s words, “a genuine data engineering millstone around the neck.” Storing, querying, and drawing insight from vast quantities of security data had become slower, more complex, and significantly more costly.
HOOP Cyber was built specifically to address this reality, developing a set of approaches centred on what the firm calls modernised security operations. At its core is the conviction that cybersecurity is, fundamentally, a data problem. The company’s flagship offering, HOOP Lake, powered by Amazon Security Lake, transforms the way security teams can stream, store, search, enrich, and comply across their entire estate of security events, using standardised schemas such as OCSF and natural language query capabilities to make security data genuinely accessible and actionable.
Why FSP? Why Now?
Simon spoke candidly about the journey that led him here, and the qualities that made FSP the right partner at the right moment.
As HOOP Cyber grew, it found itself working with some very large customers. Delivering outstanding outcomes at that scale required significant capacity and capability that, as a focused specialist firm, was not easy to build alone. Simon began actively exploring how to deliver at the pace and scale his clients needed, while preserving the values and culture that had made HOOP Cyber distinctive.
FSP Consulting Services Group is a 450-person cyber consulting business headquartered in Reading, UK. It has earned a strong industry reputation not only for technical excellence across enterprise transformation and cyber security, but also for being recognised as an outstanding place to work, a quality that Simon found was lived out genuinely by the people he encountered day to day.
“Culture is absolutely key to any successful environment. I think I definitely felt like the culture is a perfect fit.” – Simon Johnson, CEO and Founder, HOOP Cyber.
Beyond the culture question, FSP offered the capacity and multi-domain capability to take HOOP Cyber’s work further than it could travel alone. Where HOOP had historically operated within the specific domain of security operations, FSP opens the door to a broader set of disciplines including data engineering, AI adoption, and enterprise transformation, creating a proposition that can support clients across and beyond the CISO’s remit.
What This Means for Existing Customers
For those already working with HOOP Cyber, the message from Simon is very reassuring. The core focus remains unchanged: building modernised security operations with a data-centric approach. What changes is the scale at which HOOP can deliver, and the breadth of expertise it can bring to bear.
Key Points for Existing HOOP Cyber Clients:
The same specialist focus on security data operations and SIEM modernisation continues
Delivery can now happen at significantly greater scale, backed by a 450-person consultancy
Clients gain access to FSP’s broader capabilities across cyber, data engineering, and AI adoption
Services extend across the UK and, in time, across EMEA and the US
FSP’s vendor-agnostic approach means clients receive the right solution for their specific environment
As Simon put it simply, for existing clients it is “nothing changes. Hopefully it’s only just a bit more of the same good stuff.” The expanded platform means that HOOP Cyber can now help clients not only with the technical mechanics of managing security data, but also with how they securely adopt AI, understand the business case and return on investment behind AI investments, and navigate the intersection of data, security, and digital transformation.
The AWS Partnership: A Strategic Differentiator
One of the most significant aspects of HOOP Cyber’s heritage that it carries into the FSP Group is its strategic partnership with Amazon Web Services. This is, in Simon’s view, one of the most compelling elements of the combined offering.
AWS has been a strong advocate for HOOP Cyber’s growth, particularly in the areas of security data modernisation and Amazon Security Lake. FSP, for its part, had been looking to deepen its AWS capabilities to serve the many clients who have substantial investments in AWS environments or who run SaaS applications within AWS infrastructure.
The result is a focused ambition: to build a world-class AWS partnership that can offer FSP clients a completely new level of experience around the adoption of AWS solutions, cloud migrations, and the broader AWS ecosystem. It is a natural extension of the data-centric security work HOOP Cyber has always done, now with the scale and reach of a larger enterprise consultancy behind it.
The Future of Security Data Operations
The conversation with Lisa Ventura MBE FCIIS closed with a forward-looking question: what does the future of security data operations actually look like? Simon offered a grounded and thoughtful perspective, shaped by years of working at the coalface of the problem.
The fundamental challenge will not change. Security remains a data problem. The questions that will increasingly preoccupy security teams are architectural and strategic: should data be centralised or left distributed? How can data lake solutions such as Amazon Security Lake enable aggregation without the cost and complexity of traditional approaches? How can open schemas like OCSF make data simpler and faster to query? Can federated search and query capabilities decouple data storage from data interrogation?
“It’s critical to be able to have access to the right data, to ask that data the right question at the right time.” – Simon Johnson, CEO and Founder, HOOP Cyber.
AI will inevitably play an increasingly prominent role, and Simon acknowledged this candidly, noting that the industry will “continue to see a splattering of AI here, there, and everywhere.” Detection as code, doing more with less, and smarter automation are themes that will persist. But underneath all of it, the data problem endures, and that is precisely where HOOP Cyber and FSP are focused.
A Purposeful, People-Centred Approach
Throughout the conversation not only the strategic logic of the partnership emphasised, but also the spirit in which it had been forged. This is a combination built on shared values, a genuine alignment of culture, and a belief that the best outcomes for clients come from organisations where people are genuinely invested in each other’s success.
As Lisa observed in closing, this is precisely the kind of purposeful, people-centred approach to cybersecurity that the industry needs considerably more of. The joining of HOOP Cyber and FSP Consulting Services Group is not simply a corporate transaction; it is a signal of intent, and one that the security data operations space would do well to pay close attention to.
Find Out More
Whether you are an existing HOOP Cyber client, a prospective partner, or simply following the evolution of security data operations, this is a chapter worth watching closely.
We’re thrilled to announce that HOOP Cyber has joined FSP Consulting Services Group, an award winning enterprise transformation and cyber security consultancy. This marks an exciting evolution for our team, our technology, and most importantly, the value we deliver to our clients.
Why This Matters
Since launching HOOP Cyber, we’ve been on a mission to solve the security data challenge. We’ve helped organisations modernise their security operations through Amazon Security Lake, transform unwieldy SIEM costs into efficient data lake architectures, and build automated data pipelines that give security analysts time to think rather than just time to react.
Joining FSP accelerates everything we’ve been building. FSP’s comprehensive capabilities across cyber security, cloud, data, and AI complement our specialist security data operations expertise. Their awardwinning culture and numerous #1 and top rankings by Best CompaniesTM and Great Place to WorkTM ,aligns perfectly with our commitment to building sustainable, people centred security operations.
What This Means for Our Clients
If you’re a HOOP Cyber client, you’ll continue to work with the team you know whilst gaining access to FSP’s broader portfolio of services. Need help with broader cyber strategy alongside your security data lake implementation? FSP’s Virtual CISO and Cyber Strategy teams can help. Looking to integrate your security data operations with wider cloud transformation initiatives? FSP’s cloud and platform engineering expertise is now available to you. Want managed services to support your security operations long-term? FSP’s comprehensive managed services portfolio has you covered.
Your HOOP Cyber projects continue uninterrupted, with the same expertise and dedication you’ve experienced, now backed by FSP’s extensive resources and capabilities.
What This Means for Our Partners
Our strategic partnerships with AWS, Query, Tenzir, Cyware, Tines, DataBee, and Silent Push remain central to how we deliver value. These partnerships become even stronger within FSP’s broader technology ecosystem, creating new opportunities for integrated solutions that address the full spectrum of security and digital transformation challenges.
What This Means for Our Team
For the HOOP Cyber team, joining FSP means joining an organisation that genuinely lives its values of “people first, purpose led, performance driven”. FSP’s investment in professional development through the FSP Academy, their commitment to work-life balance, and their recognition as a world-class workplace creates an environment where our team can continue to grow, innovate, and deliver exceptional outcomes for clients.
Looking Forward
The security landscape continues to evolve rapidly. Data volumes are exploding, threats are becoming more sophisticated, and organisations need security operations that are both effective and sustainable. By combining HOOP Cyber’s specialist security data operations expertise with FSP’s comprehensive capabilities, we’re better positioned than ever to help organisations navigate these challenges.
We remain committed to the principles that have guided HOOP Cyber from the start: solving real problems with pragmatic solutions, respecting the humans who operate security systems, and building architectures that are sustainable, cost-effective, and genuinely improve security outcomes.
Thank you to our clients, partners, and supporters who have been part of the HOOP Cyber journey so far. We’re excited about this next chapter and the enhanced value we can deliver together with FSP.
For any questions about what this means for your projects or partnerships, please reach out to us at the usual contact details. We’re here, we’re excited, and we’re ready to do even better work together.
The UK Cyber Security and Resilience Bill has received a huge amount of commentary since its introduction. Much of that coverage, understandably, has focused on the broad strokes: expanded regulatory scope, new incident reporting obligations, and the extension of oversight to managed service providers and digital supply chains. This all matters, but for the security engineers and architects who must translate policy into working infrastructure, the questions that matter most are considerably more specific.
How long do we need to retain logs? What does a compliant audit trail actually look like? And what does this mean for how we build and operate our SIEM or data lake architecture going forward?
Those are the questions this post addresses. The Bill has not yet received Royal Assent, and some details remain subject to secondary legislation and regulatory guidance. Even so, the direction of travel is clear enough to start making informed architectural decisions now.
The Reporting Obligation and What It Implies for Log Availability
The Bill introduces mandatory incident reporting requirements for in-scope organisations, with initial notification timelines that are expected to be considerably shorter than current NIS Regulations thresholds. Early indicators suggest a 24-hour initial reporting window is likely for significant incidents, with a more detailed follow-up report to follow within 72 hours.
This matters for your SIEM in a very practical way. A 24-hour clock starts ticking from the point an incident is identified, not from the point it began. That means your ability to reconstruct an accurate timeline of events quickly is not a nice-to-have. It is a compliance requirement.
If your current log pipeline involves any significant ingestion lag, delayed normalisation, or gaps in source coverage, those gaps will become visible the moment you try to produce a credible incident narrative under time pressure. Your SIEM needs to be able to answer the question: what happened, when, and across which systems? And it needs to answer that question within hours, not days.
Log Retention: Moving Beyond the 90-Day Default
Many organisations currently retain hot logs for 90 days as a rough industry baseline, with varying approaches to cold or archival storage beyond that. The Bill, alongside likely guidance from the National Cyber Security Centre and the relevant sector regulators, is expected to push this considerably further for in-scope entities.
A 12-month minimum for core security telemetry is widely anticipated, with some regulated sectors likely to face longer obligations. This aligns with patterns already visible in NIS2 across the EU and in FCA expectations for financial services firms, and it would be reasonable to design your architecture around that assumption.
The architectural implication is straightforward but significant. Retaining 12 months of logs at SIEM licensing costs is prohibitively expensive for most organisations. The answer is tiered storage: hot data in your SIEM for active detection and investigation, warm data in a data lake or object storage layer for the medium term, and cold archival for the longer tail. The key is ensuring that data in the warm and cold tiers remains searchable and retrievable within a defined SLA, because a regulator asking for logs from seven months ago will not accept “we need two weeks to restore those from tape” as a satisfactory response.
Audit Trails: Integrity, Not Just Availability
There is an important distinction between having logs and having audit trails. Logs are raw telemetry. Audit trails are structured, tamper-evident records of events that carry evidentiary weight. The Bill’s emphasis on accountability and demonstrable resilience points firmly toward the latter.
For your SIEM architecture, this means thinking seriously about log integrity. Can you demonstrate that a log record has not been altered since it was written? Do you have hash verification or write-once storage for your most sensitive telemetry? Is your audit logging covering not just endpoint and network events, but privileged access, configuration changes, and administrative actions within your security tooling itself?
The categories of events that are likely to receive explicit regulatory attention include:
Authentication events, including failed attempts, MFA bypass, and privilege escalation
Changes to access controls, group membership, and identity configurations
Data exfiltration indicators, including large file transfers and cloud egress anomalies
Changes to security tooling configuration, including your SIEM itself
Network boundary events from firewalls, proxies, and DNS logs
If these categories are not comprehensively covered in your current log sources, now is the time to address those gaps rather than after Royal Assent.
Data Lake Architecture: Opportunity or Obligation?
The growing move toward security data lakes, whether built on platforms such as Snowflake, Databricks, Microsoft Sentinel with ADX, or open-source alternatives, is not purely a cost optimisation story. In the context of the Bill, it is also a compliance enabler.
A well-designed data lake gives you the ability to retain far greater volumes of telemetry at lower cost, run retrospective threat hunting across extended time windows, and produce structured evidence in response to regulatory enquiries. These are precisely the capabilities that the Bill’s reporting and accountability obligations will require.
However, the architecture only delivers on that promise if the data is properly structured at ingestion. Logs dumped into object storage without schema normalisation or indexing are not audit trails. They are a compliance liability dressed up as a solution. Invest in your data pipeline: consistent schemas, reliable timestamps, clear source attribution, and documented retention policies with access controls that reflect their regulatory purpose.
Managed Service Providers: A Specific Consideration
One of the more significant expansions in the Bill is the explicit inclusion of managed service providers and digital supply chain operators. If your organisation operates as an MSP, or if you rely on one to manage any part of your security infrastructure including your SIEM, you need to understand what this means contractually and operationally.
If you are a customer of an MSP: your contracts should specify who owns the log data, what retention obligations apply, and under what circumstances you can retrieve data for regulatory purposes. Data sovereignty and portability clauses matter more than they may have done previously.
If you are an MSP: you will likely be in scope for reporting obligations directly, and you will need to demonstrate that your own monitoring and audit capabilities meet the standard expected of critical infrastructure providers. Multi-tenant SIEM architectures in particular need careful review to ensure log isolation, customer data portability, and incident scoping are all functioning correctly.
Preparing Now, Before Royal Assent
The Bill is still progressing through Parliament, and secondary legislation will fill in many of the specific thresholds and technical requirements. But the core obligations around incident reporting speed, log retention duration, audit trail integrity, and supply chain accountability are sufficiently clear that waiting for final text before acting becomes a risk.
The organisations that will find compliance most straightforward are those that use this period to review their log source coverage, validate their retention architecture, and stress-test their ability to produce structured evidence under time pressure. That work has value regardless of the final regulatory detail.
Ready To Find Out More?
If you would like to talk through what the Bill means for your specific SIEM architecture or log management strategy, get in touch with the HOOP Cyber team via . We work with organisations across sectors to build detection and response capabilities that are operationally effective and built for the regulatory environment ahead.
Maturity models have always had a complicated relationship with operational reality. At their best, they give security leaders a shared language for describing where they are and a credible map of where they need to go. At their worst, they become a compliance checklist that mistakes the presence of tools for the presence of capability. In 2026, with AI now embedded across the detection and response landscape, the gap between the two has never been wider.
The traditional maturity conversation, built around frameworks like the SOC-CMM or adapted versions of the CMMI, was designed for a world of manual analyst workflows, perimeter-based thinking, and relatively predictable threat actor tooling. That world no longer exists. Adversaries are using AI to accelerate reconnaissance, generate convincing phishing content at scale, and adapt malware faster than signature-based detection can follow. The security operations function that is not actively incorporating AI into its own workflows is not standing still. It is falling behind.
This piece offers a practical framework for CISOs seeking to benchmark their SecOps function against a definition of maturity that reflects the current threat and technology environment. It is not a vendor checklist. It is a strategic lens.
Why the Old Models Need Updating
Classic SecOps maturity frameworks typically measured progress along axes like process documentation, technology coverage, and analyst capability. A Level 1 organisation was reactive and largely undocumented. A Level 5 organisation had optimised, continuously improving processes with full visibility across the estate.
The problem is not that these dimensions are wrong. It is that they are incomplete. They say nothing about how intelligence is operationalised, how alert triage decisions are made, how human cognitive load is managed at scale, or how the security function responds when the volume and velocity of threats exceeds what human analysts can process without assistance.
For a CISO benchmarking their organisation in 2026, the relevant questions are not just “do we have a SIEM?” or “are our playbooks documented?” They are: how effectively are we using AI to reduce analyst toil? How well does our automation distinguish between genuine threats and noise? And critically, where does human judgement remain essential, and are we protecting the conditions that allow that judgement to function well?
A Revised Maturity Model: Five Levels Redefined
The framework below defines five maturity levels across four dimensions: detection capability, response capability, AI integration, and human factors. The final dimension is one that conventional models consistently overlook, and it is increasingly the differentiator between organisations that realise the value of their AI investment and those that do not.
Level 1: Reactive
At the base level, log collection is inconsistent and alerts are reviewed manually on demand rather than continuously monitored. Incident response is entirely ad hoc, with no documented playbooks and no clear escalation paths. AI and automation play no meaningful role; tools may be present but are not operationalised. Analysts are overwhelmed, fatigue is endemic, and turnover is high. The function is consuming resource without generating reliable security outcomes.
Level 2: Defined
A SIEM is in place and standard use cases have been deployed, but the false positive rate remains high and detection logic has not been meaningfully tuned to the organisation’s specific environment. Response playbooks exist on paper, but execution is largely manual and escalation paths remain unclear. Automation is limited to basic alerting rules within the SIEM. Workload is beginning to be managed, and some skills development is planned, but the function is still largely reactive in practice.
Level 3: Integrated
Telemetry coverage is broad and threat intelligence feeds have been operationalised. Detection logic is tuned and generating fewer false positives. Documented runbooks are supported by partial automation, and mean time to respond is improving measurably. AI-assisted triage is in place, SOAR has been deployed, and analyst time is increasingly focused on higher-value investigative work rather than manual alert processing. Workload is managed more deliberately, some specialisation is emerging within the team, and trust in automated tooling is beginning to develop.
Level 4: AI-Augmented
Behavioural and anomaly-based detection models are tuned to the organisation’s environment and generating a low false positive rate. Automated containment handles known threat patterns, with human oversight reserved for complex or ambiguous cases. AI is integrated across detection, triage, and response, with feedback loops in place to continuously improve model performance. Analysts are focused on work that genuinely requires human judgement. Cognitive load is actively managed, and the team understands and can interrogate the AI tooling they work alongside.
Level 5: Adaptive
Detection capability improves continuously, with adversarial simulation informing model tuning and full visibility across the attack surface. Response is near-autonomous within defined parameters, with human decision-making reserved for novel or high-impact scenarios that require contextual judgement. AI is treated as a strategic capability, with explainability and bias monitoring in place and a mature governance model governing its use. The function attracts and retains strong talent, maintains a clear model for human-AI collaboration, and develops skills continuously in response to the evolving threat landscape.
The Human Factors Dimension: What the Models Miss
Every SecOps function, regardless of how sophisticated its tooling, ultimately depends on human beings making good decisions under pressure. The AI augmentation conversation has, in many organisations, focused almost exclusively on reducing the burden on analysts. That is necessary but insufficient.
The organisations that score highest on genuine maturity in 2026 are those that have thought carefully about the conditions under which their analysts perform well, not just the tools those analysts are given. That means addressing alert fatigue not just through better filtering, but through workload design. It means ensuring that the move toward automation does not erode the deep investigative skills that are essential when novel threats appear. And it means building psychological safety into the SOC culture so that analysts feel able to escalate uncertainty rather than resolving it by closing tickets.
This is not a soft concern. Organisations that neglect the human factors dimension will find that their AI investment underperforms, because the feedback loops that make AI models improve over time depend on skilled humans correcting, refining, and contextualising automated outputs.
Where Most Organisations Actually Are
Based on conversations across sectors, the honest picture is that the majority of UK enterprises currently sit between Level 2 and Level 3 on this framework. They have a SIEM. They have some playbooks. They are beginning to deploy SOAR or AI-assisted triage tooling. But their detection logic is still generating significant noise, their automation coverage is patchy, and their AI deployments are often pilot projects that have not yet been fully operationalised.
The gap between Level 3 and Level 4 is where genuine strategic investment is required. It is not primarily a technology gap. Organisations at Level 3 typically have adequate tooling. The gap is in data quality, process maturity, and the organisational confidence to trust automated decision-making in defined scenarios. Building that confidence requires transparency in how AI models reach conclusions, governance frameworks that define the boundaries of autonomous action, and a track record of measured, low-stakes automation that earns trust incrementally.
Practical Steps for CISOs Benchmarking Their Function
For a CISO seeking to use this framework practically, the starting point is an honest assessment across all four dimensions: detection, response, AI integration, and human factors. The temptation is to score the function at its highest-performing dimension. The more useful approach is to identify the dimension where the lowest score sits, because that is almost always the constraint that limits overall maturity.
A few questions worth working through with your team:
If a novel threat actor technique appeared in your environment today, how long before it would generate an alert? How long before an analyst would act on it?
What percentage of your alerts are closed by automation without analyst review? Of those, what proportion are genuinely benign versus suppressed because the signal was too noisy to manage?
Can your analysts explain how your AI-assisted triage tooling reaches its conclusions? Do they trust it? Do they have a mechanism for flagging when it appears to be wrong?
When did someone last deliberately test your detection capability using realistic adversary simulation? What did you learn?
The answers to these questions will tell you more about your actual maturity position than any tool inventory.
The Strategic Ambition: Good Is Not Static
The most important characteristic of a Level 5 SecOps function is not any specific technology or process. It is the ability to adapt continuously. The threat landscape in 2026 is not what it was in 2023, and it will not be what it is now by 2028. AI-driven attacks are becoming more sophisticated. The regulatory environment is becoming more demanding. The attack surface, spanning cloud, OT, supply chain, and identity, continues to expand.
A mature SecOps function is one that learns. It updates its detection logic when adversary techniques evolve. It revises its response playbooks when post-incident reviews identify gaps. It invests in analyst capability as the skills required for effective human-AI collaboration shift over time. And it governs its AI tools with the same rigour it applies to any other critical operational system.
That is what good looks like. Not a fixed destination, but a demonstrated capacity to keep up, and occasionally get ahead.
Ready To Find Out More?
If you would like to work through where your organisation sits on this framework, or to explore what a realistic roadmap toward Level 4 capability looks like for your specific environment and constraints, get in touch with the HOOP Cyber team via . We work with organisations across sectors to build detection and response capabilities that are operationally effective and built for the regulatory environment ahead.
For organisations operating at scale, Amazon Security Lake presents both an opportunity and a challenge. The opportunity is centralised security visibility across vast, distributed infrastructure. The challenge is architecting that visibility in ways that respect organisational boundaries, regulatory requirements, and operational realities.
Single-account Security Lake implementations work well for small organisations with straightforward requirements. But global enterprises face complexities that demand thoughtful architectural decisions: multiple AWS accounts managed through AWS Organisations, operations spanning numerous geographic regions, data sovereignty requirements that prohibit cross-border data movement, and diverse business units with different security maturity levels and compliance obligations.
This article explores proven design patterns for multi-account Security Lake architectures, drawing from real-world implementations across financial services, healthcare, and technology sectors.
Understanding Multi-Account Complexity
Before diving into specific patterns, it’s worth understanding why multi-account architectures matter and what unique challenges they introduce.
Most large organisations adopt multi-account strategies for good reasons. Account boundaries provide security isolation, limiting blast radius when credentials are compromised. They enable separate billing and cost allocation across business units or projects. They allow different compliance postures, with highly regulated workloads in tightly controlled accounts whilst development environments operate with greater flexibility. They support organisational structure, mapping accounts to business units, geographic regions, or operational functions.
When overlaying Security Lake on this multi-account reality, several questions immediately arise. Where should security data be stored? Who should have access to which data? How do you balance centralised visibility with data sovereignty requirements? How do you manage costs when data volumes vary dramatically across accounts? How do you handle accounts that join or leave the organisation?
These aren’t merely technical questions. They involve organisational politics, regulatory compliance, operational procedures, and long-term strategic considerations.
Core Architectural Patterns
Based on extensive field experience, four primary patterns have emerged for multi-account Security Lake deployments. Each addresses different organisational requirements and makes different trade-offs.
Pattern 1: Centralised Security Lake
In this pattern, all security data from all accounts flows to a single Security Lake instance in a dedicated security account. This creates a single source of truth for security data across the organisation.
The centralised pattern offers significant advantages. Security teams gain unified visibility across all accounts from a single query interface. It provides simplified operational management with one Security Lake instance to monitor and maintain. Cross-account investigations become straightforward when all data resides in one location. Cost management is simplified through centralised billing and easier application of lifecycle policies.
However, centralisation introduces challenges. Data sovereignty requirements may prohibit moving certain data to a central region. Blast radius increases, as a compromise of the security account affects all security data. Network data transfer costs can become significant when moving large volumes across regions. Compliance complexity increases when mixing data with different regulatory requirements.
This pattern suits organisations with relatively homogeneous compliance requirements, operations concentrated in one or two geographic regions, and strong central security teams with broad authority across the organisation.
Pattern 2: Regional Security Lakes with Federated Querying
For organisations operating globally with strict data residency requirements, regional Security Lakes provide a better fit. Security data remains in the region where it was generated, with federated querying enabling cross-regional analysis when needed.
This pattern addresses critical requirements for global organisations. Data sovereignty compliance is maintained by keeping data within required geographic boundaries. It provides regulatory isolation where different regions may have different compliance obligations. Network costs are reduced by avoiding unnecessary cross-region data transfer. Blast radius is limited, as compromise of one regional Security Lake doesn’t affect others.
The trade-offs involve increased operational complexity from managing multiple Security Lake instances. Cross-regional investigations require federated querying capabilities rather than simple SQL. Cost visibility becomes more complex with billing split across multiple regions. Ensuring consistency in data retention, access policies, and integration configurations across regions requires careful orchestration.
This pattern is essential for organisations with operations spanning multiple regulatory jurisdictions, particularly those subject to GDPR, data localisation laws in China or Russia, or strict data residency requirements in healthcare and financial services.
Pattern 3: Delegated Security Lakes per Business Unit
Some large organisations operate with highly autonomous business units that maintain separate security teams and tools. In this pattern, each business unit manages its own Security Lake instance, with optional data sharing to a central security team.
This pattern provides business unit autonomy, allowing independent security operations and tool selection. It enables clear cost allocation with each unit bearing its own Security Lake costs. Access control is simplified within business unit boundaries. It supports different security maturity levels where advanced units can implement sophisticated detections whilst others start with basics.
However, it introduces enterprise-wide visibility challenges requiring coordination across multiple teams. Duplication of effort occurs as each unit potentially reimplements similar integrations and detections. Compliance verification becomes complex when auditors need to review multiple independent implementations. Knowledge sharing suffers when security teams operate in silos.
This pattern suits large conglomerates with diverse business portfolios, organisations that have grown through acquisition where business units retain operational independence, and federated operating models where business units have profit and loss responsibility.
Pattern 4: Hybrid Architecture
Most large organisations ultimately implement a hybrid approach that combines elements of the patterns above. A typical hybrid might include regional Security Lakes for data sovereignty compliance, centralised replication of specific high-value data sources for enterprise-wide threat hunting, and business unit autonomy for tool selection with standardised data sharing.
Hybrid architectures match the messy reality of large organisations but require sophisticated design to avoid creating a complicated mess rather than an elegant solution. Success requires clear principles for deciding what data goes where, well-defined interfaces for data sharing and federated querying, strong governance to prevent architecture drift over time, and robust automation to manage complexity without overwhelming operations teams.
Design Considerations for Multi-Account Deployments
Regardless of which pattern you adopt, several design considerations apply across multi-account Security Lake architectures.
Access Control and Permissions
Multi-account environments complicate access control significantly. You must decide how to structure cross-account access, typically using AWS Organisations service control policies to establish baseline permissions, cross-account IAM roles for Security Lake access from centralised security tools, resource-based policies on Security Lake S3 buckets for granular access control, and AWS Lake Formation for fine-grained data access permissions.
Implement least privilege rigorously. Just because data flows to a central Security Lake doesn’t mean everyone in the security organisation should access all data. Consider implementing separate query roles for different teams, data masking for sensitive fields like personally identifiable information, and time-based access controls for particularly sensitive data sources.
Cost Allocation and Chargebacks
With security data flowing from multiple accounts, cost allocation becomes critical for accountability and budgeting. Implement tagging strategies that identify data sources by account, business unit, application, and environment. Use AWS Cost Explorer to track costs by tag and establish chargeback mechanisms if business units are responsible for their security data costs.
Consider implementing quotas or cost controls to prevent runaway spending. A misconfigured source generating excessive events can quickly inflate costs. Automated alerts when costs exceed thresholds provide early warning before month-end surprises.
Data Retention and Lifecycle Management
Different data types and regulatory requirements demand different retention periods. CloudTrail management events might require seven-year retention for compliance, whilst VPC Flow Logs might only need 90 days for operational analysis. Network connection logs from development environments might warrant 30 days, whilst production logs require a year.
Implement lifecycle policies that automatically transition data to cheaper storage classes based on age and importance. Recent data stays in S3 Standard for fast query access. Data older than 90 days moves to S3 Infrequent Access. Data older than a year moves to Glacier for long-term compliance retention.
Tag data at ingestion with appropriate retention requirements and use automation to apply lifecycle policies consistently across accounts and regions.
Data Quality and Consistency
With data flowing from numerous sources across many accounts, ensuring consistent data quality becomes challenging. Implement schema validation at ingestion to catch format changes before they cause downstream issues. Monitor for data freshness to detect when sources stop sending events. Track event volumes to identify unexpected increases or decreases. Validate OCSF compliance to ensure consistent normalisation across sources.
Consider creating a data quality dashboard that provides visibility into ingestion health across all accounts and sources. This operational visibility is critical for maintaining trust in Security Lake as the authoritative security data platform.
Real-World Implementation Example
Consider a global financial services firm with operations in North America, Europe, and Asia-Pacific. They operate over 300 AWS accounts across development, staging, and production environments for multiple business units.
Their regulatory requirements include GDPR in Europe, requiring EU data to remain in EU regions, APAC data localisation laws in several countries, and PCI-DSS for payment card data requiring specific controls. Their organisational structure includes a central security operations centre with 24/7 coverage, regional security teams in each major geography, and business unit security leaders with varying technical sophistication.
Their implemented architecture uses regional Security Lakes in eu-west-1, us-east-1, and ap-southeast-1 to address data sovereignty. It includes selective replication of critical alerts and high-value events to a central security account for threat hunting. Business units maintain autonomy for tool selection but must implement standard OCSF data sharing. It features automated data classification at ingestion to enforce appropriate retention and access controls.
Implementation followed a phased approach. Phase 1 established the regional Security Lake infrastructure and onboarded AWS native sources. Phase 2 integrated third-party security tools using standardised OCSF transformations. Phase 3 implemented federated querying across regional instances for cross-border investigations. Phase 4 deployed automated compliance reporting and cost chargeback mechanisms.
Key success factors included executive sponsorship from the Chief Information Security Officer who resolved cross-business-unit conflicts. They used standardised infrastructure as code templates for consistent deployment across regions. A centre of excellence provided guidance and shared best practices across business units. Phased rollout allowed learning from early deployments before full-scale implementation.
The results after twelve months showed 40% reduction in security data costs compared to previous multi-SIEM architecture, mean time to detect cross-account threats decreased from days to hours, compliance audit preparation time reduced by 60%, and security team satisfaction improved significantly due to unified data access.
Governance and Operating Model
Technical architecture alone doesn’t ensure success. Multi-account Security Lake deployments require thoughtful governance and clear operating models.
Establish a Security Lake Centre of Excellence responsible for architecture standards, providing reusable integration templates and guidance, reviewing new source integrations for quality and compliance, and managing the roadmap for shared capabilities.
Define clear data ownership where account owners are responsible for data quality from their sources, regional security teams own regional Security Lake operations, the central security team owns cross-regional analytics and threat intelligence, and business unit leaders approve access to their data by other teams.
Implement change management processes for architectural changes that affect multiple accounts, new data source integrations that might impact costs or performance, modifications to retention policies, and access control changes.
Create feedback mechanisms including regular architecture review meetings, incident retrospectives that identify gaps in data coverage, cost review sessions that identify optimisation opportunities, and user feedback forums where analysts can request new data sources or capabilities.
Migration Considerations
For organisations with existing security data infrastructure, migrating to a multi-account Security Lake architecture requires careful planning.
Start with a clear understanding of current state: inventory all security data sources across accounts, document existing access patterns and user workflows, identify compliance requirements by data type and geography, and quantify current costs for security data storage and analysis.
Develop the target architecture using one of the patterns discussed, with explicit design decisions documented and rationale captured. Create a phased migration plan that starts with AWS native sources in a pilot account group, then expands to additional accounts while monitoring performance and costs. Integrate third-party sources using lessons from pilot phase, and finally decommission legacy infrastructure once Security Lake proves sufficient.
Plan for the hybrid state where both old and new systems operate in parallel. This might last months or even years in large organisations. Ensure analysts have access to both systems during transition. Develop runbooks that specify which system to query for different time ranges or data types.
Measure success through operational metrics like mean time to detect and respond, compliance metrics such as audit findings and remediation time, cost metrics including total cost of ownership and cost per gigabyte, and user satisfaction from security analysts and investigators.
Common Pitfalls and How to Avoid Them
Multi-account architectures introduce failure modes that don’t exist in simpler deployments.
Avoid over-centralisation by respecting data sovereignty requirements even when technically possible to centralise. Compliance violations can result in significant fines. Don’t under-centralise either, where excessive fragmentation makes cross-account investigations impractical. Find the right balance for your organisation’s requirements.
Don’t neglect cost controls. Multi-account environments can hide runaway spending until it’s too late. Implement monitoring and alerting early. Avoid inconsistent configurations where each account implements Security Lake differently. Use infrastructure as code and automation to enforce consistency.
Don’t underestimate the operational complexity of running multiple Security Lake instances. Ensure you have the team skills and capacity before implementing distributed patterns. Avoid ignoring the change management and communication required when deploying across multiple business units. Technical excellence alone doesn’t guarantee adoption.
Future-Proofing Your Architecture
Security Lake is evolving rapidly, with AWS adding new capabilities regularly. Design your multi-account architecture with flexibility for future enhancements.
Build on open standards, particularly OCSF, to avoid proprietary lock-in that complicates future migrations. Implement abstraction layers so that changes in underlying AWS services don’t require rewriting all your integrations and analytics. Use infrastructure as code so that architecture updates can be rolled out consistently across accounts and regions.
Stay engaged with the Security Lake roadmap and AWS announcements. Plan for upcoming capabilities that might simplify your architecture or provide new functionality. Regularly review your architecture as your organisation evolves, as mergers and acquisitions might require accommodating new accounts, and geographic expansion might require new regional instances.
Conclusion
Multi-account Security Lake architectures require more sophisticated design than single-account deployments, but they’re essential for organisations operating at scale. The patterns and considerations discussed here provide a framework for making informed architectural decisions that balance visibility, compliance, cost, and operational complexity.
The organisations succeeding with multi-account Security Lake implementations share common characteristics: clear architectural principles that guide design decisions, strong governance that prevents architecture drift, automation that makes complexity manageable, and phased implementation that allows learning and adjustment.
Most importantly, they recognise that architecture is a means to an end. The goal isn’t an elegant diagram but effective security operations that detect and respond to threats faster than adversaries can operate.
Ready to Design Your Multi-Account Security Lake Architecture?
At HOOP Cyber, we’ve designed and implemented Security Lake architectures for global enterprises across industries. Our team understands the regulatory, operational, and technical complexities of multi-account deployments.
Whether you’re planning a greenfield implementation or migrating from existing infrastructure, we can help design an architecture that meets your requirements whilst avoiding common pitfalls.
Contact HOOP Cyber at to discuss your multi-account Security Lake requirements and learn how we can support your implementation.
Amazon Security Lake has transformed how organisations centralise and analyse security data from AWS services. With native integrations for CloudTrail, VPC Flow Logs, Route 53, and other AWS sources, getting started with Security Lake is straightforward. However, the real value emerges when you extend beyond AWS’s native ecosystem to include third-party security tools, on-premises infrastructure, and custom applications.
This guide provides a practical walkthrough for security teams looking to onboard custom sources into Security Lake, addressing the technical challenges, architectural decisions, and operational considerations that matter in production environments.
Understanding the Custom Source Integration Challenge
Most organisations operate heterogeneous security stacks that generate valuable telemetry from sources Amazon Security Lake doesn’t natively support. These might include:
Third-party security platforms such as endpoint detection and response (EDR) solutions, network detection and response (NDR) tools, or cloud access security brokers (CASBs). On-premises infrastructure including legacy SIEM platforms, firewalls, intrusion detection systems, or proxy servers. Custom applications and services that generate security-relevant events but lack standard integration paths. SaaS applications outside the AWS ecosystem such as Microsoft 365, Google Workspace, or Salesforce.
Without these sources, your Security Lake provides incomplete visibility, undermining the core value proposition of centralised security data analysis. The challenge is bringing these disparate sources into Security Lake whilst maintaining data quality, ensuring proper normalisation to OCSF, and avoiding the operational overhead that plagued traditional SIEM implementations.
The OCSF Normalisation Imperative
Before diving into integration mechanics, it’s crucial to understand why normalisation matters. The Open Cybersecurity Schema Framework (OCSF) provides a vendor-agnostic taxonomy for security events. When you normalise third-party data to OCSF at the point of ingestion, you enable:
Cross-source correlation: Query across AWS CloudTrail and your EDR platform using unified field names and data structures. Tool interoperability: Any OCSF-aware security tool can consume your Security Lake data without custom parsers. Future-proof architecture: Adding new sources or switching vendors becomes dramatically simpler when everything speaks the same language. Query efficiency: Normalised schemas enable query optimisations that reduce costs and improve performance.
The investment in proper normalisation at ingestion pays dividends throughout the data lifecycle. Conversely, storing raw logs and normalising at query time creates ongoing operational friction and inflated costs.
Integration Architecture Patterns
There are three primary architectural patterns for custom source integration, each suited to different scenarios:
Pattern 1: Direct API Push
In this pattern, the source system pushes data directly to Security Lake using AWS APIs. This works well when:
The source system supports webhook or API-based event forwarding. Event volumes are moderate and predictable. You require near real-time data ingestion. The source provides structured data that can be transformed to OCSF with reasonable effort.
Implementation involves configuring the source to send events to an AWS Lambda function or API Gateway endpoint, which transforms the data to OCSF format and writes to the Security Lake S3 bucket with appropriate partitioning.
Pattern 2: Agent-Based Collection
For sources that don’t support push mechanisms, agent-based collection provides a reliable alternative. Deploy collection agents that:
Poll source systems on configurable intervals. Buffer events locally to handle network interruptions. Transform data to OCSF before transmission. Support backpressure and rate limiting to prevent overwhelming sources.
This pattern suits on-premises infrastructure, legacy systems, or sources with unreliable connectivity. The trade-off is additional infrastructure to manage and potential for higher latency.
Pattern 3: Streaming Pipeline
For high-volume sources or scenarios requiring complex enrichment, a streaming pipeline architecture provides the necessary throughput and flexibility. This typically involves:
Initial ingestion to a Kinesis Data Stream or Kafka cluster. Stream processing using Kinesis Data Analytics, AWS Lambda, or Apache Flink for transformation and enrichment. Writing normalised events to Security Lake with partitioning optimised for query patterns.
This pattern handles the highest event volumes and supports sophisticated processing logic, but requires more architectural complexity and operational expertise.
Step-by-Step Implementation Guide
Let’s walk through integrating a third-party EDR platform into Security Lake using the direct API push pattern, as it represents the most common integration scenario.
Step 1: Define Your Data Requirements
Before writing any code, clearly define what data you need and how it maps to OCSF. For an EDR platform, you might prioritise:
Process execution events (OCSF class 1007). Network connection events (OCSF class 4001). File activity events (OCSF class 4010). Authentication events (OCSF class 3002).
Document the source data schema and create explicit mappings to OCSF fields. This mapping document becomes your integration contract and ensures consistency across your team.
Step 2: Set Up the Ingestion Infrastructure
Create an API Gateway endpoint to receive events from your EDR platform. Configure a Lambda function as the backend processor. Ensure the Lambda execution role has permissions to write to your Security Lake S3 bucket with the prefix format: aws/REGION/ACCOUNTID/ext/SOURCE_NAME/.
Implement proper error handling, logging to CloudWatch, and dead letter queues for events that fail processing. Production implementations should include retry logic and circuit breakers to handle transient failures gracefully.
Step 3: Implement OCSF Transformation
The transformation logic is where integration quality is determined. For each event type, implement a transformation function that:
Extracts relevant fields from the source event. Maps them to the appropriate OCSF class and attributes. Enriches with contextual information such as asset criticality or threat intelligence. Validates the resulting OCSF event against the schema. Handles missing or malformed data gracefully with appropriate defaults.
Avoid the temptation to include every field from the source. Focus on fields that provide investigative value or support specific detection use cases. Unnecessary fields inflate storage costs and query complexity without adding value.
Step 4: Implement Partitioning Strategy
Security Lake uses Hive-style partitioning with the structure: region=REGION/accountId=ACCOUNTID/eventDay=YYYYMMDD/. When writing events, ensure your Lambda function:
Calculates the correct partition based on the event timestamp (not ingestion time). Creates partition directories if they don’t exist. Writes events in Parquet format with appropriate compression. Batches events when possible to reduce S3 API costs and improve write efficiency.
Proper partitioning is critical for query performance and cost optimisation. Queries that filter by date can skip entire partitions, dramatically reducing the data scanned and associated costs.
Step 5: Configure the Source System
With the ingestion infrastructure in place, configure your EDR platform to forward events to your API Gateway endpoint. Key considerations include:
Event filtering: Only forward events you’ve decided to ingest to avoid processing overhead. Batch size: Larger batches improve efficiency but increase retry complexity. Authentication: Implement API key validation or mutual TLS to prevent unauthorised submissions. Rate limiting: Protect your ingestion endpoint from overwhelming your processing capacity.
Start with a subset of endpoints or event types to validate the integration before expanding to full production scale.
Step 6: Validate Data Quality
Before declaring the integration complete, thoroughly validate data quality:
Schema compliance: Verify that all required OCSF fields are populated correctly. Field mapping accuracy: Spot-check that source fields map to the intended OCSF attributes. Timestamp handling: Ensure event times reflect when events occurred, not when they were ingested. Completeness: Confirm that event volumes match expectations and no data is being silently dropped.
Query the integrated data in Security Lake using Athena to verify that the partitioning is working correctly and that queries return expected results. This validation step prevents discovering data quality issues weeks later when they’ve undermined detection logic.
Step 7: Implement Monitoring and Alerting
Production integrations require observability to detect and resolve issues quickly:
Monitor Lambda execution metrics including invocation count, error rate, duration, and throttles. Track S3 write operations to detect ingestion interruptions. Alert on data freshness to identify when events stop flowing. Monitor transformation error rates and review failed events regularly.
Include integration health in your operational dashboards alongside native AWS source metrics. Custom integrations shouldn’t be second-class citizens in your observability strategy.
Advanced Considerations
Data Enrichment at Ingestion
Consider enriching events during transformation with additional context:
Asset criticality: Tag events from critical systems to prioritise investigations. Threat intelligence: Enrich with indicators of compromise to accelerate detection. User context: Add department, role, or other attributes to support access analytics. Geolocation: Enrich IP addresses with geographic information for anomaly detection.
Enrichment at ingestion is more efficient than enriching at query time, particularly when the same enrichment would be repeated across many queries.
Handling Schema Evolution
Source systems evolve, adding new fields or changing data formats. Plan for schema evolution by:
Versioning your transformation logic to support multiple source schema versions. Implementing schema validation to detect unexpected changes. Designing transformations to degrade gracefully when optional fields are missing. Maintaining backwards compatibility when updating OCSF mappings.
A robust integration anticipates change rather than assuming static schemas.
Multi-Region Considerations
For global deployments, decide whether to:
Replicate all data to a primary region for centralised analysis. Maintain regional Security Lake instances with federated querying. Hybrid approach with regional storage and selected replication based on data sensitivity or compliance requirements.
Data sovereignty requirements often dictate architecture, particularly in regulated industries or when operating in jurisdictions with strict data residency rules.
Cost Optimisation
Custom source integration introduces costs that require active management:
Lambda invocation costs: Batch events when possible to reduce function executions. S3 storage costs: Implement lifecycle policies to transition older data to cheaper storage classes. S3 API costs: Minimise PUT operations through batching and efficient partitioning. Data transfer costs: For on-premises sources, compression reduces network transfer costs.
Monitor integration costs alongside security value to ensure positive ROI. Not every event type justifies the cost of ingestion, storage, and analysis.
Common Integration Pitfalls to Avoid
Having implemented numerous custom integrations, we’ve observed recurring mistakes:
Incomplete OCSF mapping: Failing to populate required fields creates downstream query problems. Ignoring timezone handling: Inconsistent timezone treatment causes temporal correlation failures. Over-collecting data: Ingesting high-volume, low-value events inflates costs without improving security posture. Inadequate error handling: Silent failures allow data gaps to persist undetected. Neglecting partition pruning: Poor partitioning design makes queries expensive and slow.
Learning from others’ mistakes is cheaper than making them yourself.
Real-World Example: EDR Integration
Consider a practical example integrating CrowdStrike Falcon with Security Lake. Falcon provides rich endpoint telemetry but uses its own event schema. The integration approach:
Leverage Falcon’s Event Streams API to receive events via HTTP push. Deploy a Lambda function that transforms Falcon’s DetectionSummaryEvent to OCSF Detection Finding (class 2004). Enrich with host criticality from your CMDB during transformation. Partition by event date and write to Security Lake in OCSF-compliant Parquet format.
This integration enables querying Falcon detections alongside AWS GuardDuty findings and CloudTrail events using unified OCSF fields, dramatically simplifying cross-platform investigations.
Moving Beyond Native Sources
Custom source integration transforms Security Lake from an AWS-centric data repository into a comprehensive security data platform. The investment in proper integration architecture, OCSF normalisation, and operational excellence pays dividends through:
Comprehensive visibility across your entire security stack, not just AWS services. Simplified vendor evaluation and switching when data normalisation eliminates lock-in. Improved detection accuracy through cross-platform correlation. Reduced operational overhead compared to maintaining multiple data silos.
The organisations gaining maximum value from Security Lake are those that view it as a unifying data layer for all security telemetry, regardless of source. With proper integration practices, Security Lake becomes the foundation for truly data-centric security operations.
Ready to Extend Your Security Lake?
At HOOP Cyber, we specialise in custom source integrations that maintain data quality whilst optimising costs. Our team of ex-Splunk engineers and AWS security specialists has integrated hundreds of third-party sources into Security Lake for clients across industries.
Whether you’re integrating a single critical source or building a comprehensive multi-source architecture, we can help design, implement, and operate integrations that deliver long-term value.
Contact HOOP Cyber at to discuss your custom integration requirements and learn how we can accelerate your Security Lake adoption
