From SIEM Cost Overload to Federated Security Data Mesh: A Real-World Amazon Security Lake Success Story
How a Global Enterprise Solved Multi-Region Compliance and Splunk Cost Challenges with HOOP Lake Architecture and Query Federated Search
When Cloud Growth Outpaces Your SIEM
Security teams today face a fundamental paradox: the more your infrastructure grows, the more security telemetry you generate, and the more your SIEM costs spiral out of control. For one global enterprise operating across 16 AWS regions, this challenge became critical.
Operating in a heavily regulated industry, they needed to keep security data in local regions to meet compliance requirements. Their SOC relied on Splunk Cloud as their primary platform, but it had limited visibility into their vast, distributed AWS infrastructure. The cost of ingesting the massive volumes of CloudTrail, Route 53, VPC Flow Logs, AWS WAF, and EKS audit logs into Splunk wasn’t just expensive, it was unsustainable.
This is where the power of a data-centric approach to security operations comes into play.
The HOOP Lake Approach: Cyber Security as a Data Problem
At HOOP Cyber, we’ve always maintained that cyber security is fundamentally a data problem. The traditional approach of centralising everything into expensive SIEMs made sense in simpler times, but modern cloud architectures demand a different strategy.
This enterprise adopted Amazon Security Lake to collect and store their security telemetry across all 16 regions in OCSF (Open Cybersecurity Schema Framework) format, the same standard that powers HOOP Lake. But collection alone doesn’t solve the problem. The real challenge was making that data accessible, searchable, and actionable for their security analysts without breaking the bank.
Their requirements were clear:
- Compliance-first: Data must stay in local international regions
- Analyst-friendly: Security teams needed interactive console access, not manual SQL queries
- Federated operations: Single searches across authorised regions and sources
- Detection automation: Apply existing detection logic to distributed data
- Cost predictability: Escape the unpredictable data volume licensing trap
The Solution: HOOP Lake Architecture + Query Federated Search
The winning architecture leveraged the core principles of the HOOP Lake methodology combined with Query’s federated search capabilities:
Stream: Normalising to OCSF at Point of Ingestion
The customer was already collecting logs from their AWS services into Security Lake in OCSF format. This normalisation at the point of ingestion is crucial, it’s the foundation that makes federated search possible. HOOP Lake’s streaming approach automatically receives log information from data sources and transforms it into optimised, enriched OCSF format, creating a unified data model across disparate sources.
Store: Efficient, Distributed Data Lakes
Rather than centralising everything into Splunk, the data remained distributed across 16 Security Lake instances in compressed Parquet format. This approach delivered massive cost savings whilst maintaining compliance requirements. The HOOP Lake store principle focuses on keeping data in a high-performance format with automatic compression, leveraging Parquet tables for optimal performance.
Search: Federated Access Across the Mesh
Here’s where Query’s federated search capability integrated seamlessly with the HOOP Lake architecture. Query enabled analysts to search across all nine sources (three data types across three POC regions) from within their familiar Splunk console using the | queryai command. The searches ran in parallel across distributed sources, with Query automatically breaking down, distributing, normalising, and collating results.
Enrich: Context at Point of Query
The OCSF normalisation meant that an IP address appearing in CloudTrail, Route 53, and VPC Flow Logs could be searched as a unified entity. Query’s natural language to optimised query translation aligned perfectly with HOOP Lake’s orchestration principles, where data flows are manipulated based on unique requirements without rewriting code.
Comply: Real-Time Visibility Across Regions
With data properly normalised and federated search in place, the SOC gained real-time visibility across their entire 16-region estate. Dashboards and detections could run directly against distributed data, maintaining the compliance posture whilst dramatically improving operational efficiency.
The Results: Extended Visibility Without Breaking the Budget
The POC validation demonstrated compelling outcomes that align with HOOP Lake’s core principles:
Cost Transformation
- Avoided Splunk’s unpredictable data volume licensing
- Leveraged low-cost Security Lake storage + pay-per-query Athena
- Query licensing based on connector count, not data volume
- Massive savings on data ingestion and indexing
Operational Excellence
- Analysts maintained familiar Splunk workflows with minimal changes
- Extended visibility to massive Security Lake datasets across 8 event classes
- Federated detections running on distributed data without centralisation
- Faster investigation and hunting cycles with unified entity searches
Future-Proof Architecture
- Scalable to additional AWS regions and data sources
- Path to gradually transition from ingest-heavy SIEM to federated mesh
- Foundation for onboarding third-party and custom sources
- Built on open standards (OCSF) preventing vendor lock-in
Why This Architecture Works: The HOOP Lake Difference
This success story demonstrates several principles that are core to the HOOP Lake methodology:
- Data-Centric Foundation: By normalising to OCSF at point of ingestion and storing in efficient formats, the architecture created a unified data fabric that multiple tools could leverage.
- Federated Operations: Rather than centralising everything, the architecture kept data distributed whilst making it accessible, meeting both compliance and cost requirements.
- Tool Flexibility: Analysts could continue using Splunk whilst Query provided federated access to Security Lake. This pragmatic approach meant no disruptive tool replacements.
- Standards-Based: Commitment to OCSF ensured interoperability and prevented vendor lock-in, giving the organisation freedom to evolve their architecture over time.
- Cost Optimisation: By separating storage from compute and using federated search, the organisation escaped the “ingest tax” whilst actually expanding visibility.
The Broader Lesson: Rethinking Security Data Architecture
This case study isn’t just about one enterprise solving their Splunk cost problem. It represents a fundamental shift in how security operations should think about data architecture in cloud-native environments:
Old Model: Centralise everything → Index into expensive SIEM → Pay exponentially as data grows
New Model: Normalise at source → Store in efficient lakes → Federate search across sources → Pay predictably for compute
The HOOP Lake approach, powered by Amazon Security Lake and enabled by partners like Query, represents this new paradigm. It’s about building a security data mesh that scales with your cloud infrastructure without scaling your costs proportionally.
Ready to Transform Your Security Operations?
If you’re facing similar challenges (multi-region compliance requirements, exploding SIEM costs, or limited visibility into cloud infrastructure), the HOOP Lake approach combined with Query federated search offers a proven path forward.
HOOP Cyber specialises in:
- Amazon Security Lake architecture and implementation
- OCSF data normalisation and enrichment
- SecOps architecture modernisation
- SIEM optimisation and cost reduction
- Data source mapping and integration
Whether you’re looking to extend your existing SIEM with federated search capabilities, migrate to a data mesh architecture, or optimise your current security data operations, HOOP Cyber brings deep expertise in building scalable, cost-effective security data architectures on AWS.
Learn more about:
- HOOP Lake Services
- Amazon Security Lake Implementation
- SecOps Architecture Modernisation
- Cost Optimisation Services
Contact HOOP Cyber to discuss how we can help transform your security operations with a data-centric approach.
