Be a part of the occasion trusted by enterprise leaders for almost 20 years. VB Remodel brings collectively the folks constructing actual enterprise AI technique. Study extra
Gaining visibility — and, in the end, insights — into enterprise cloud property is rising ever more difficult.
Cloud estates are sprawling and fragmented, and stock capabilities in present instruments might be slim and unintuitive, separating parts like price and safety knowledge into disconnected platforms with restricted flexibility.
Cloud governance firm CloudQuery is positioning itself to deal with this downside by centralizing cloud property, safety metadata and price in a single place, and making it accessible by way of straightforward, built-in SQL queries and stories. The corporate is taking a developer-first method to cloud governance, pulling knowledge from 60-plus sources — together with AWS, GCP, Azure, Okta and Wiz — right into a single, queryable knowledge warehouse.
The corporate is now saying a $15 million funding spherical led by Partech to additional scale its method to cloud visibility.
“The biggest challenge with existing tools is that they’re siloed — one for security, one for cost, one for asset inventory — making it hard to get a unified view across domains,” CQ founder Yevgeny Pats informed VentureBeat. “Even simple questions like ‘What EBS volume is attached to an EC2 that is turned off? are hard to answer without stitching together multiple tools.”
CloudQuery below the hood
CloudQuery makes use of two key applied sciences below the hood: Information warehouse and open-source database ClickHouse and the Apache Arrow framework for growing knowledge analytics functions.
This high-performance plugin structure in-built Go connects on to APIs like AWS, Azure, Google Cloud Platform (GCP) and plenty of different platforms pulling in configuration, safety, and price metadata. The platform repeatedly syncs knowledge from dozens of cloud suppliers and companies right into a normalized, centralized asset stock.
“We place a strong emphasis on data accuracy and freshness, syncing at high frequency to ensure teams are working with the most reliable, up-to-date information,” stated Pats.
That knowledge, he defined, is structured relationally to energy CloudQuery’s SQL engine and built-in stories, in order that groups can have full flexibility with out counting on black-box instruments.
The corporate additionally “selectively” makes use of massive language fashions (LLMs) for pure language querying, SQL era and proposals, “but always on top of a foundation of accurate, transparent data,” stated Pats. He identified that as a result of AI understands SQL properly, instruments like Claude and OpenAI can create custom-made stories and evaluation in plain English.
Taking a developer-first method is essential, stated Pats, as a result of builders are in the end those constructing, working and securing at present’s cloud infrastructure. Nonetheless, many cloud visibility instruments had been constructed for top-down governance, not for the folks really within the trenches.
“When you put developers first, with accessible data, flexible APIs and native language like SQL, you empower them to move faster, catch issues earlier and build more securely,” he stated.
Clients are discovering methods to make use of CloudQuery past asset stock. “Many start with visibility, then quickly grow into use cases like compliance monitoring, security posture management, cost optimization, all from the same core platform,” stated Pats.
How Hexagon constructed a serverless knowledge lake for all its cloud shops
One enterprise already seeing outcomes is Hexagon. The software program firm’s cloud middle of excellence (CCoE) workforce had a objective to construct a completely serverless knowledge lake that might gather knowledge from all of its cloud accounts and retailer it in a single knowledge lake.
In addition they wished the power to question this knowledge utilizing SQL and visualize it with instruments they had been accustomed to (akin to AWS QuickSight), and discover the historical past of their cloud configuration over time.
The workforce constructed a serverless knowledge pipeline utilizing CloudQuery to gather knowledge from all accounts and retailer it in S3. AWS Glue then ingests knowledge into Glue DB in a format that Amazon Athena can question, which Athena then does, then visualises in QuickSight.
“Having a fully serverless solution was an important requirement,” Hexagon cloud governance and FinOps professional Peter Figueiredo and CloudQuery director of engineering Herman Schaaf wrote in a weblog submit. “This decision brought lots of benefits since there is no need for time-consuming updates and virtually zero maintenance.”
They did have to beat some challenges, notably with Amazon S3 help plugins. The CCoE workforce was one of many first to check out CloudQuery options within the S3 vacation spot and provided insights resulting in new options. These embody:
Parquet help: The CloudQuery file vacation spot initially solely supported CSV and JSON knowledge codecs. Errors in JSON interpretations led CloudQuery so as to add Parquet help.
Information partitioning: A CloudQuery file vacation spot plugin now permits partitioning on preliminary write (which beforehand wasn’t out there, leading to additional pointless steps).
Useful resource view for Athena: CloudQuery initially solely provided a sources view for AWS suitable with Postgres. However Athena didn’t help this, so CloudQuery added a operate that may retrieve a listing of all tables to construct or replace a sources view.
Figueiredo’s workforce used CloudQuery to switch AWS’s VPC IP deal with supervisor (IPAM) — which he known as costly and restricted in that it doesn’t cowl different cloud suppliers.
Finally, his workforce runs CloudQuery in ‘data lake’ mode utilizing “ultra cheap infrastructure” together with AWS S3, ECS, Glue, Athena and Lambda,” Figueiredo informed VentureBeat. This retains prices low and permits Hexagon to merge all its IP addresses throughout completely different cloud suppliers.
“We can quickly query any IP across the board and find who the owners are,” stated Figueiredo. “We are now able to collect all we need at a very low cost with near zero maintenance. This is the holy grail for our team.”
Day by day insights on enterprise use instances with VB Day by day
If you wish to impress your boss, VB Day by day has you coated. We provide the inside scoop on what corporations are doing with generative AI, from regulatory shifts to sensible deployments, so you possibly can share insights for max ROI.
An error occured.
