6 in 10 Enterprises Can't Find the Root Cause When Their AI Workloads Fail | Paul Appleby, Virtana

AI Infrastructure Challenges

0:00Six in ten enterprises cannot automatically identify the root cross across AI infrastructure domains when AI workloads fail. Why a root cause is so much harder in an AI factory than in traditional enterprise IT to discover? What you're saying, we're entering a phase where AI ROI will be judged, or AI will be judged less by model performance and more by operational efficiency, ergo cost. And business impact, you know, it might not just be a cost dynamic, the kind of gold rush to

0:35be in the AI investment cycle is overriding some of the good governance principles that companies

Introduction to Vertana

0:42have. We'll see the equilibrium come back. So let's start by having you introduce yourself to listeners. Thanks, Craig. Yeah, I'm Paul Appleby, the CEO of Vertana. And we're a company that exists really for one really important purpose, where, you know, companies across many industries, whether it's banking, telecommunications, healthcare, retail, airlines, whatever, are so reliant on their

1:15technology. In fact, you know, a lot of chief risk officers say that the single biggest point of catastrophic risk of failure for a business now is their technology and infrastructure. Vertana's in the world of trying to protect that. We live in this world called observability, and that's all about business resilience and operational efficiency. How do we make sure those services that are critical to your business and your customers stay performant and available? And even more relevant in this era of accelerated AI adoption, of course.

Vertana's Background

1:49Yeah. And how long have you been with Vertana? And what was your background before coming? I've been here now for a couple of years, inherited an amazing business that in the past, in fact, under its prior name of virtual instruments, was run by John Thompson, the former chairman of Microsoft, Microsoft. So the company's been around for some time and has an amazing history with its core technology. But I came to the company two years ago with a charter from the board to really lean into

2:26this world of, you know, the broad digitization of services and the scale-out adoption of AI. Prior to that, I've been working in technology for years, sometimes in startups, because I love, you know, that whole idea of the scale-up. But sometimes in much larger companies, you know, through phases of transformation and growth, like South Horse, where I was for a number of years, and also companies like Elasticsearch more recently as their president. So I've got a long background in

3:04enterprise software and enterprise technology and growth and scale of businesses.

Observability Platform

3:11Yeah. In Vertana, we were talking before we started recording, is an observability platform for large-scale operations or technology operations, not necessarily AI. I mean, it existed before the generative AI boom. Is that right? Yeah. I think it's a couple of really important things to say. I mean, this class of software,

3:41Craig's been around for a long time. For as long as, you know, technology's been supporting critical business services, there's been a need to monitor that infrastructure to make sure it stays available and performant. But interestingly, if you think about observability, what really is its purpose? Its purpose is to identify threats and risks in real time, you know, identify what the cause of those was and remedy those. So as a consequence, we've been building deep AI and ML capabilities.

4:15You know, for over a decade. So AI is not new to us. It's really part of our core reason for being, you know, streaming in all of that event data in real time and doing, you know, the mapping and correlation needed to identify, you know, risks and threats. What we've done more recently, of course, is lean in heavily to, you know, a lot of agentic capabilities, you know, to support automating a lot of these IT operations. But the other thing that we've done is recognize that as companies scale

4:50out the, you know, yes, the true industrialization of AI with massive AI data center investments that the market loves to call AI factories, is build full end-to-end observability for the AI fact. So what we've essentially done is taken an incredible legacy of observability and fully embrace this whole, you know, AI era to support the, you know, companies as they scale up their AI

AI Factory Reality Check

5:22investments. Yeah. And before we get to the study, you were saying that this is a layer that sits above the technology stock, even above the orchestration layer. Correct, Craig. And, you know, what I'd say is if you, if you think about any critical business service, let's call it a trading platform or internet banking is, you know, something that, that, you know, is core to any retail bank. You know, we all do most

5:52of our banking online these days. In fact, I do most of mine on my, on my mobile phone. And, you know, what supports our ability to interact with our bank is a really complex system, if you like, of, of, you know, data and applications and networks and compute and storage infrastructure. Some of it potentially in the cloud, some of it in legacy infrastructure, some on-prem and, and really what observability needs to be is something that views the whole system. Cause I like to think about

6:28it as a system that's incredibly, you know, in most cases, heterogeneous in most cases, um, you know, hybrid, um, which, which is the, the reality for most companies today. Um, and, and the ability to, you know, dynamically, um, observe that whole system and, and manage that system is, is where our class of software sits. And, um, uh, the, um, it, it's becoming increasingly critical, this

7:00observability as these systems scale, right. And, and they're scaling not only to keep up with, I mean, some are scaling to keep up with customer, uh, customer base, but some are scaling simply because the business is growing and becoming increasingly complex. I mean, you're not only looking at customer facing, uh, or public facing applications. You're, you're looking at any system,

7:30uh, that, that has AI in the stack. You're absolutely correct. So, um, you think of, of any system, you know, even for large retailers, some of their most important systems aren't just the customer facing systems, their website or, or their point of sale environment, but it's their whole distribution supply chain infrastructure as well. All of the back office functions of a retailer, um, the same with an airline, you know, obviously we experienced the airline through

8:02baggage handling and ticketing and, and, you know, all of those things, but there's all of the crew scheduling and flight scheduling and everything else that exists. And, you know, the thing that, you know, obviously makes that even more complex is this world of remote and hybrid, hybrid work. So supporting huge remote and hybrid workforces is, is key as well. So when we look at a large company, we're thinking about all of these critical systems that are key to the operation of the business. Um, and you know, you think about, you know, healthcare companies, you know, the way we

8:38experience healthcare today, you know, with things like, you know, digital medicine and, and remote diagnostics and, you know, remote visits and all the rest of it is wonderful as a consumer to be able to do that. But the medical practitioners are consuming all of that as well as they're able to access your electronic health records and collaborate with other practitioners. So it's, it's being able to support all of that infrastructure for big healthcare org or a big airline or a big bank. That's, that's

System Complexity

9:08where we sit. Right. And, and the study that you've done, uh, the AI factory reality check, I believe it's called, uh, you're, you're, you're finding that these AI factories to use that term, but, but these systems, uh, are growing faster than, uh, than, than there is ability to, to, to operate them or, or maintain

9:40them or. Yeah, I think that's, that's a great way of putting it, Craig. I think that, um, you know, the, I think would all agree that there, there's been a wild explosion of investment, um, in AI infrastructure and AI data centers and, and, you know, even above and beyond, you know, these AI platforms that we're all very familiar with, um, as we think more about large enterprises and, and governments adopting AI,

10:11um, that, the clear narrative is that, um, you know, most of this AI infrastructure or a lot of it is going to be built on-prem, um, you know, to support these, you know, huge AI use cases for, for these companies. And, you know, what, um, our study has found is that whilst companies are leaping into investing in AI data centers and, you know, as you said, AI factories, you know, which is a, a, a, a, a phrase that's, you know, abandoned about a lot by, you know, the likes of NVIDIA and, and Dell, um, the,

10:46these AI factory investments, um, we're not seeing the governance that needs to come with the kind of controls, uh, to optimize the, um, the resilience and efficiency of those data centers, um, you know, the investments in governance and controls aren't, uh, aren't running at the same pace as the investments in the infrastructure, which increase risk. And that's, that's what we're calling out with this study.

Governance and Controls

11:13It's like, Hey, checkpoint here, you know, huge opportunities in so many different industries for adopting AI at scale, make sure the governance and controls that you've got built into all of your, you know, legacy infrastructure get replicated in, you know, this AI, AI, um, data center world. Yeah. And I think one of the, of the, the, the top findings is that, uh, nearly, uh, well over half,

11:46I think six in 10 enterprises cannot automatically identify, uh, the root cross cause across AI infrastructure domains when AI workloads fail. Can you talk about that? Uh, why a root cause is so much harder in an AI factory than in traditional, uh, enterprise IT, uh, to discover. Craig, what I'd say

12:16is that root cause is hard to discover either way, but it's much harder in an AI, AI factory or a massive scale AI data center because of the complexity of that, um, system, if you like, from the data pipelines to the AI workloads and AI orchestration into, you know, the, the infrastructure layer of computing and network and storage, et cetera. Um, if there is either a slowdown, um, or a failure

12:53of, uh, an AI workload, the, the challenge is where did that occur? And, and it's really quite fascinating is that, you know, the vast bulk of, um, these failures don't have an identified brute cause. Now, the problem with that, of course, is if you don't know what causality is, you get to remediation and, um, that leads to, you know, a really, really high percentage of AI workloads actually failing. Now that doesn't make sense. If you're going to operate at scale with critical services,

13:28you can't have, you know, a really high percentage of AI workloads failing. Um, and, and that's just one element of it. The second element of it is, you know, incredibly powerful compute infrastructure and capability with these, these GPUs. But if AI workloads are getting throttled and GPUs are sitting idle, you've made these massive investments and you're not getting a return on the investment. So how do you make sure you maximizing utilization and throughput to get great ROI? And then the final

14:00thing is we're all aware of the massive energy consumption, um, and environmental implications of AI data centers. So if we're actually operating them efficiently, we can actually significantly reduce the, um, you know, the, the energy requirements and, and ultimately the footprint and impact that AI data centers have. So it's really at multiple levels that we need to be thinking about this whole system. And unfortunately right now, Craig, what, you know, what exists for many companies is,

14:31you know, bits of telemetry out of each level of the stack that they're trying to stitch together to identify, Hey, what's really going on and how do we actually better manage this environment? And that's not the answer. Yeah. Well, as a matter of that was going to be my question, what do people do without a platform like Vertana? Because this isn't a new problem. It may be increasing in its, uh, the risk, uh, as things scale, but what, what have people been doing or what are they doing as they scale?

15:06They can't just be holding their breath and hoping everything's okay. Um, well then they're not just holding their breath. You know, there, there's clearly data coming from, you know, certain elements of that, you know, AI factory stack. And what it's left to is that the IT professionals whose job is to run and operate the, the, the, these data centers to try and stitch that data together. Um, that's why when you, you, you have a service failure or service

15:37disruption, you end up with a group of people of, you know, historically it would have been physically in, in, uh, uh, you know, in a, in a knock, but now it's combination of people in the, in the knock and a bunch of people on, on a call like this, um, trying to work out what the hell is the problem and how do we fix it? And it's trying to stitch together all of this data. Now that's the reality for most companies today. In fact, you know, that they've got four or five, sometimes six

16:07different sources of data that they're somehow trying to weave together. And I think the, the clear message moving forward, as we think about critical services at scale in this AI world is that you need visibility into the full system. Um, you have to be capturing rich telemetry and doing that dynamic mapping and correlation in real time to be able to really truly govern these systems. And, you know, as we were chatting about earlier before, you know, the, this conversation

16:42started, um, it's then you can actually apply agentic capabilities because if you've got a full system view, you could then use agents to accelerate, um, getting to causality and then actually automate remediation. How does Vertana's platform work? I mean, do you have multiple models that are, uh, cross-referencing, uh, each other? Do you have, uh, sensors in different, uh, servers that are

17:16watching things? I mean, and, and then all of that is, is fused, uh, and, and analyzed. So yeah, can you just talk about the architecture of it all? Yeah. Yeah. The answer, the short answer is yes, yes, and yes. But let me talk a little bit more about what I mean there, Craig, because, um, to, to be able to do the kinds of things we're talking about of getting to true autonomous operations, you need to capture telemetry from the whole stack. So over, you know, a period of many years,

17:50we've developed deep integrations into every layer of the stack and both, you know, traditional infrastructure. And now of course, in this AI infrastructure, um, and, and we're capturing 20,000 different metrics, um, in a sub-second, um, time frame and correlating and mapping, uh, those, uh, metrics in near real time. Um, so it's, it's by capturing all of those metrics and correlating and

18:26mapping those and using our models to, to, you know, to see patterns that, that identify causality. And of course the patterns that recommend remediation, um, are really core to what the TANA does, but it really is a really important thing to understand. And one of the things I understood by having worked in other organizations in this observability sector is that you can't really apply agentic capabilities and expect to drive true autonomous outcomes unless you've got system-wide

19:02visibility and, and that kind of real time, you know, dynamic mapping and correlation, because it just doesn't work. Otherwise you've got some automation to a part of the stack as opposed to the full system. And that's real, that's really what, uh, you know, uh, Vatana does. And it, it is a really, really tough problem to solve, Craig, that ability to, you know, stream and ingest, you know, that, that volume of metrics and then analyze, correlate, understand, and recommend dynamically, um, is

19:37really core to, you know, what Vatana story is all about and why it's so powerful now in this AI era, um, because we can bring those core capabilities into these so-called AI factories, um, and play a really important part in helping govern them. Yeah. And the, uh, um, does, we also talk briefly about cybersecurity, whether this enhances, uh, your cyber attack monitoring, because this is, uh, LLM

20:13adjacent, right? You, uh, models, uh, to do some of the analysis, uh, are most of the anomalies detected things that you've seen over and over again? And so the, you, you, you, the system has a pretty good bead on, uh, on why that's, uh, you know, what, why that's happening or does it, uh, find new things that no one's seen? And, uh, and in those cases, is it capable of addressing them autonomously?

20:53Um, yeah, I mean, a lot, a lot to unpack there, Craig. Um, so let me, let me take them, um, piece piece by piece, you know, on the, um, the, the question of cyber, you know, we don't position ourselves, uh, as a, you know, cybersecurity, um, solution. We're primarily focused on the infrastructure. But the, the, the point that we were talking about earlier, of course, is that if you've got this ability to ingest all of these, ingest all of these events and analyze them,

21:25correlate them and identify anomalies, um, you know, and it's such a powerful engine, it can be applied for cybersecurity use cases. So in fact, although we don't position ourselves as a cybersecurity company, we've got a bunch of customers who've, you know, who've gone, wow, this data is so rich. Um, we can actually augment what we're doing in terms of our cyber capabilities by using the ability to detect anomalies. So, so that is certainly happened,

21:56but it's not really the core of our business today. Um, although it is a very powerful engine, um, as it relates to, you know, um, your other questions, um, yes, we do see, you know, because we have so much rich data and because, you know, a lot of these events follow similar patterns, we can identify those patterns and get to causality really quickly. Um, but the other fascinating thing is because we're capturing so much rich data, even when we're identifying,

22:27you know, new or newer anomalies, we're very, very quickly, uh, able to learn what those patterns are and then start recommending remediation. Um, even if it's, you know, and, and depending on a company's policies, um, you know, we can either automate that remediation or, you know, pass the, you know, the recommendation together with the supporting evidence around the recommendation to a human operator to then act on as well. So, you know, really depends on the, you know, the, the company,

23:00where the company's at on their AI and automation journey and, um, you know, you know, what policies they want to adopt associated with that. But what we found in going into a number of these, um, AI labs of some of the larger, um, you know, technology providers and, um, some of the larger customers is that we're identifying anomalies and usage patterns, um, that, that they didn't even understand about their technology themselves, um, because of, of our ability to do exactly what

23:33we've been talking about earlier. Yeah. Uh, well, I have so many questions, but, um, so who owns this?

Ownership and Decision Making

23:43I mean, this is not, uh, another alarm dashboard, uh, because most of the remediation is being done autonomously. But as you said, there are times when you have to, uh, pull a human into the loop. Uh, who is that? Is that in the IT department? And then the other question is, uh, your report shows a disconnect between executives and practitioners. Yeah. What does that say about how AI infrastructure

24:14decisions are being made? Uh, or the, the, the, the purchase of Vertana, for example, maybe the, the guys in the IT department are saying, we can't handle all of this. We need Vertana, but the, but the C-suite is skeptical. I mean, yeah. Can you talk about that? Craig, Craig, you're the master at asking multi-layered questions. So I'm going to, I'm going to try, I'm going to try and, um, and, and pass them

24:45all there. Um, you know, what I'd say, I'll tell you a story. I can't name the company for obvious reasons, but I visited with a very large company, uh, here in the U S recently. And I met with, um, the, the person tasked with this. And in this case, it was the senior vice president responsible for IT operations. So that kind of senior vice president of IT operations is really a critical role in companies. And, and the exec that I was chatting with had been with the company for many

25:18years, a couple of decades and said, when, uh, when they first joined the company and were in that role, they, they reported out to the, the CEO and, and the C-suite about what they're up to and creating rigidity and resilience once a year. And then it became once a quarter. And then it became once a month. And, and what they shared with me is that they report out once a week on where they're at or a year around providing resilience, rigidity, and operational efficiency to the CEO and the CEO's

25:52leadership team. And, and, and what that says, and the reason why I wanted to share that anecdote is that this role of the person who's responsible for ensuring the resilience and efficiency of, of IT infrastructure and the services it supports has now become a C-level, virtually a board level. Um, so, so, so it's that persona that, that we sell to and, and then, but, you know, beneath them, it's all, obviously all of the IT operators, the, you know, the, the, the specific role is called site

26:26reliability engineer, and their job is to ensure the reliability of, of that infrastructure. It's, it's that, that group that we sell to. And I think what I'd say to the other layer of your question is that the disconnect between the IT operators and the business, if you look at any business today, every board is saying, what's your, what's your AI strategy and how are we embracing AI to stay relevant and compete and all of those things. So this huge focus on, you know, getting in and investing

26:58in AI and, and being an AI forward company. So that pressure is coming from the business. Um, the, the, the, the IT operators, you know, these, uh, VPs and senior vice presidents of IT ops are sitting there going, but Hey, we need the governance and controls to make sure that, you know, we can, you know, protect the, the resilience of the business. So that's kind of the disconnect, the pressure coming on investment and, and investment at speed. And then that kind of,

27:28you know, IT and the professional IT executive turning around and going, Hey, you know, we're, we're increasing our risk profile here. And, and what are we going to do about that? Yeah. Um, the, um, token prices are falling. Uh, how, how should, uh, CEOs and CIOs think about GPU spend differently as AI moves from pilot to production? And, and I think I've mentioned to you,

28:00I've been seeing people warning that the drop in, in token costs, uh, is great, but it tends to, to then people deploy all these agents. The next thing you're, you're, uh, your cost is going through the roof, even though the token costs is lower because you're using more tokens. Spot on Craig. And, you know, as we were saying, you know, just as we were chatting beforehand,

28:31um, things are changing on almost a daily basis. So yeah, sure. So I can cost tokens going down and will continue to fall, you know, you know, for, for the foreseeable, the token consumption is going through the roof. Um, and you, it's really just even looking at, at labor arbitrage and that question of, you know, what is, you know, where, where is the work done most efficiently? There's been conversations over the, the, the last few days about, you know, is the labor arbitrage conversation

29:07headed in the right direction. And, and in fact, will AI be more efficient? And I think that is going to lead to some really important decisions that companies have to make about what use cases and business problems, um, we need to solve leveraging this kind of infrastructure, um, that are going to make a huge impact. And, you know, there, I can think of some great examples, you know, like accelerated drug discovery, reducing the time of bringing new medicines to market,

29:39making them far, far more highly targeted, um, and improving the efficacy of those is like, there's no argument about that, you know, using AI to, you know, better analyze, you know, medical imaging, um, using AI to better identify, you know, um, uh, fraud, um, you know, or criminal financial transactions and those sorts of things, you know, there are a lot of, I could go on and on of really powerful use cases for AI, but I think, you know, what companies are

30:11going to do, and I think they are doing right now is looking at why, where do we deploy AI that's going to have a meaningful impact on customers, customer experience and, and, you know, and drive increased ROI that that's going to be key. But I think that, um, you know, a lot of companies are rushing headlong into the investment right now without thinking through the implications for ROI. And there's a lot of studies being done on this, you know, the hundreds of billions of dollars

30:43that are being invested right now. And when we, when the curve between the investment and the revenues will actually cross and, and, um, based on the data I've seen, it is not in the near future, Craig.

AI ROI and Operational Efficiency

30:55Yeah. And, and, and so we're, we're entering a phase, what you're saying, we're entering a phase where, uh, AI ROI will be judged or AI will be judged less by model performance and more by, uh, operational efficiency ergo cost. Yeah. And, and, and business impact, you know, it might not just be a cost dynamic. It could be, you know, a revenue dynamic, or it could be a, you know, customer experience and retention dynamic,

31:28but there'll have to be measurable impacts that's commensurate with the level of investment that we're making. Um, and as I've said, you know, there are so many use cases where AI is just the, the only answer. Um, but it's, it's about, you know, um, being really smart about how and where we deploy AI, um, building the right governance and controls around those, uh, investments and, um, you know,

31:58uh, continuing to scale thoughtfully. Yeah. Uh, uh, and does Vertana help in tracking ROI metrics? Because the report says, I think nearly a third say they need clear metrics, ROI metrics before confidently scaling AI further. So, yeah. So what, what metrics do you think matter most and, and how does Vertana handle that? Yeah, Craig, great question. Um, you know,

32:33what I'd say is that if, if you think about any large scale IT organization, um, they're in a pretty tricky position, not just as it relates to AI investments, but just, you know, IT investments in general, um, because they're being asked to do a couple of things, support more and more complex services, AI being one of those things with more and more complex infrastructure, um, with incredible resilience, but they're also being asked to do that efficiently. So, you know, I haven't met a CIO

33:07yet that hasn't told me that they've got pressure on costs and operating costs. So it's like, there's this duality that's kind of, it's competing, um, you know, priorities. And, um, so if you're going to play a role in supporting enterprise resilience, you got to play a role in, in making sure that you help customers use their infrastructure efficiently. So, um, that whole area of understanding and managing control for a cost for both, um, on-prem or physical infrastructure, as well as cloud

33:43infrastructure is a core part of our solution. Um, the only thing that we don't include is the business metrics because we're not bankers. We don't run insurance businesses or whatever, but we can provide all of the, the cost analysis and metrics that then get combined with the business metrics to determine, um, you know, uh, return on investment and, and payback and, and the like. Yeah. The study found though, that, uh, uh, a lot of this governance work is being deferred

34:16just as complexity is increasing. So, uh, uh, why is that? And, and how do you see that impacting, uh, enterprises? Yeah, it's, it's interesting. And I think that what's happening Craig is the, is the, the kind of gold rush to be, to be in, you know, in the AI investment cycle is overriding some of the good governance principles that, that companies have. Um,

34:48I think that we'll see the equilibrium come back. And in fact, that's why we published the study to show that, you know, there is a delta between what, you know, the business executives think and what it operators think there is a huge failure rate for AI workloads. Um, and that, you know, we didn't publish the, the findings to be naysayers around AI because we're not, but what we are saying is, Hey, as we, as we make these investments, we got to catch up with the governance. I think we'll

35:19see that happen quickly, Craig. Um, you know, uh, especially as people see the data and are confronted with the data, um, and the reality of, you know, putting governance and controls in place, particularly as it relates to AI is not only in the public interest, but it's in the business interest as well. Uh, you've just extended, uh, AI factory observability to Dell AI factory and AWS, uh, bedrock and Nutanix.

35:53Uh, what does that say about where observability needs to live on-prem at the model layer? Uh, yeah. Or, yeah. Um, you know, I, I think, um, there, there, again, a couple of elements to that, Craig. The first thing that I'd say is that what we're seeing in terms of the, you know, AI data centers that are getting deployed is that they, they are leveraging as companies architect these solutions,

36:26they're leveraging in, you know, what they believe is the best solution for every layer of that stack. So you end up with this heterogeneous environment of different providers providing, you know, components, if you like, of the AI factory that come together. So deepening our relationships with the likes of AWS and with Dell and with Nutanix and many, many others in video, um, you know, deep integration into the video stack and, and their GPUs, of course.

36:58Um, it, it's all about ensuring that we're capturing the richest telemetry. So it is really, I think the, the, the first thing to note is that heterogeneous nature of these, these AI data centers pulling on the, the best technology for each layer of the stack. And then for observability companies, it's our job to capture all of that telemetry and weave that together for our customers. Um, I think as it relates to, you know, on-prem or in the cloud, I think, you know,

37:31like we're seeing with, uh, you know, a lot of enterprise technology, we're going to see a hybrid world. Um, you know, the, the, the world of hybrid is the reality for just about every large company on the planet. Um, certain workloads and data, um, will remain on-prem and, and that's not going to change. Certain workloads and data will be in the cloud and, and it will be a kind of a dynamic model, you know, model where, where we'll see, you know, sometimes, you know, workloads and data

38:03shift depending on some kind of price arbitrage or, or, you know, speed to outcome or, you know, data protection and privacy issues. Um, so what's the big responsibility for organizations like ours is to make sure that we can, you know, manage that complex hybrid world and take that complexity away for our customers. And, and, you know, hence the investments in, you know, clearly significant on-prem infrastructure capability, as well as partnering with the hyperscalers and, and Neo clouds,

38:37um, to ensure that we're supporting customers wherever they need to be. Yeah. Uh, but, but the platform resides, uh, with the, the, the enterprise or is it, is it, are there instances of it, uh, in every, uh, uh, place that you have data flowing? You know, the one thing that we've also tried to do is ensure that our technology can be deployed anywhere. So our, our, our technologies are deployed both on-prem and

39:12in the cloud. So there are instances on-prem, there are instances in the cloud. And in fact, um, with a number of providers, um, who are providing, you know, as a service infrastructure, um, like Hitachi, um, our platform's embedded in their platform. So, um, you know, uh, uh, the, the interesting thing is, um, Vatana shows up in, in all sorts of different places, um, you know, really depending on, you know, how you're consuming the service and, and the whole idea

39:43and the whole philosophy around that was, you know, providing ubiquitous access and not going, hey, we are purely going to reside in the cloud. And that is that, that is our, our vision and strategy because that's not where our customers are. Our customers leveraging the public clouds, they're leveraging service providers, managed service providers and consuming services there. And they've got their own on-prem infrastructure that they're managing themselves. We need to be

40:14available across all of those environments. And so we've made a choice to, to do that. And it's ended up being the right choice because it allows us to be wherever our customers need us to be. Yeah. And this is a SaaS offering. I mean, it's not a, uh, yeah, single purchase and then there. Yeah. Yeah. No, where, you know, where, where, um, you know, we, we, the great thing about being, uh, you know, a subscription software company, Craig, is that, that you have to continually earn the right

40:46to do business with your customers. And that's, that's what I've tried to encourage my team is that that philosophy is, is that we have to continue to earn the right. And, you know, the amazing thing for us and that I'm incredibly grateful to all of our customers for is that almost every single one of our customers grows year on year. Um, and they're not growing year on year for any other reason than that. They're growing and scaling their usage of us across their infrastructure, because we're solving big, big, big, important problems for them. I want to ask as kind of an off the topic question,

41:21if I may, do you, you've got me worried now, Craig? No, no. It's, uh, you know, with, with the, the economics of, um, deploying AI changing, and there's a lot of work on pushing inference out to the edge. And there's a lot of work on, uh, you know, uh, making, uh, it's more power efficient and, uh, cheaper. Are we over building data centers? Do you think? I mean, that would argue to me that you

41:57can do more with less. Um, wow, that's a big question. How, how, how long do you think we've got to answer that one, Craig, in, in one minute? Um, you know, I think that, um, that you,

42:16all of those things are true. If we, we think about, you know, the promises of IOT, I mean, we, we stopped talking about the internet of things, but if you talk about, you know, city scale automation, you know, large scale adoption of autonomous vehicles, you know, massive scale manufacturing, we talked about physical AI earlier and the role of AI and manufacturing processes. Um, you know, a lot of this has to occur out on the edge. Um, there, there is no doubt about that.

42:48Um, but you know, what I would say is that, that that's going to be additive. I think that the, the physical build out of AI infrastructure is going to be absolutely key, particularly as companies and government adopt AI, they're going to be building out more and more their own AI data centers, AI factories. And, and you only have to listen to Michael Dell talk about the, the explosive growth that Dell is saying, um, in the adoption of their AI factories to realize that companies are doing

43:22that. So I think they'll happen in parallel Craig, and maybe we'll see ultimately the promise of, you know, large scale, you know, autonomous, um, vehicle adoption. We'll see the, the advent of, of truly distributed power generation and, and, and smart utilization of, of, you know, uh, energy in the home and all, all of the things that have been promised for a long time, I think, you know, could potentially be deployed with AI out at the edge, but, um, that's,

43:54I'm sure a much longer conversation. Okay. Okay. Is there anything I didn't cover that you want listeners to hear? Um, no, I think Craig, we've, we've, we've touched, we've touched on all of it. It's been a great conversation and it's fun chatting about what's going on out there in the world and kind of, you know, debunking some of the, the myths around AI, but also raising consciousness and awareness around the risks. And I think it was a really great conversation. Yeah.