Earley AI Podcast – Episode 57: AI Observability and Data Readiness with Camden Swita

Guest: Camden Swita, Head of AI and ML Innovation at New Relic

Host: Seth Earley, CEO at Earley Information Science

Published on: September 6, 2024 

In this episode of the Earley AI Podcast Camden Swita, the Head of AI and ML Innovation at New Relic, joins us for an insightful discussion on the transformative role of AI in modern technology. With a rich background spanning journalism, product management, and cutting-edge AI/ML initiatives, Camden brings a unique perspective on leveraging artificial intelligence to enhance both professional and creative workflows. 

Camden joins our hosts, Seth Earley and Chris Featherstone, as they navigating the complexities of data readiness and AI operations

Key takeaways:

• Large language models require substantial additional context to perform effectively—they cannot operate as black boxes expecting accurate multi-step reasoning right out of the box.
  
• AI observability demands visibility into the complete trace of agentic workflows from input through routing, tool use, and response to identify misunderstandings and errors.
• Modular RAG breaks down monolithic vector databases into componentized approaches, allowing engineers to improve specific context areas rather than entire systems.
• Organizations must conduct honest assessments of their data infrastructure with qualified data scientists before investing significantly in AI automation initiatives.
• Natural language APIs between AI assistants simplify integration across platforms but introduce potential for error compared to traditional structured APIs.
• Context remains king in AI implementations—even simple prompts require implicit assumptions about system state, user identity, and environmental conditions.
• Starting with narrow, high-value use cases allows teams to solve specific context problems incrementally rather than attempting comprehensive automation immediately.

Insightful Quotes:

"Context is still really king. You still really have to think hard about how you're going to organize and serve up that additional information and context to the assistant to make sure that it's got the right information to make the right decisions." - Camden Swita

"At the end of the day, this is still very much a data game. In order for these assistants to be successful, you oftentimes still have to make sure that they're going to have access to that specific context you need them to have access to." - Camden Swita

"Get an honest and expert opinion on whether or not you're even set up at a data level to really take advantage of a lot of the things that these agents can do." - Camden Swita

Tune in to discover how organizations can move beyond AI hype to build robust, observable AI systems grounded in solid data infrastructure and thoughtful context management.

Links:
LinkedIn: https://www.linkedin.com/in/camden-swita-54a41aa/

Website: https://newrelic.com

Ways to Tune In:
Earley AI Podcast: https://www.earley.com/earley-ai-podcast-home
Apple Podcast: https://podcasts.apple.com/podcast/id1586654770
Spotify: https://open.spotify.com/show/5nkcZvVYjHHj6wtBABqLbE?si=73cd5d5fc89f4781
iHeart Radio: https://www.iheart.com/podcast/269-earley-ai-podcast-87108370/
Stitcher: https://www.stitcher.com/show/earley-ai-podcast
Amazon Music: https://music.amazon.com/podcasts/18524b67-09cf-433f-82db-07b6213ad3ba/earley-ai-podcast
Buzzsprout: https://earleyai.buzzsprout.com/ 

Podcast Transcript: Monitoring AI Complexity, Context Management, and Data Infrastructure Requirements

Transcript introduction

This transcript captures a conversation between Seth Earley and Camden Swita about moving beyond magical thinking around AI systems, exploring the critical importance of observability in complex agentic workflows, the evolution of retrieval augmented generation architectures, and why organizations must honestly assess their data infrastructure before embarking on ambitious AI automation initiatives.

Transcript

Seth Earley: Good afternoon, good evening, good morning, depending upon your time zone. Welcome to the Early AI Podcast. My name is Seth Early. We're without Chris Featherstone today, but I'm really excited to introduce our guest for today to discuss the critical importance of agentic oversight and AI mechanisms and processes to monitor complex AI stacks and complex technology stacks. The significance of course, organizing, curating data sources and again being able to make sure that we can do troubleshooting, traceability, monitoring of AI systems. It's something that's fairly overlooked and as our systems get more and more complex, it gets more difficult to, to actually define exactly where problems might be. Our guest today is an expert who seamlessly bridged the gap between journalism and technology. He carved out a niche niche in the AI space. He currently is the head of AI and ML innovation at New Relic and a company that is renowned for observability and dedicated to empowering organizations by monitor AI enabled systems. With rich background in product management and a key player in various technology roles, he's been instrumental in pushing New Relics capabilities forward. Camden Sweeta, welcome to the show.

Camden Swita: So happy to be here. It's quite the intro. Looking forward to this conversation.

Seth Earley: Great, great. So a lot of times we like to start with kind of misconceptions in the industry and when it comes to AI environments and especially the work that you do within monitoring, you know, tell me what, what, what do you run it up against in terms of lack of understanding, misunderstanding myths, you know, think, what do you, how do you have to educate the executives and the customers that you talk to and what are the things that they don't necessarily understand? When you, when you walk in the door, your organization walks in the door, what are you seeing in the industry?

Camden Swita: Sure, sure. Yeah, yeah, makes sense. I think there's still a lot of magical thinking going on around large language models, generative AI assistants in general. You know, there, I think there's still the perception that, you know, especially the foundational models are sort of black boxes that are capable of doing multi step reasoning with a high degree of accuracy right out the box and that they don't need any additional context to complete these tasks or that, you know, and things of that nature. So I think the thing that I still try to emphasize in any of these conversations with, with executives or other people who are interested in investing in especially agentic automation is that context is still really king. You still really have to think hard about how you're going to organize and serve up that additional information and context to the assistant to make sure that it's got the right information to make the right decisions and give whoever the end user is the right insights.

Seth Earley: Yeah, and I think you're right in that people still think, oh, you just plug this in. In many cases they're starting to learn the hard way that no, you don't plug it in. You do need that data foundation. And then building that data foundation requires a lot of, a lot of investment. But more frequently you can use large language models and agentic approaches to actually fix the data. That's some of the experience I've had. Have you seen organizations doing that actually using the models themselves, say programmatic approaches or templated prompting fixed data?

Camden Swita: Oh, sure, yeah. I mean, everything from, you know, there's, there's a few different ways in which, you know, the, the data may need to be fixed. Right. You can use the large language models, of course, to synthesize additional contexts or alignment data for other models. So in other words, you want model A to be better at writing queries in this specific query language. Well, maybe you'll use some other model to write a bunch of examples of queries in there and, and natural language examples. You also see it, of course, with prompt adjustment after the user enters something. You see this with some of the foundational language models, chat platforms, especially some of the text to image generators, where they'll take the user prompt and the model will sort of rewrite it in a way that's gonna be a little bit more successful for the text to image writer. And then, you know, you can even use these models as sort of judge models. Right. So oftentimes what you'll see is you'll see a customer using a ling, a large language model that for whatever and they've determined is a good enough quality and kind of economical enough for them to use in production. And maybe they'll use a more expensive and better language model as a judge LLM that will sort of look at at least some subset of the responses that that first large language model is giving and assess whether or not it thinks the answer is good or not or meets the criteria that they've set for it. So a few different ways.

Seth Earley: Right. And what's interesting is you said that you could use a more expensive language model to kind of look at the results of a less costly model. So that kind of saves the money by leveraging them appropriately. So this may be, may not be the best language model, but it produces pretty good results. But now you're using another language model and then of course modular rag, which is using different Algorithms for retrieval using can handle that through say fusion types of programs. And I don't want to get too deep into the weeds there, but you know, this whole approach of modular retrieval, augmented generation is giving us a lot of capabilities, but they, they are agentic approaches. In other words, they will cause the LLM to call out at an API, have an API call out to talk to another system to produce information. Now what you're starting to see is a very complex environment where there's so many dependencies between one system and another and there's so many interrelationships that it does become a very complex beast. So let's talk about AIOps and when you start thinking about AIOps, first of all, why don't you break that down and define that for us and then we can talk about how that fits in in with some of these more complex environments and heterogeneous agentic environments that start to get really complex.

Camden Swita: Sure, sure, yeah, I mean, I guess, you know, starting from the beginning here, when I think about AIOps, right, there's, there's obviously can be a broad set of definitions there and there's an official definition and there's a Gartner definition and all these different things. But you know, I really just think about the automation of tasks that an IT operations individual typically, historically may have performed to restore a system to its normal state after some problem is detected. Right. So some problem is detected, the mitigating action is identified, the mitigating action is taken and it's confirmed that the system has returned to a good state. You can automate all of that. Right? You have, you have AIOps now obviously embedded in, that is tons of complexity to your point. And you know, in a modern software tool chain or SDLC or ITSM tool chain, there may be four, five, six or more different kind of platforms or tools involved in all of that. Right? And maybe IT operations folks are using one and engineers are using another and SREs are using something else and support engineers are using something else. And so you know, to your point, it's when we talk about, you know, automation of all of that, especially automation with some amount reasoning that you get by using modern day foundational large language models, you're really talking about kind of an enterprise data issue still, which has always been the problem and then an integration issue still, which has always been the problem with, with automation across those platforms. Now I will say the introduction of generative AI agents has sort of in my opinion begun to change the integration game a little bit. And by that I Mean, we can make natural language APIs now where every one of these platforms is going to have its own assistant. They already do. Right now those assistants can send natural language prompts to one another and get back the data that they need, the payload that they need. Does that introduce some additional potential for error or misunderstanding? Yes, but it also simplifies that maintenance and upkeep and construction of all of these APIs as well.

Seth Earley: So, yeah, that's really interesting. So I, I also think of the area of AI DevOps, right? So where you're building those systems and testing those systems and iterating. So, so AI Ops is a little bit different in that you're saying you're supporting your, your sort of maintenance and keeping the lights on. Using AI versus AI DevOps is really building the algorithms and building the systems and then testing and iterating and do impact analysis and so on. Do you see that those as two distinct areas are just very interrelated? I mean, obviously they're, there's, they're different aspects of similar processes.

Camden Swita: Sure, sure, yeah, definitely, definitely related. I mean, the, you know, obviously problems can occur and mitigating actions for problems need to be identified throughout the construction process of modern software. Software. And I think, of course, one of the areas where we're seeing kind of the, the, the most effective adoption, you might say, is in this space of writing software and DevOps, right? And increasingly you see a greater percentage of accepted code at a lot of organizations coming from a large language model. I think that'll, I think that'll obviously that will continue. I have no doubt that that percentage of that code will increase. I think it will be interesting too, to, to see how, as these foundational models or as the frameworks around them get better at writing software, not just code, but actually building out the software as a total package, are they going to get good at understanding software architectures? Are they going to get good at just really sort of like going from zero to a working application with infrastructure and all that in one go. And I guess what I'm kind of curious to see is historically, over the last 10, 15 years, we've moved to this very microservice kind of architecture as the norm. And I wonder, as we as engineers release more and more of the actual writing and management of the code itself to these large language models, is there as much reason to have such a microservice architecture? In other words, if you have a foundational language model that is very good at writing code and every time you need to make an update it can just regenerate the entirety of your code base. You don't necessarily need to outsource all of these different services. Right. As microservices. Anyway, not to go on a tangent.

Seth Earley: But that is interesting because it does call out, you know, how we're architecting these environments. And yes, there has been a great dependencies on microservices and the integration of different systems using microservices. And you're saying that that might not necessarily be as important down the road?

Camden Swita: Yeah, yeah, I mean, I think, I mean, you know, of course, we'll see but, but you know, of course, as you know, one of the main reasons why microservice architecture has, has become popular in the first place is individual organizations or software teams don't necessarily want to own and manage all of the different aspects of, of their system. You know, maybe that's the checkout process or the payment processing or the authentication or whatever it is, but at the end of the day, every single one of those services that we kind of outsource because it's toilsome for a human to manage all of that and risky. Well, if these foundational language models are increasingly good at just generating every single one of these services with every deployment, or maybe it does it in chunks or something, right? No need to waste the energy regenerating all of them. But if all that toil is sort of now managed by an automated system run by a foundational language model, then yeah, why would you need to depend on those external parties for these individual microservices to manage your system? Why not just have the foundational language model that you've chosen to manage your code just regenerate that payment processor module, that whatever it is, you know, every single time. And it's interesting because, you know, in, in some ways it could mean like a little bit more of shift backs, like a monolithic code base where you know, every time you, every time you, you, you make a significant update or an update, your entire pillar of code that represents your entire system is just getting re deployed. And so I wonder myself like, are we potentially eventually going to see, as these foundational language models take over more of this work, a shift away from microservice and external dependencies to everything's in house, it's a monolithic code block. And every time you deploy it's just, you have this cheap and effective language model that's regenerating it all.

Seth Earley: That's really interesting and in a way it almost addresses one of the other challenges. I mean, when you, when you're deploying and maybe you could just talk a little bit about not a sales pitch, but talk a little bit about what your, your solutions give us the, the 30, 000 foot. I guess I do want the sales pitch, but this isn't about selling, this is about educating. But you know, when you go into an organization and you, you, you talk about the complexities, you know, what is your 30,000 foot sales pitch for what you do? Because you're using agents, you're deploying agent approaches to monitor lots of different systems and processes. And in a way, if you, as you say, if you have fewer dependencies and you have more of a monolithic code stack, there's fewer dependencies and fewer places where things can go wrong. So in a way that's kind of an evolution away from the increasing complexity, but it seems that we're going in the other direction. We're going into greater complexity. So you want to talk a little bit about that evolution and then how your approach to monitoring, troubleshooting, you know, diagnosing those complex environments has evolved or is evolved.

Camden Swita: Sure, sure, yeah, makes sense. I mean, I think where I would start here is, you know, with the rise of using large language models in modern software stacks, you know, New Relic itself has evolved to treat that layer of the stack, in other words the generative AI layer of the stack, as a first class citize and as really kind of part of our APM solution. Right. So New Relic historically has been known for application performance monitoring and in a lot of ways monitoring the performance of, you know, the generative AI layer of all that is, is much the same. You know, you want to understand just at a very basic level. And then we'll spiral into the complexities here. At a very basic level, what is the latency around, you know, this layer of your, of your service stack? What is the cost? What is the, you know, what's the throughput there? Are there, you know, any logs that are describing errors that you might need to know about and so on and so forth. So that in and of itself is important for I think any organization adopting large language models as part of its, how it serves up its products to customers to really treat that LLM portion as another layer, just like they would treat any service or browser app or mobile app or post, you know, in, in their stack. And of course that's, that's kind of square one as far as this AI monitoring goes. But of course organizations become more sophisticated in how they're using especially generative AI to create value for their customers. That's where a lot of the fun and games begin. Right? Because if you think about what a lot of these AI agents are doing behind the scenes, especially using the more popular frameworks, is, you know, there's usually, there's an interface with the user and then there's usually a, some kind of like parent or router agent that is connected to the large language model that is receiving input. And that's, you know, let's, for simplicity just say it's a prompt and based on that prompt it's deciding what to do. It's saying, I think the user wants this and I, you know, I need to go figure out how to do that. And in order to help it know how to do that thing, what a lot of organizations, new relic included, to create this kind of agentic experience, we're giving these things individual tools or sometimes they're called skills. And these individual tools or skills are sometimes they're just another prompt that kind of like helps focus the large language model's attention in a specific domain on a specific task. Sometimes they're specific queries that we want it to run. So in other words, you know, the user says, like, go get me logs for the last hour. And the parent agent says, oh, I've got a logs tool for that. And it goes and hits that tool and that tool says, okay, I've, I have a query I know I can run to get logs for the last hour. I'm going to go do that and return the results back up to the parent agent who will take that back to the user. And maybe that parent or routing agent does some additional work on it to kind of summarize them or analyze them or whatever. And you know, you can imagine that that kind of architecture can get quite complex quite fast, especially because some of them are valuable use cases. And the things that save a lot of users time are when you start chaining the use of those individual tools together. And so you say something like, you know, I want logs for the last hour and also go find, go find like correlated like anomalies in my service metrics or something like that. And so this assistant starts piecing these things together and in order to understand what's happening with your agent, and if you're an agent engineer, as I've heard them referred to recently, ML engineer, and you want to really understand where things are potentially going wrong or where there's room for improvement, what you're really talking about is being able to have visibility in this actual, essentially a round trip trace of this input, to routing, to tool use, to response, and so on. And so it's in, it's a game of telephone. And so what you really have to understand is what's going into each of these steps. What context was there, what. And then what was the output that was generated as that step and what was, and, and what, you know, what additional context was retrieved and how did this happen all along the way? And you have to, you know, you're, you're on the, you're on the lookout, of course, for, for standard software type errors, but you're also on the lookout for essentially almost like misunderstandings. Right. Just like any game of telephone, like things are misinterpreted. And the same applies here. If you really want to debug why the assistant's giving poor answers to this specific type of prompt, you have to look at each of these steps and say, I think I see that it's maybe not getting like enough context about this specific topic here, right?

Camden Swita: Yeah, yeah, it's fascinating. It's a really fascinating kind of evolution where it's almost like this like semantic layer on top of debugging that you really have to pay attention to.

Seth Earley: Yeah, yeah. In some ways it makes it easier, as you say, because it's more understandable. The inputs and the outputs are more understandable. But then there's more complexity and there's more areas for misinterpretation, hallucination, incorrect answers and correct interpretation, ambiguous queries. And then of course, the data sources have to be the right data and the right structure and right quality and so on. So, so talk and what, what is your understanding of modular rag? So modular RAG is something that we're increasingly seeing. You know, there's RAG is just saying I'm going to use my corporate data source for this query. In other words, instead of relying upon the large language model for its understanding of the world, you're, you're saying you can interpret the query, but go to the source to find the answer and then make it more conversational. But, but that in between piece where you're retrieving, you know, I've seen some people use just full text search, right? And that's going to give you as much value as regular full text search will. And if you ask anybody on any corporate intranet how well search works, it's usually not very good, right? I mean, unless your content is really good and well curated and so on.

Camden Swita: So. But then we have more other iterations and. Do you want to talk a

Seth Earley: little bit about rag? Advanced rag Modular RAG and then how that kind of fits in with your world.

Camden Swita: Sure, yeah. I mean, retrieval augmented generation is still probably the leading way to reduce the likelihood that your LLM enabled application will hallucinate. Right. Because you're essentially filling in knowledge gaps that the language models may have in that, you know, a lot of these foundational models are trained on the entirety of the Internet, but maybe they're not trained on your specific, you know, business practices in this one area. Because of course, how, how could it be? Right? So you want to supplement its knowledge base with rag. And as, as you've already said, you know, initially it was, it was, it was a, you know, RAG was kind of like I'm going to put as much context as I possibly can, as well organized as I can kind of get it into one big kind of slushy vector database. And then my assistant, when it gets a prompt, it's going to go and do some kind of like, you know, search over that to, to find the right bits of information and use that to, to augment its, its response here. And modular rag, right, or advanced RAG is sort of making that, breaking that kind of that big wide open. I'm going to search across this entire vector database more into like a componentized approach to this, which I, I think makes it, at least in my experience and my understanding it's going to significantly improve an agent developers or machine learning engineers ability to actually, in a very targeted and fast way kind of improve the certain like components or certain parts of the context that it, that they want to improve to improve the agent's response. So of course understanding how, how and where to supplement your agent with more context in itself is a big problem and a big piece of this puzzle. And, and modular RAG or advanced rag I think is trying to solve some of that problem by putting a little bit more of a standardization around the components that go into RAG and allow you to kind of improve these individual components and say, I understand that like in order for, to get the right context to my assistant to answer these types of prompts, I need to improve like the data processing you did or I need to improve like this specific knowledge base or this specific data store that it draws from to answer questions about my, you know, I don't know, my fall clothing line or whatever it is. And so I think it's, you know, we're going to see a lot of evolution in RAG over time. And I think you're seeing evolutions both in terms of what you're talking about with modular rag, but Also, the, the means by which the assistants actually look out across the provided context, you know, adding like additional semantic search on top of that or just sort of improving how these agents actually locate the right context at the right time to answer these questions, but you know, to kind of get back to the core of the problem. And I know you've mentioned this before, it's like at the end of the day, you know, this is still very much a data game in a way. It's like in order for these assistants to be successful, you oftentimes still have to make sure that they're going to have access to that specific context you need them to have access to. And you have to make sure that it's organized in a way that's going to make it easier for them to locate. Right. And

Camden Swita: sorry, one other thing I'll mention on this that I think is kind of interesting too is, is there's, there's, there's a lot of opportunity in observability of rag, Right. Because if your assistant is giving poor answers to a specific type of question about a specific topic, you know, I guess, let me back up. How do you know that it's giving poor answers to a specific, you know, on a specific topic? Like, what if you could kind of look at a map of all of the context that you've vectorized and you've provided to your assistants and you can actually see like, we're getting a lot of prompts about this kind of specific topic area. But look how sparse our actual like RAG context is for this specific topic. We need to spend time sort of filling in that hole or, you know, we have a bunch of context down here and we never get asked any prompts that force the agent to utilize it. Maybe we can just leave that alone for now. Right. So I think there's some, there's a lot of opportunity and kind of untapped opportunity right now for helping people understand where they need to actually improve their, their RAG system.

Seth Earley: And, and you're, you have the ability to visualize that or how, how are you actually doing that?

Camden Swita: Oh, sure, yeah. I mean, no matter vector spaces and looking at how often

Seth Earley: they are leveraged or looking at how the prompts are accessing those vector spaces.

Camden Swita: Sure. I mean, one, one thing that new relic is looking at, you know, this isn't, this isn't on the market yet, but yeah, it would be, you know, essentially like a, or a scatter plot of like, here's the entirety of my RAG system and here's a visualization right of, of clusters of context by topic or however the, the vectors are organized using dimensionality reduction.

Seth Earley: Yeah, right, yeah, it's,

Camden Swita: I think it's going to be very interesting for sure.

Seth Earley: Yeah. And I think that at the end of the day when you say it is about context, you know, that's such a critically important topic and I think it's bandied about quite a bit. But even if you said the prompt was, you know, send me logs for the half last hour, you know, there's an implicit assumption about what that context is. Right. What, what system are you in or the model has to say. Well, logs of what. Right, yeah. And, and so what organizations need to understand is that cuts across all of their content. If you're asking support questions and you're asking about a particular model. Right. Just like a human would need context, the LLM needs the context. You can't just give it the world. Right. Because even at that granular and atomic level of those pieces of content, especially if you're doing question answering types of systems, you need to have that tagging and that metadata at that level of nuance and detail so that the agent can return the right information and for that right person in that right context. And it's such a very trivialized area. People don't necessarily think about that granularity in those use cases. And so how do you talk to people about data and content curation and alignment with things like use cases and how do you, how do they get to the right level of detail and granularity? Or do you not run into that? I mean, tell me about your world. Because if an organization is saying, well, I'm not getting the right results from my, from my LLM, we're trying to troubleshoot that. Right. So how do you kind of bump into that and then how do you address that or help organizations address that?

Camden Swita: Sure, yeah. A very common problem, as you've noted, there's a lot of modern day software systems and especially user interfaces have, have all been built obviously around the assumption that a human is sitting there looking at it and is able to interpret certain, you know, things or control certain things about the UI in order to understand where they are, why they're there and that sort of thing.

Seth Earley: So a very simple example, giving them context.

Camden Swita: Yeah, exactly. I mean, anytime we

Seth Earley: try to do a usability improve improvement, user experience improvement, it's always helping people match the system with their mental model, which is contextualizing the information and they're according to their mental model.

Camden Swita: Right, exactly. And now if you want an if you want a generative AI assistant to automate some of whatever maybe the end user is doing or maybe automate some of the support aspects of what somebody else would do. It's like, you know, historically, you know, in order to help say a support engineer or a service desktop person understand what was going on when a user encountered a problem. For example, which is, you know, a very common use case for a lot of generative AI assistants is kind of this frontline support. You know, you might send the support human a screenshot of what the user was seeing or something like that, or oftentimes they'll ask for screenshots. So they'll ask for a copy of the error message or something like that. And I think that to your point, that from there the support human would oftentimes need to like go look up, up the, the user's email address to figure out like what account they belong to and all this different stuff. And I think what we've encountered at New Rolla, because we've built our own agent is right, is we've, we've done a lot of work to try to see how much of the individual context of that user, you know, all the way up to their account, like or, or what page they were on, what part of the page they were on. We can actually serve to the assistant just as part of the context it automatically gets from the page. So we, we kind of click, we'll call it this like context API and it ex of New Relics platform pages right now. And it's constantly sort of serving up to this assistant a variety of metadata about the location, the account, the user, the entity, you name it. And, but there's, there's, it's a, we still haven't like quite cracked the code I will say because there's a lot of, you know, we're constantly finding places where it's like, ah, well, we need to give the assistant this additional metadata to do this. And I think that the, what I would, what I would if somebody were starting fresh here and I think we're looking into something like this too. But there's a lot of really interesting things you can do here around, you know, with like image to text models. Right. Like as an example originally when we were first building our generative AI assistant, I don't think you could have reliably taken a screenshot of a page in New Relics platform and sent it to an image to text model and gotten really good results. Like it wouldn't, it would have been very error prone. But what we're seeing increasingly is like, rather than trying to like slurp all of this context from the page individually, if we can avoid it, why not like take a image of the page and have that image to text model Actually like just like see the account ID at the top of the screen and like see the username in the bottom left and, and all this different stuff. I think there's some really interesting things that we're gonna, we're gonna be able to do as far as serving these agents more location specific context just using automatically snapped, you know, screenshots of what the user was doing. Now that may or may not be possible depending on the kind of like regulations and constraints that an organization is under. It's like some companies are not going to be willing to take a screenshot of what the user was doing when they encountered a problem, but it's still interesting. And then I think the other advice I would give anybody who's interested in creating kind of like sophisticated agentic like flows using context and all this different stuff is like really, really narrow, narrow the use case initially. I mean, I think it's really tempting with this technology to start broad to go and to try to solve all of your problems or solve all of your frontline support problems with one of these agents. And I think especially for the reasons that you've cited around the difficulty serving up the right context at the right time and the adjustments you'll need to make along the way, picking that one use case that, hey, if we can solve this, it's really going to prove the value and then we can kind of add on additional use cases over time and that'll allow you to kind of solve individual problems with some of this context, you know, along the way. It's like maybe start with a use case that helps you solve the user's account context problem first and then move. On

Seth Earley: whatever it is, right. I call that the excuse case. It's the use case that gives you the reason to solve a lot of other things, right? When you have one use case that can say, justify the revamping of an information architecture or an ontology. Now all of a sudden you have all these other applications and all these other problem areas that you can solve once you have that infrastructure. And so you're essentially saying the same thing. Pick a very narrow set of tasks or problems, solve that, and then you have the ability to expand that scope and expand those use cases to broader and broader context or broader and broader sets of problems. And then it makes it much easier for you to kind of have wins that you can build upon in the organization.

Camden Swita: That's right. What kind of advice? Go ahead. You were going to say something else? No, no, no, no. I was going to say, what kind of advice would you

Seth Earley: give to CEOs and leadership teams who are wrestling? They're hearing tons and tons of pitches from vendors every day. And I find that, you know, a lot of vendors have what I call aspirational functionality, right? They say they want to do it, they intend maybe they'll get there one day, but they don't have it today. So the stuff that's showing up in slides is a little bit farther ahead than the reality of what they can do. And so you have to separate the wheat from the chafe, right? You have to have the BS detector turned up on to high. And so. And of course, the safe way to go is with the big consultancies and the big services firms, but they don't necessarily have the answers either. So what kind of advice would you give to leadership teams who are wrestling with this stuff, trying to find the right resources, the right partners? How do they go about that? This is kind of, it's not like choose us, right, because you're talking about, you know, looking at this field more broadly and saying, how do you separate the hype from reality and what are some things that executives and leadership teams need to do?

Camden Swita: Yeah, good question. I think, I think what I would say is, is there a bubble in with, with this technology, especially businesses anchored to this right now? And I think this is particularly relevant to, to a lot of executives who are wondering whether or not they should invest more of their budget in generative AI initiatives because they're, you know, there's all this news around, like, I don't know, however many tens of billions of dollars spent over the last year and how much revenue was generated, and it's hugely disproportionate. And I think that's, I hear from some executives at companies that that's some kind of turning them off of this. But I guess my advice would be don't get too cynical. I mean, when you look at the Internet, the World Wide Web, the exact same thing was true initially. The amount of money that was getting poured into essentially the development of the Internet and websites vastly outweighed the actual benefit anybody was getting. And look where we are now, and look where some of the companies that went big into development in that space relatively early, you know, are now. Right. And so I think my advice would be, you know, I think a certain amount of, of like you say, have your BS detector up, make sure that you're really taking the time to understand what a vendor is really offering you. And if it's something that is essentially like a kind of like a reskin of something that you can get otherwise, like, you know, you see a lot of these like chatbots out of the box kind of offering these days of like, hey, we'll come in and index your doc site and give you like a docs bot, you know, and it's like, well, that's not really that difficult to build in house. And I would be kind of wary of things like that. But I also would, would encourage executives, especially those that are holding the purse strings, to not be overly cynical or overly skeptical when they're considering whether they should allocate more of their budgets into research and development using this technology. Because I do think that even if there's a bubble now, we'll, we'll start to see the, the value to investment, you know, curve really kind of turn around and not too far in the future. But the, but the thing I would kind of advise, and I think where a lot of companies who are investing in this hit a pretty significant wall is first and foremost talk to legitimate data science and data engineers, whether that's via firm or in house or just, just bring in an advisor and understand is your company's, you know, knowledge basis is, is your company's sort of like data layer, whatever it is. And by data I mean it can be marketing data, it can be business intelligence, it can be the telemetry, it can be whatever it is, get an honest and expert opinion on whether or not you're even set up at a data level to really like take advantage of a lot of the things that these agents can do. Because very often as, you know, organizations go down this path of sort of saying, we're going to build this autumn, you know, we're going to build this bot that's going to automate a lot of what we do internally or maybe automate things that our customers do. And then they find out that, oh Lord, we have a huge massive underlying, you know, data organization or data storage or data queryability problem that they need to solve. And so that's where it's like, I would strongly advise that every organization that believes that there's something to this AI stuff starts there and takes a really cold, hard look, getting real advice from real data scientists and data engineers and chief data officers and that sort of thing about like, where they stand in terms of being able to actually use their data for machine learning.

Seth Earley: Yeah. And it's, it's data, it's content, it's knowledge, it's all that stuff. And I'm going to do a shameless plug that we do an AI, a cam Knowledge management and data assessment for AI workshop. That's a sprint. That is exactly what you're talking about. It's saying, you know, where are you? And you know, many organizations are saying we're not ready to do AI, but we are ready to do ia. Right. I had coined that phrase AI without IA and you know, there's no artificial intelligence without information architecture. Well, that is the critical piece. So many times we go in and fix those pieces, which will solve lots of other problems anyway. Right.

Camden Swita: Yeah. You know, because that's basic blocking and tackling and then you're building

Seth Earley: the foundation for doing that future stuff. Right, right. So, so I get to throw in a nice. Yeah, once in a

Camden Swita: while. But, but no, I think it's important and I think it's important for organizations

Seth Earley: to understand and you know, hearing it coming from other folks to say your data infrastructure, your content, your, your knowledge, that is the stuff that these tools are going to operate on and retrieve. And if you don't have that, that then there's, there's a problem because they're going to hallucinate, they're not going to be able to have the right answers. So just want to shift gears a little bit. Tell me a little bit about your background. I know you came from the journalism area. What, what do you do for fun? What you tell us a little bit about Kendon the person as opposed to Kendall, the role.

Camden Swita: Sure, yeah, I've, I've had kind of a wild and tumbling journey through the tech industry. I, I, it's funny actually, when I first started working in tech it was like peak crowdsource praise circa, I don't know, 2011, 2012 or something like that. And that was also back in the SEO cramming days. The first startup I worked at for example was we Overstocks and Amazons and other large retailers of the world realized that they had a hundred thousand product skus that they needed rewritten and re optimized fairly regularly. Enter crowdsourcing and we had a platform and all this stuff. And the reason why I was able to make a hop from journalism to that was because there's this startup that was sort of anchored to the idea that they were going to produce high quality content for E commerce, but nobody on the staff knew anything about content like they none of them were writers, none of them were editors. So they needed someone who could come in and kind of like almost look at the assembly line for crowdsourcing and apply like QA principles to it to make sure it was, it was coming out the, the, the, the other end and looking good. Right. And so it's funny I mentioned that because I think there's, you know, as we, you hear all this recent news about as generative AI begins writing more and more content, what is that doing to like search result optimization and things like that on the line? And I start to feel some of the, the same themes kind of come back to the surface as, as were happening back then. And then along the way, I' in technology from marketing to business operations to account management, now of course product management, which is where I think really my happy place is here. And I think what I've really appreciated about working on AI and ML initiatives as the head of AI ML here at New Relic is that it's such a generally applicable technology to a lot of different problems and it's relatively easy to understand and understand the value of, of, you know, it's a great technology, I think, especially for higher level leaders who maybe aren't in the, in the technical weeds as much to think about and to devise kind of visions around. Because while there's always the risk of magical thinking, I think it really does lend itself well to beginning to kind of reconsider whether certain things that happen on your platform or in your products or in your internal processes could in fact be automated or at least greatly reduced as far as the toil goes. Outside of work anyway, though. I mean, outside of work. You know, interestingly, actually, I still love to write. I still love creative writing. I write short fiction, which has been another really interesting space with, with the rise of large language models, of course, because, you know, suddenly writers can potentially train their own small language models using all of the things that they've written before to now generate more content that sounds just like what they're writing. And I think there's, there's going to be some. I don't know. In the arts especially, we see this a lot with image generation and video generation. But I think the true, the same thing will be true of writing. It's like as the maybe more arduous or toilsome aspects of art generation, whether it's written or otherwise, are increasingly automated, what's going to distinguish good, let's say, written fiction from bad written fiction? Or will we start to see, you know, entirely different requirements for what is good written fiction emerge. Anyway, it's just a fascinating field, so that's. That's kind of how I spend a lot of time.

Seth Earley: Any. Any other hobbies or any kids or animals? Pets?

Camden Swita: I got two cats, Frog and Toad. Love running. Names. What are the names? Frog and Toad.

Seth Earley: Frog and Toad. Yeah. Yeah.

Camden Swita: Love running, love hiking. I live in the Pacific Northwest, so we're. We're big on the outdoors up here. And, you know, outside of. Of writing and work is. There's not much time left in the day otherwise.

Seth Earley: I get it. It's funny, you know, back to your question or your comment about AI enhancing creative writing. I mean, I do a lot of professional writing, and I find that it slows me down because I. I don't use AI generated content for my writing. I. I write my. I might use it to do a little editing. I might use it, but sometimes it messes up the content because it makes it sound like AI Right. And don't want that. But I'll use it for research, I'll use it for outlines, but then I end up with more stuff that I need to go through to synthesize from. You know, for some paradoxical reason, I find it reduces my productivity, which is a strange thing. You know, it's easier for me to do some research than it is for me to try to generate stuff. So I don't know, it's. But I do need to kind of do what you said is train it with all of my prior content so that it has more of my voice. Sure. I got around to yet.

Camden Swita: Yeah, it's worth trying. And I think you may also find, I mean, to your point, it's like some. Some aspects of your process are probably good candidates for using an LLM and others, not so much. I mean, I personally. Not so much for creative writing, it's more of just kind of tinkering with that. But at least for professional writing, it's like, you know, that you can either start with a blank page for me, or I start with something that's like even 40% acceptable. And if I can get 40% acceptable out of an LLM and just take it from there, a lot of times it's time savings.

Seth Earley: What kind of fiction do you write?

Camden Swita: I would consider it just kind of, you know, general literary fiction. So it's not any kind of genre fiction. It's just. Yeah, kind of the short fiction. You'd seen magazines.

Seth Earley: Well, Camden, thank you so much for sharing your insights here on this episode of our early AI podcast. I really appreciate it. Thank you for your time. Thanks for your insights, of course.

Camden Swita: Yeah, happy to be here. Thank you so much for having me.

Seth Earley: And thanks everyone for tuning in and Carolyn, for your work in the background and for the other folks who've been helping out. And we will see you next time. And by the way, in the show notes you can find where to locate Camden by the way on Twitter. It's sweet at Camden S w I T a C a D E n on Twitter. The company website is new relic n e W-R-E-L-I c.com and again, thank you for being here and we will talk to you next time on the next episode of the Early AI podcast. Thanks again.