Structured Content, Information Architecture, and the Critical Role of Metadata in Successful Generative AI Deployment
Guest: Lief Erickson, Content Strategy Consultant and Technical Documentation Expert
Hosts: Seth Earley, CEO at Earley Information Science
Published on: February 27, 2024
In this episode, Seth Earley and Liam Kunick speak with Lief Erickson, a content strategy consultant with decades of experience in technical documentation and information architecture, particularly with DITA frameworks. They explore why generative AI is not a magic bullet for content problems, how structured content enables effective retrieval augmented generation, and why organizations need clear business cases before launching AI projects. Lief shares insights on the critical relationship between prompt engineering and knowledge engineering, real-world chatbot failures that led to increased support calls, and the emerging role of content designers who specialize in training and validating LLM responses.
Key Takeaways:
- Generative AI appears to be a magic bullet in constrained demos but fails with large organizational corpuses without proper content curation and validation.
- Most executives pursue AI projects driven by fear of missing out rather than clear business cases, leading to impressive prototypes that fail at deployment.
- Structured content using frameworks like DITA provides semantic tags that help machines locate information, similar to organizing books in a library system.
- Retrieval augmented generation produces made-up responses that don't exist in source content, requiring trusted data sources with metadata to ensure accuracy and prevent hallucinations.
- Prompt engineering should align with knowledge engineering, treating prompts as metadata-rich queries that navigate multi-dimensional vector spaces to find relevant content.
- Organizations compete on knowledge, not technology, making content architecture and metadata critical for differentiating LLM-powered applications from generic implementations.
- Real-world chatbot failures include a 12-fold increase in support calls and legal losses when chatbots provided inaccurate policy information contradicting website content.
Insightful Quotes:
"It's not a magic bullet. When we start dealing with exceptionally large corpuses, large content sets, then the problem starts to be one of curation, of validation, and we end up with all sorts of errors or unclear responses coming out of the generative tech." - Lief Erickson
"More than anything, a generative AI project is a content project. Getting the terms aligned, having the taxonomy, moving things into an ontology so those relationships are clearly defined—that all informs the LLM, which is going to produce a more accurate, better response." - Lief Erickson
"How is the person supposed to know if interacting with the chatbot on the website? The onus shouldn't be on the person to go find every page to verify that there's no contradicting information." - Lief Erickson on a chatbot lawsuit
Tune in to discover why successful AI implementation requires strategic content architecture, structured metadata, and clear alignment between prompt engineering and knowledge engineering—not just advanced technology.
Links:
LinkedIn: https://www.linkedin.com/in/lief-erickson/
Website: https://www.intuitivestack.io/
Ways to Tune In:
Earley AI Podcast: https://www.earley.com/earley-ai-podcast-home
Apple Podcast: https://podcasts.apple.com/podcast/id1586654770
Spotify: https://open.spotify.com/show/5nkcZvVYjHHj6wtBABqLbE?si=73cd5d5fc89f4781
iHeart Radio: https://www.iheart.com/podcast/269-earley-ai-podcast-87108370/
Stitcher: https://www.stitcher.com/show/earley-ai-podcast
Amazon Music: https://music.amazon.com/podcasts/18524b67-09cf-433f-82db-07b6213ad3ba/earley-ai-podcast
Buzzsprout: https://earleyai.buzzsprout.com/
Thanks to our sponsors:
VKTR
Earley Information Science
AI Powered Enterprise Book
Podcast Transcript: Content Strategy, Structured Architecture, and the Reality of Generative AI Implementation
Transcript introduction
This transcript captures an in-depth conversation between Seth Earley, Liam Kunick, and Lief Erickson about the fundamental importance of content architecture in successful AI deployment, exploring common misconceptions about generative AI capabilities, the critical relationship between structured content and retrieval augmented generation, and real-world examples of chatbot failures that demonstrate why proper curation and business strategy must precede technology implementation.
Transcript
Seth Earley: welcome to the Early AI Podcast. My name is Seth Earley and I'm here with Liam Kunik who is our guest co host. Today I would have to say hello Liam. How's it going everybody?
So Liam helps with the pre production and post production. Chris Featherstone couldn't make it today, so I had Liam join us. So today we're going to discuss content strategy and content architecture and information architecture as they relate to large language models and generative AI. And our guest today has decades of experience in technical documentation and information architecture in dita. If you're not familiar with dita, it's Darwin Information Typing Architecture and he has done a great deal of work in the across industries and working in content strategy as a master's doing content strategy. Welcome Leif Erickson. Thank you for joining us today, Leif.
Lief Erickson: Seth, I am really looking forward to our conversation.
Seth Earley: Wonderful, wonderful. So I wanted to start off with what would you say are some common misconceptions around generative AI or large language models and content specifically? What are your thoughts there?
Seth Earley: Well, I think the first thought is that it's a magic bullet that is going to solve all of the existing content problems that have existed in organizations up to this point. It's not, it's not a magic bullet. Why
Lief Erickson: do people, why do people think that and what problems, how are they, why are they thinking that? And then what problems is that going to lead to? They're thinking that, I suppose because
Seth Earley: some marketing folks are really good at their jobs and in specialized use cases, very constrained demos, the technology looks fantastic and it, it can be, but when we start dealing with exceptionally large corpuses, large content sets that the organizations that we tend to deal with, then the problem starts to be one of curation of validation and, and we end up with all sorts of errors or unclear responses coming out of the generative tech. So it's interesting,
Lief Erickson: you know, when you, when you look at generative AI and you look at the problem of content and large volumes of content, it is a context problem, right? Just like we've always had a context problem. If you ask an ambiguous question, you're going to get an ambiguous result and the Same thing happens with large language models, which is why the whole idea of prompt development, prompt engineering is so important. So tell me a little bit about what are your thoughts about. So there's a misconception that's going to solve all your problems. What else, what else are you running into in the industry? Well,
Seth Earley: the. The other big issue that we run into is why do you want to do the problem or why do you want to do the project in the first place? Like, what's the business case for this AI project? And we are seeing that executives are struggling to come up with the reason why, you know, the one that is justified in the dollars and cents of it all. And right now, I think what everyone is doing is chasing the hype, wanting to not be left behind.
And I think if we zoom out, there's. I'm really speaking to two different issues there. The. The number of folks that are really dealing with a generative AI in a systematic approach that have the business case for it is very small. At least what we're seeing, the number of folks that think it's a great idea worth pursuing and don't have that business case is a much greater number. And that first group, that first cohort is they're going to be fine.
The second group, the much larger one, is really going to have problems and struggles as they go through the process. I think they're first going to approach it as a technology project and they're going to get to a certain point and things are going to look great. There's going to be lots of progress. The engineers are going to be connecting things together, spitting things out, seeing, look, we can do something, and then the rubber is going to hit the road and there is going to be erroneous answers.
You had mentioned earlier, context, an intent that is going to be open to interpretation and the executives are going to wonder what they got themselves into.
Lief Erickson: So when you start thinking about building that business case, I mean, what have you seen or what do you think the business case should look like? And I totally agree with you. I agree that there's a lot of fear of missing out. There's a lot of executives saying, well, we have to do something. We have to do something with Gen AI. We have to show that we're leading edge or have to show that we can. We're AI savvy. But then when it comes down to it, it's like, okay, what do we really do? What problems are we really trying to solve? And as you mentioned, you know, there's a lot of technology Firms or vendors that are offering solutions that are more marketing hype, you know, they're communicating stuff, saying, oh yeah, you know, we'll, we'll put all your content in here and you'll have, you know, much greater access, much more precise access, et cetera. So the question is, what do you see as a high value use case that people should be thinking about and making that business justification?
Seth Earley: Well, I don't know every business and what their needs are. So the executives themselves are going to have to know what their problems are. And we can help them define that and we can help put some parameters and some objectives and metrics around what those look like. But are they looking to increase brand loyalty in some way? Are they looking to be more efficient? Are they looking to reduce support calls? Do they need to manage risk? We'll come back to risk almost certainly in our conversation. Do they need to increase revenue? What is it that they're looking to achieve with their AI project? But that, those issues that I just raised or the, the objectives, those apply for any project, not just an AI project. So it's, it's why are we doing this in the first place and what does success look like? Right.
Lief Erickson: I think that's a great point. And, and you know, a lot of organizations are approaching this for support, customer support, and it makes sense to do that. But again, we need the context and we're not, it's just not going to work as expected if we just unleash it on all of the content. You know, the example I use when I explain this is if you are looking to troubleshoot, you know, an error code and you have a certain device and the error code is 50. Well, if you're on the phone with somebody, you know, you can't just say, what's error code 50 mean? Right. That's an ambiguous question. They're going to say, well, what model, what device do you have and what model and what's the configuration? And the same thing, if we're using a virtual assistant or we're using a chatbot, the chatbot doesn't have that context. It's going to need that context. And on that piece of content, we have to tag that content with that context. We have to say error code 50 for this type of modem or router or bridge for this type of device with these configuration parameters. That all has to be there because otherwise there's no context. So maybe we can talk a little bit about content architecture and things like dita. For people who are not familiar with dita, let's talk about that, maybe define it and talk about it. Talk a little bit about how our content architecture helps.
Seth Earley: Right? So for folks who are not aware, DITA is an XML framework for identifying semantic chunks of information. So with the XML you would have semantic tags describing the content, you know, as a set of steps, maybe so, or reference material or more specifically a field setting. For instance, you could you, it's not part of the out of the box ditto, but you could start to tag your content semantically that this is a field setting or this is an error code and everything within the error code elements or tags would be findable than by our machines that are going to extract the information out of the content set.
Lief Erickson: So when you think of content architecture it's telling you the isness and the aboutness. What is this thing? This is a piece of documentation and what is it about? It's about this particular modem, it's about this particular router, it's about this particular error code, whatever it is, right. Or troubleshooting step and so on. What we have found is that you know that that kind of tagging does the, the recall and precision for the LLMs. So Seth, can I jump in here and add
Seth Earley: a little bit more to that and think of, think of that aboutness or isms the way you might think about if you were to need to find a, a book in a library. You walk into a library, some people still do that and the books are organized. They're organized following a particular system. And you just need to find where the book is located in the rows and the shelf, the structure within. The markups help the machines and humans find that information.
So it helps both with the, the input and, and the output of, from a generative AI perspective, it helps the machine understand what and where the content is located. And then when it combines that with other technologies to generate the, the response, it helps then the, the human on the other side of that query get their answer so they can go about what they were looking to achieve. So this kind,
Lief Erickson: kind of comes to this whole idea of retrieval, augmented generation. Do you want to define that for people? And you know, they're probably somewhat familiar with it. But I want to hear it in, in your words, how, how would people think about, because you just talked about the fact that you're finding something, you're using these tags to find something. You're trying to find the book in the library, what shelf, you know, what room, what shelf, what you know, et cetera. So talk about what that means with regard to ChatGPT Types of applications and large language models. I think we need to sort of separate
Seth Earley: that into a couple of different aspects. And I think we need to start with the understanding that the response that ChatGPT or other systems is going to produce doesn't exist in the original source content. So what ChatGPT is producing is made up, that is the generative regenerative part of the retrieval augmented generation. So if we back up into what. Was the
Lief Erickson: process made up, it's made up based on its understanding. It's not, we're not saying necessarily that it's fictional, we just are saying that this
Seth Earley: response doesn't exist anywhere in its source content data set.
Lief Erickson: Yes, and, and the answer is coming from the large language model's internal representation of concepts and terminology and relationships about the world. Right. Based on content that's been training. Right,
Seth Earley: so let's then understand what, let's break that down a little bit. And this exchange started with a human putting a query into the system and that query has a number of words in it. And the ChatGPT takes that and unders, you know, starts to break it down into intent based on a number of other things that we don't have to get into here. But that then goes back to its source corpus and starts to retrieve information that is relevant based on the keywords and entities that it is finding in the original query. Behind the scenes, there's some additional information being added around the prompt that gives it some context. But, but that's the process of the retrieval, the augmentation before you get to that generative made up response. Right,
Lief Erickson: so, so it's interesting, you know, I, I think of it as you are, as you said, you are defining the intent of that query, right? You're trying to resolve it. Just like when we have a chatbot and we have utterances, there are different ways of expressing an idea or concept. You know, my, my password doesn't work, I'm of my computer or I forgot my username. Right. Those all mean the same thing, right? Change password, update password, reset password, whatever it is. And then we're doing that on, on steroids, right? With, with a large language model. Because some of these prompts can be quite complex, right? And you can stack prompts and you know, I'm, I'm doing a course right now where we're looking at building these really extensive prompts. And what's interesting about those is that the LLM has to say, well, what does this really mean by putting those prompts into a vector space? And then saying what's near this vector space in terms of other content, in terms of related content. Now if it does it based on its own understanding of content, that's one thing. But as you say, that can generate erroneous results. But instead if we're retrieving some information from a trusted source, then we're actually taking that information and then making it more conversational with the LLM. So it's a matter of saying that those tags and those, and that architecture and that structure adds the ability to fine tune what those results will be and to make sure that it's valid. Right. It's coming from a vetted quality information source within the organization. So it's interesting when you think of the retrieval piece, you know, organizations compete on your knowledge, right? So it's, it's about, it's not. If you, if everybody just used an LLM to answer questions, well, they're not getting any differentiation, right. They differentiate on their knowledge. That's competitive advantage. So the idea is to say use your own knowledge, your own content, and structure that content in such a way that the LLM can make the best of it. So that's kind of, you know, where this is extending to. Do you have any more thoughts and context on that? Yeah, you, you made me think of a couple of different
Seth Earley: things. You'd mentioned search and you know, that is certainly one way that people are going to be using chatbots now is that, you know, with, and the, the virtual assistants. It's that we've been trained over the last 25 years of Internet searches getting progressively better for, for those of us that have been around a while, we have seen how searches have improved. I play a sport called squash, but squash in and of itself is somewhat of an ambiguous term because it has a meaning in a botanical sense that is very different from a sports sense. And 25 years ago, putting search into, you know, Yahoo at the time or Google returned ambiguous results. I would be getting recipes for squash. And very little about the sport might have something to do with the number of people who play the sport, but that's irrelevant. But it's dealing with search and intent and accuracy.
And over the years, searches results have gotten better and better and better because search engines have gotten better at determining intent out of our queries. You know, that. Did you mean instances? That started coming up many years ago. Now the other thing that you made me think of is the idea that the quality of the data and you'd spoken to. Well, knowledge is what separates one organization from another organization. And it is that taking your organization's information and pairing it with a particular LLM is what is going to be the, the, the technical connections of what will make your virtual assistant to your chatbot respond differently than somebody else's. Yeah, and, and, and that's what it's all about. Again, because
Lief Erickson: we're trying to have competitive differentiation and standardization gives us efficiency, but differentiation gives us competitive advantage. Our competitive advantage is based on our content and our knowledge and our expertise, knowledge of customers, their needs, routes to market, the technical solutions, the competitive landscape, market, you know, ecosystem, and so on. So let's see when you start thinking about organizations that at.at.at. @ content strategy and being more serious about this. I mean, do you think that they, you know, what, what are you seeing is driving the need these days for content strategy and componentization versus what we'll see in the future, which I think is going to be more about retrieval, augmented generation and, and having that as a reference point. But what are your thoughts about what's driving. Today? It comes to
Seth Earley: efficiency in, in many cases it's how if we're a multinational organization and we translate and localize our content, you know, having structured content makes translation and localization significantly easier. Not necessarily easy, but significantly easier. And that can lead to massive savings over unstructured content, translation, localization. Another reason is managing risk. If you're in a regulated industry and you have content that has to be written in a very precise and particular way, having a very finite set of, or discrete set of information that needs to go through that review process to help manage and control that risk, that would be another reason. Another thing is maybe you have content that can function in multiple places, so the context is different and that allows you to produce more products and therefore sell more. So having, and that increases revenue.
So we've just talked about, you know, three different ways that from a managing risk, reducing cost as well as increasing revenue, that is all supported through, through structured content. And if we zoom back out the strategy. So the structured content is how we go about achieving the business objectives and goals and the strategic. Desires. Talk a little bit more about that. And
Lief Erickson: Liam, I know you have a question, but talk a little bit more about that leaf in terms of increasing revenue, where are you seeing the increased revenue play with structured. Content? Well, it was
Seth Earley: just like I was, I just spoke to. If you have a component, oftentimes we talk about documentation and specifically in the structured authoring world, as little bits of Lego blocks and that you can com, you know, combine them in different ways to produce different things. Well, that idea isn't unique to structured authoring. That is also how many products, engineered products, software or physical products are built as well. So what we are doing is with our structured content is really matching what's already happening on the engineering side. It's just that that same level of thinking hasn't been brought to bear on the documentation sets in as many organizations as that thought process and thinking is with the engineering side. And so in some ways, our industry is catching up, our practices are catching up to other things that are already happening. And that all goes to velocity and scalability of an organization. An organization always, most organizations always want to be growing and doing things faster on a shorter time frame. And, and in order to do that, in order for the documentation team to meet those objectives, they. Structured content really helps move the needle forward for them in.
Lief Erickson: That. In that way. Yeah. And I, and I suppose, you know, the way we look at some of the personalization approaches, they use structured content to experiment with different variations of messaging. Right. Different call to action, different hero image, different target, different value proposition and so on. And the more personalized we can make that content and the more appropriate the content is, depending upon where someone is in their life cycle, where they are in their journey is going to improve conversions. Right. Which will lead to increased revenue. So I think that's a really important piece. You know, there's a lot of things. Liam, you were going to toss out a question there. What were your.
Speaker D: Thoughts? Yeah, I had a. I had a couple of things cross my mind, but I've been so hyper focused on this conversation. It's really, really cool. I didn't want to interrupt anything. Gonna kind of rewind a little bit when we were talking about the importance of prompts with content creation and generative AI. Leif, I know you have a lot of experience with content creation or content management and you really kind of understand the science and the processes behind all of it. My question is to kind of get both perspectives from you and Seth. I know one of like the big AI misconceptions that a lot of people have is they get worried about job security or like certain things that they're going to be doing day to day. Is now, oh, irrelevant. What do I do instead? Now that kind of, I guess, ties into content creation and things like graphic designers when it comes to, you know, making AI generated images or like copy for this podcast episode description, for example. Weird. Use that as a tool. Do you see kind of like any sort of, I guess, room in the future? When it comes to utilizing generative AI for content creation with like any sort of specialization in like prompt engineering or I guess, like, I guess I'd call it prompt people or prompt specialists, do you think there's kind of like room in the content creation or content management space for people that specialize in just being the best prompters ever?
Whether it be for some like an rag, you know, focus LLM or just like using chat GPT that might I guess show some sort of space in the tech industry for growth or money or education, you know, just kind of wanted to get your perspective on that. Prompt. People. Yeah, Liam,
Seth Earley: that's already happening. Okay, There is a, as we sit here and record this podcast, there is a newly posted. It wasn't a prompt engineer position, it was a content designer for chatbots or something like, like that job description where that person was going to be entirely responsible for feeding the content into the, the LLM, training the LLM, querying and ensuring the accuracy of the responses. And they were going to be responsible for creating tickets for the engineering team as well as for the content team. So there is absolutely going to be that type of work. Now, this is an organization that recognized that they needed to avoid certain issues. They don't want the LLM producing fictional responses, no hallucinations, or they want to be able to catch the hallucinations before their customers do. And so yes, there is going to be absolutely room, a new opportunity. And on the other side of that is as content creators, people are going to become more efficient.
There is an assistant sitting there available to them. The blank page is always the hardest one to fill. And the generative AI systems will certainly help authors and content creators fill in that blank page. Maybe it's just with an outline of if they need to create a certain structure of content that is, that is known and understood, they can ask for some defaults and maybe that's where more specific prompts will need to be written. But it's a start. And Seth, you had mentioned earlier, it's sort of the stacking props on top of each other, the queries building on each other, and that is where we go. Next. Yeah, I would agree
Lief Erickson: with that. And Liam, it is a good question because the prompts are really the queries, they're the ways of asking the questions. And those should be aligned with the content architecture and the knowledge engineering piece of this. Because basically when we're asking a question of a data source, whether it's an LLM by itself or an LLM with a structured repository, a Source of truth. What we're doing is we're giving it hints about what entities are important. A marketing report or write a proposal, write a presentation, write an outline for a speech. Content types in there, there's topics in there. And again, as we start doing this with more of our own content, rather than relying on the LLM, those tags are going to be important and aligning those tags with the prompts will be important. So the prompt engineering is really query building, right? It's really understanding how to ask the question of the LLM or how to ask the question of the repository, leveraging the LLM's ability to normalize that, that query, that prompt, that question, right? And, and thinking of how you're getting to, you know, these are all in multi dimensional vector space. It's hard to think in multiple dimensions, right? Beyond three dimensions. But you know, they're N dimensional. In fact, you know, there are as many dimensions as there are features. And so we're talking about, you know, hundreds of thousands, millions, you know, in a large language model, you know, there could be a billion parameters, right? I mean, there are, there are the, the whole, the parameters of the large language model itself can be multiple billions of parameters. But even a vector representation of a piece of content could have tens of thousands of features, right? Because each feature is really metadata. And what we're trying to do is navigate this three dimension, this N dimensional space to get closest to what that right answer would be. And so the prompt is actually being converted into a vector and then that vector is being compared to other vectors and saying, well, what's the closest vector across these multiple dimensions where I can get the answer that is most suitable to my query. And again, those queries can be so complex they're going to have thousands of dimensions. And that's where, you know, the power of these things is, just, is vast. It's, it's incredible. And that's why it's so computationally intensive and heavy. So the idea of saying, you know, let me, let me give all this context, right? For a, for a prompt, you know, you're talking about audience, you're talking about topic, you're talking about voice, you're talking about all these things that could be broken down into metadata and metadata of the prompt. If you say, I'm going to build prompts, I need to include certain metadata, that metadata needs to be aligned with my content model. So I look at it as both sides of saying build the prompt using knowledge engineering approaches or align your prompt engineering with your knowledge engineering. So that you can make the best of both of those worlds because otherwise, you know, you're, you know, you're not able to correctly navigate that space, that, that vector space. And I know it's hard for people to imagine that, but that's what we're doing. You know, we're navigating multiple dimensions. And, and those dimensions can be thought, they can be thousands of dimensions. Right. You. Impossible for the human mind to comprehend, but that's what's happening mathematically with these tools. And, and as I say, the more, the greater number of parameters in a model. That's why we say, you know, model has a billion, two billion, whatever, billion parameters. I don't know what the largest number is right now. Do you know offhand, Leaf, how big the models. Are? I.
Don't. 1.8 billion. I, I forget what it is in the background. Liam, you can do a search on. That. Sure. What are the parameters? The largest, largest of large. Languages. By the time this gets posted,
Seth Earley: we'll have. Changed. Yeah, exactly. But the point here is, you know, what is the, what is the actual. And the greater the complexity and the more
Lief Erickson: the parameters that large language model has, the more complex the queries it can manage and the prompts it can. Manage. But, but in some ways it's, it's way simpler than that.
Seth Earley: From, okay, from the user's perspective. The user and you, you used a lot of language that spoke to information architecture without actually saying information architecture. And you also spoke to user experience from the user's experience. From the user's perspective, they're typing in to a text field. They have one dimension that they're looking at and they have no idea what the, the navigation or the content structure looks like. A chatbots interface is very different than a website's interface. A website gives you a number of clues to what the, the content structure might be. There's navigation and there, you know, headings and texts that labels that you can see. There are facets that you might be able to choose. None of that exists with a chatbot or a virtual assistant. So from a querying perspective, there is no, nothing exposed to the, the, the user and the interfaces that I have seen that give any context to what the corpus structure might be. So the user has to ask blindly what is in what is in the cont. The corpus. And the response will give them some more information, which then might trigger a conversation of well, no, no, no, that wasn't exactly what I.
Lief Erickson: Needed. Right. And I have to change my. Question.
Speaker D: Well, Yeah, I was 175 billion for OpenAI chat. GPT. But that was, I think,
Lief Erickson: 3.5 maybe, let's see, 3 GPT3 was 175 billion. They said that GPT4 possesses trillions of parameters. Trillions. So the large language models, that's the number of parameters. And again, features are one of the parameters. The feature can be thought of as metadata, right? So Leif, your point of saying that the user doesn't have any understanding that structure. That's why where I'm making the argument that we do want to build these things with some awareness of those structures, right? To say, let's do our prompt engineering that contains certain entities and our content that will contain certain entities. And then part of this is exposing that as a dialogue, right? As a, a conversation point here is that if, if I ask an ambiguous question, the LLM has to be able to respond with a clarifying question. Just like with faceted search, when we put in, if we type in tools on the Granger site, it's going to give us hand tools, power tools, this kind of tool, that kind of tool, right? It's going to give us a whole bunch of different parameters where we can clarify. That's the question. You know, if you walked up to counter in a, in a, you know, in a hardware store and you said tools, what would they say, okay, what do you want to know about tools? Or what kind of tools? What are you trying to solve? So, so the same thing has to happen with our chat interfaces where the chat bot has to be able to ask clarifying questions in order to get context. And I don't think anybody's really doing that right now, but I think the idea is to say, you know, ask questions and use those questions to guide the, the, the LLM in where to go in that vector space so that you can leverage facets you can leverage. And you're not, maybe you're not doing it explicitly, but you could, you could do it explicitly. You could ask clarifying questions which could have, you know, results from a, from a faceted retrieval. But I think that it's, it's having that awareness of the content and saying, well, we do need some structure of the content. You know, we had done that study that showed that adding metadata and a content architecture to content that's ingested into a vector space is actually more accurate, produces more accurate results, right? It was 53. And this is where we're preventing hallucinations by saying, you know, if you don't have the answer from this data source, say, I don't know. So again, that was something where, you know, the LLM was restricting its, its query to a specific data source. And then within that data source we had the additional signals, which are essentially features or parameters of the content. And that additional metadata, you know, again, can finely tune that retrieval. But at the same time, we have to think about those questions, right? So it's, it's, it's saying, well, what are people, this is why use cases are so important. What are people trying to accomplish? And if they're trying to get a certain type of information that has to be in the content model, so those prompts need to be engineered or the, or the content model needs to be reverse engineered from the prompts, essentially. Does that make. Sense? I understand what you're saying and
Seth Earley: what, you know, you, you started backing out to, you know, that the area of my focus, which is the, the strategy of it all, like, why are we doing this all in this project in the first place? What is it that we're looking to achieve from the, from the business or for the business? And being very specific about that helps define what success looks like for that project. So, you know, rather than, you know, having a chatbot that will respond to any and all queries, maybe it's at first only within a specific domain within your content set so that you can understand where the, the wiggle room is in the, the responses that come from the, the chatbot, the LLM, so that you can then go back and refine your prompts and get more specific with the, maybe the training, the curation of the content that goes in. Because more than anything, a generative AI project is a content project. Getting the terms aligned, having the taxonomy, what, moving things into, you know, an ontology so those relationships are clearly defined. Because that all informs the LLM, which is going to produce a more accurate, better. Response. That's right.
Lief Erickson: So I think we have to start with those narrow sets of use cases and a defined corpus and say, let's get this laser focused to solve this specific problem. Maybe it's supportive of a particular product, maybe it's customer service queries around whatever the service offering or whatever the product suite is. But that's where technical documentation comes in. Because we can take that technical documentation ingested into the LLM with the additional metadata, right? Because those become the features in the context and then use that as a way to query that content. And essentially, you know, right now people are on the other end of the spectrum, right? They're just saying, well, let's just, you know, put all our content there and ask it questions. Well, that's not going to work. Right. We have to be more focused. You know, we get to be laser focused on a set of use cases, domain of knowledge, a corpus, and then we need to think about the relationship of the prompts and the prompt engineering with the knowledge. Engineering. I want to
Seth Earley: focus in on that response a little bit, Seth, because it actually will work. Because from the other side, it will produce a response. It's only as you get into the details of the response that I think you and I would agree that maybe the response or the answer wasn't accurate or precise enough, or maybe it was even misleading. Right. But the technology is going to produce a response. And so at one level, it will look like. Success. Right. But again, it comes back to those use cases,
Lief Erickson: because you need a gold standard. You need to say, how do I judge the success? How do I judge the accuracy? Because I was talking to someone the other day at a life sciences firm who said, and, you know, we asked the question of what's the lunch expense policy? And it said, you know, $180,000. Right. For lunch. Right. And I could have. I could. I could get lunch. For. I could have a good lunch with. That. Yeah. And it was. And it was. And it was saying for Argentina or someplace. Right. It was not necessarily saying for their location. And then no matter how they asked the question, you came back with that ridiculous, erroneous answer. So you're right. It will produce a result. It's not producing satisfying results. It's not producing trustworthy results. And sometimes it's not producing accurate results. So that's what we need to do. Liam, you had another. Question? Yeah, I wanted to talk more about content
Speaker D: management and strategy with generative AI because this, to me has just been, like, really, really interesting topic lately the past couple months. So when it comes down to, like, excuse me, so let's say fast forward into the future, you realize that the entire Internet is just every single company using generative AI to make things, make stories, write articles, make photos. And then it comes down to, like, legal or copyright concerns and considerations, who really does own that. And I guess, like, that's kind of like how I think about it. And I've had a. I had a conversation with a friend in one of my IT classes about this where he was like, hey, man, if da Vinci made a machine that painted the Mona Lisa, was it da Vinci that painted the Mona Lisa or was it the machine that painted the Mona Lisa? You know what I mean? And I Just wanted to kind of get your perspective or hear what you have to. Say.
Seth Earley: That is not my area of expertise. I don't, I, I'm not certain that I would feel comfortable wading into the, the waters of copyright. I will leave that to other folks more trained and you know, thinking about those issues, no doubt it is one that we should be concerned about very much so, both in the sense of what is the legality of the, the text that's being generated, who owns that, but also from the, the ethics perspective, you know, is the, the responses that are being generated ethical to any and everybody?
And then you know, lastly, you know, are they internally consistent with each, each other within a content set? And we're already seeing instances where organizations have put chatbots out there in efforts to reduce their support calls or have another avenue of interacting with the organization. And depending on how successful they were at their training, their curation, sometimes they have failed spectacularly.
One, one organization I read about had a 12 fold increase in their support. Calls. Wow. That chatbot didn't stay up. But for two weeks before it was taken off the site for, for obvious reasons and relatively recently there was a well publicized situation where a user had used a the chatbot on a company's website to ask about a policy and it received the person received an answer and had made a decision based on the response from the chatbot, only to learn later that the chatbot had produced an inaccurate response and that the information from the chatbot conflicted with information on the website. And the person sued the organization and the organization ended up losing because the judge ruled that how is the person supposed to know if interacting with the chatbot on the website, the onus shouldn't be on the person to go find every page to verify that there's no contradicting. Information. Exactly. Exactly.
Lief Erickson: Hey, we are right at the top of the hour. This has been fantastic. I would love to see that, that, that case study or that, that example if you want to send me that information. But Elif, thank you so much for joining us today. We really appreciate.
Seth Earley: It. Seth and Liam, this has been. The time went by so. Quickly. What a. Joy. It was a. Pleasure.
Lief Erickson: All right, and thank you to you, to all the folks that are listening, thank you Carolyn and of course Liam for your, for your efforts and we will see you next. Time.
Seth Earley: Thanks. Everyone. Thank you