Metrics & Measurement
Cost: $50.00
Date:
April 26, 2006 - 12:00 - 1:00 EDT How do we measure the value of a taxonomy?
- What are the costs associated with indexing?
- How can we make the business case credible?
- What is the best way to develop baselines?
Several presenters addressed these questions: Seth Earley, Lynn Boyden of Symantec, Christine Connors of Raytheon, and Ron Daniels of Taxonomy Strategies.
Seth Earley
Seth started out by demonstrating his characteristic sense of humor. For those of you who have seen him present you know he keeps things lively and interesting. I'll give you a hint: one of his bits could be called 'shrunken head' and another 'spam-taxonomy-spam'.
On a more serious note, he introduced the challenge of measuring taxonomies. Taxonomies are often far removed from actual business processes, and if we measure an "indicator", it's difficult to link the result directly to the taxonomy. He then introduced the question of qualitative and quantitative measuring, both approaches being used to measure taxonomy success.He finished with an example of NCR, who attempted to justify a KM system through quantitative links between effectiveness and the system.
Measuring Search Success
Lynn Boyden, Symantec
Lynn presented on her experience at Symantec in tracking terms used to tag security writeups (in regards to computer viruses) on the corporate website. She described her process of optimizing hits on the site and her taxonomy by using better terms as: check the blogs, check the logs, check the website, then tag.
To measure success, she compared volume and successful hits before and after tagging, making the value of proper metadata more obvious to stakeholders.
She went over an example of a recent virus case in which search success was greatly improved thanks to retagging based on her evaluation. An interesting case study which showed how evaluating search terms can support business objectives.
Taxonomy Metrics and Measurement: Using search and web analytics tools to gather data
Christine Connors, Raytheon
Christine presented the tools she uses at Raytheon to evaluate her taxonomy's terms, based on search behavior and results. The tools she discussed included:
-
Search logs and indexes
-
Web server logs
-
Web analytic tolls
-
Spreadsheets and editors
Her main message was "use what you have – know the capabilities and limitation of your tools".
She went over multiple helpful examples of how she uses these various tools and how their outputs can be visualized and interpreted. She gave helpful tips on how to parse and present data from logs (using wizards and filters) so that it gives you the maximum amount of helpful information.
She ended with advice on identifying what parts of the taxonomy need attention: "the top 20% (what works) and the bottom 20% (problem areas)".
How do you Measure a Taxonomy?
Ron Daniels, Taxonomy Strategies
Ron's presentation focused on qualitative and quantitative methods to evaluate the success and appropriateness of your taxonomy. He presented 5 major methods:
- Walk-thru (show & explain) to check appropriateness
- Walk-thru (check conformance to rules) to check consistency
- Usability testing (card sorting, etc.) to check ability to complete tasks
- User satisfaction (surveys) to check reactions
- Tagging samples to check fit with content.
He went over each method in detail, explaining how each can be accomplished and providing many helpful examples.
He ended his presentation with an informative discussion of cost estimation for taxonomies, showing how to evaluate the cost of tagging, as well as ROI calculations for a taxonomy project.
