Jeff Eaton: Content Observability in Complex Systems – Episode 212

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Spotify | Android | TuneIn | RSS

Modern content systems are complex and abstract, presenting problems for managers who want to understand how their content is performing.

At Autogram, Jeff Eaton and Karen McGrane have developed a content observability framework to address this complexity.

Their framework evaluates the composition, quality, health, and effectiveness of content programs to help enterprises measure the return on their content investment.

We talked about:

• his work at Autogram, the consultancy that he and Karen McGrane operate
• his high-level take on the notion of content observability
• how the growing complexity of content systems drives the need for content observability
• how content observability connects with content strategy
• the inadequacy of current analytics and other tooling to permit true content observations
• the role of content intent in discerning content performance
• the content ecosystem insight that led to their explorations in content observability
• the four pillars of Autogram’s content observability framework:
  1. composition – the make-up of your content assets
  2. quality – organizational standards, regulatory compliance, voice and tone, etc.
  3. health – is the system working (regardless of the quality of the elements in it)
  4. effectiveness – is the content achieving the intended outcomes
• the ability within the framework to account for different content intentions in order to evaluate the ROI of the whole system
• some of the inspiration for his content observability work
• the talk that he and Karen are giving on content observability at the 2025 IA Conference

Jeff’s bio

Jeff helps large organizations understand, model, and manage their content. Whether he’s fixing problems with CMS architecture or editorial workflow, his solutions sit in the overlap between design, communications, and technology.

Connect with Jeff and Autogram online

• LinkedIn
• Bluesky 
• Eaton, FYI
• Autogram

Video

Here’s the video version of our conversation:

Podcast intro transcript

This is the Content Strategy Insights podcast, episode number 212. As content systems have become more complicated and abstract, understanding the effectiveness of your content efforts has become a real challenge. At Autogram, Jeff Eaton and his business partner Karen McGrane routinely work on very complex content projects. To help their clients understand the impact of their content, they have developed an observability framework that measures the composition, quality, health, and effectiveness of their content programs.

Interview transcript

Larry:
Hi everyone. Welcome to episode number 212 of the Content Strategy Insights podcast. I am really delighted to welcome back to the show, Jeff Eaton. Jeff, I think is one of the few three-time guests I’ve had. Maybe Preston So was the other. That’s right. Might have had one other, but welcome back, Jeff. Oh, and for folks who don’t know, Jeff is a partner at Autogram, the legendary consultancy. And he’s also probably, it’s safe to say the most famous, infamous, renowned content nerd out there. Welcome back, Jeff.

Jeff:
Well, it’s a pleasure to be here. Always fun to come and talk shop and exchange news about what wild stuff we’ve all been working on and thinking about. It’s great to be here.

Larry:
No, and one of the things, the reason I wanted to get back on, we were talking a while back a few weeks ago about this notion of content observability, which immediately to me was like, yeah, thank God. Let’s do that. And thank God Jeff is thinking about it. But I think a lot of folks, it’s a pretty well-known concept in the tech world, especially the dev ops and those kinds of worlds. But I think to a lot of our listeners, it might not be a familiar concept, so can you walk through the notion of observability and its application that you see it in the content world?

Jeff:
There are a lot of different tactical things that we tend to do in the content world to measure what we’ve got or what’s going on, whether it’s auditing, inventories, analytics, stuff like that. But the concept of observability, that’s a label that became popular in the software engineering world, particularly as organizations started depending more and more on large numbers of different systems. Like, oh, well, our company’s web presence isn’t just the server that runs the website, but in fact it’s, oh, well, these three databases, these four services that we subscribed to, that our servers talk to. And there’s the front-end stuff and there’s this and this, and this. Increasingly, the web and digital infrastructure for companies has gotten complicated enough that it’s really a system of different moving parts, each of which have their own health checks and monitors and stuff like that.

Jeff:
And observability really emerged as an umbrella term for this idea of building a system that you can monitor in order to see, well, what are the different moving parts of the system? How healthy is each one? Is something going wrong? All of that is with an eye towards giving yourself good feedback mechanisms for the operation and the health of the system as a whole, and it’s individual parts so that you’re not left to saying, well, something’s wrong, somebody should go quote “debug” it or whatever. It’s with a system of that complexity and as those systems have more and more central to a lot of businesses, that proactive approach to saying, well, how do we keep an eye on how everything’s doing and put that as a part of our day-to-day processes for knowing what’s an emergency or when we should change the oil on the car, so to speak, versus just waiting for it to break down on the side of the road.

Jeff:
That oversimplifies things a lot, but that has been a significant theme in the software engineering world for quite some time. And it’s both at a strategic level like, what services do we need to monitor everything? But also at a just detailed tactical level questions like, well, how do we build those individual pieces such that we can monitor them effectively? That doesn’t work if it’s just a black box that either works or fails and who knows? The idea of being able to say, oh, well, we should be able to reach in and say, how is this piece working? What’s it doing? Is it slower than usual? Does that matter? Yes or no? Those are the kinds of questions that software observability needs to be able to concern itself with. And in the world of content with digital publishing, there’s a lot of overlap. There’s adjacency because that kind of complex system that we just talked about is how a lot of digital content gets published.

Jeff:
It’s going through systems like that, but the idea of the content itself, having that kind of observability framework isn’t something that has really been there. And I think as we’ve been feeling the pain of that for a long time on different fronts, but haven’t necessarily had a word to put to it. I can’t remember where I picked it up, but a while ago I was watching someone talk about the concept of strategy and how all strategy is a theory. It’s a story about what we believe is going to happen if we do X and why we think Y is going to happen. Because it’s a story we have about, okay, if we do these things, here’s how the ecosystem works, or here’s how our customer relationships works, so if we do A, then because B and C, we’re going to get D on the other side of that. And you can do all sorts of stuff from that but at its heart a strategy is that kind of a story you have about here’s what’s going to happen, the cause and effect.

Jeff:
The problem is, is for that to work, you need to be able to check back and figure out, okay, so was that story true? Was that an accurate view of the world? Because if it’s not, we’re doing the wrong stuff, we’re making the wrong content, we’re publishing it in the wrong channels, we’re not getting it to the right people, or they’re liking it, but it’s not causing the behaviors that we had hoped it would or grabbing them in the way that we anticipated. And oftentimes it’s a long, long feedback cycle before we really see whether our theories are true or not. And even then it’s often with rough proxy metrics that aren’t really telling us whether our theory of the world is true or not.

Larry:
That’s what I was going to ask about because all we’ve ever had as content people is page-level delivery metrics, and it’s like, oh, for crying out loud, but what you just said, I want to-

Jeff:
Views, how many views and what’s that bounce rate.

Larry:
Like age views. But I love the way you described this as like, hey, does this story check out? And I’m picturing a magazine fact-checker looking at your strategic narrative and going, that doesn’t really check out. And so that gets into-

Jeff:
We thought long form video content was going to really be a killer for us. How did that work out? Did that do what we thought it would?

Larry:
And you can’t do that with Google Analytics, right?

Jeff:
Well, Google Analytics is incredibly flexible. If you take the time to go in and set up all kinds of custom triggers and events that you’re tracking and stuff like that, you can track all sorts of things. But out of box, Google Analytics is set up to track page views and bounce rate. And if you’ve got commerce stuff wired up, you can track conversions in very specific ways. But those are very, very tailored kinds of conceptual questions. Like bounce rate is basically when someone comes to the site, do they stick around? What are the odds that they’ll come look at what we’ve got and leave? And this was the germ for us at Autogram of what I ended up developing into this idea of a bigger concept of observability. The classic bounce rate question was, well, what about wayfinding content? The stuff that you make, if you build a large repository of information, there might be some that’s like, this is the stuff that’s top of funnel. It’s like things that people search for on Google and they get to the site.

Jeff:
And then there’s certain closer content that, this is to get them to sign up for an account or to purchase from us or make a call or something. But at a certain scale, there’s other content that’s basically about getting people to the right place, helping them find what they need, saying that, oh, well, once you come in and you read our blog post on the best ways to do X, well what gets you from there to the other stuff? Because every page can’t do everything for you. Well, but once you’ve got a page like that, a wayfinding page, bounce rate doesn’t work for it. You want somebody to come, get to where they’re going as fast as possible, and that’s a success. But that means that any page that’s designed to route people effectively is going to be measured a failure if you’re keying on bounce rate. Now, for other types of content, bounce rate may still be a good effective measure, but even for direct support materials, if someone comes gets their question answered in under a minute and leaves happy, that’s a bounce.

Jeff:
How do you distinguish between the different needs of different kinds of content? And thinking about that stuff was one of the early parts of this that really got us rolling of-

Larry:
A lot of what you’re saying I can kind of picture, like you said, you can tweak Google Analytics a lot, but at the same time it’s like you got to do the intellectual heavy lifting ahead of time or have an organizational operational mechanism for the intent of any one content thing you’re doing.

Jeff:
And we’re not super happy with the label, but intent is what we’ve been calling that, like the idea that you have to capture what are the different intentions you have for the kinds of content that you’re building. It’s like the jobs to be done framework of what things do people want out of the content when they come to a site or they come to a platform or whatever, what are they hiring that content to do for them? But the other side of the equation is, what does an organization believe content is going to accomplish by being out there and what are the things they’re going to look at to determine that it’s actually accomplishing that goal? And I think another example we’ve used is legal compliance content, the terms of service, the privacy policy, certain kinds of fact sheets for products that are basically there so that, oh, you can always get the full details of like, oh, here’s all the open source software that we use when we produce it. Here’s all the parts, here’s the FCC. Stuff, if you’re selling physical products or whatever.

Jeff:
None of that is conversion content. None of that’s stuff that people are like, “Woo hoo, I’m going to go download the product fact sheet.” But it’s all there and you need it to be there. And sometimes the measure of whether it’s succeeding is things like, we didn’t get sued this year. High five, everybody. All that legally required stuff is there, but it’s critical, so how do you assess that? Even if you have say maybe half a dozen different intentions for your content that you’ve mapped out, none of them lives in a vacuum. They work together, they complement each other. Things like trust-building content, educational materials that may not directly land a sale, but maintain your company’s identity as a trusted advisor rather than just a vendor. And then product oriented features that are being talked about as solutions to specific problems.

Jeff:
That’s like, oh, well, when a user has a very specific need, we have content that its goal is to let them know a particular product we have can fill that need and get them on the route to perhaps a sale. That’s a different kind of goal and a different kind of purpose. And if your trust-building content leans too heavily into we got to do the hard sell on our new product family, it may not accomplish what it’s supposed to be doing. And similarly, if you’re, yeah, this product, it’s going to nail that. If it doesn’t land that selling point effectively, people may like to read it, but it may not be accomplishing what it needs to do. And all of those things work together in a holistic fashion. Your content compliments each other. And for us, this idea of observability really started percolating when we were looking at a couple of specific clients like ecosystem of content from that standpoint and saying like, well, okay, we can start describing all of your stuff as this big picture of content that’s supposed to work together.

Jeff:
Different types of stuff, have different jobs for different audiences, different ways of being measured, their effectiveness, but what would be necessary to keep an eye on the system? If you start thinking about all of your content working together as a system in the same way that modern software infrastructure has lots of different moving pieces, lots of different moving parts, each doing important jobs, but maybe succeeding or failing or being healthy or unhealthy, wildly successful in this particular area, but how do you look at the big picture and zoom in when necessary? What’s necessary for that, with content? And that’s where this idea of content observability hanging on that concept from the software world really started coming together for us.

Larry:
And I love that you’ve… Because you’ve looked at that a lot, and that’s a pretty well established practice in the dev ops world and places like that. But when you look at it, applying content to it, I love that you’ve developed a framework. My favorite thing about this podcast, I don’t play favorites, but I do love my people who come with frameworks. And I’d love if you could talk a bit about how you’ve thought this through, and I think for people who want to take something home from this and start thinking about how they might implement better observability in their content ops, I’d love it if you could share that.

Jeff:
Again, we’re not trying to do a design thinking and turn it into some sort of trademark framework or whatever, but we feel like there’s really four key pillars to this observability puzzle for content. One is the idea of, and again, if anybody out there has a better idea for a label for this, I’m all ears, but the idea of composition. I think this one in particular, for those of us who deal with structured content and content modeling and content architecture, this is very familiar. It’s the idea of what are the assets we’ve got content wise? What types of content do we have? How much is there? How much of it is ephemeral comments posted on our public blog posts and how much of it is core assets that are actually stored in our dam versus our CMS? Do we have different kinds of content?

Jeff:
What are its properties and structures? If we’ve put a bunch of energy into making it possible to have say, runnable code samples in our documentation website, well, how much of our content is actually using that? This actually comes up a lot when we look at people who start to use page builder type tools where you can use small little components to assemble new kinds of content arbitrarily. They don’t have any way to look into their content and say, well, how many places are we using three-up call to action blocks? How many places are we pulling in cross references and dependencies? What is the makeup of our content? How does it reuse stuff or depend on other assets? Who’s making it? Are there different departments that are responsible for more or less? That meat and potatoes stuff that a really deep inventory or audit might reveal, that’s what we think of as composition. It’s both the architecture and then the big picture of, what is it all?

Larry:
And so that’s where the metadata would come in, it seems like-

Jeff:
Yes, exactly.

Larry:
… of, what is this thing?

Jeff:
And some CMSs, it’s possible to build out dashboards for that kind of stuff, but if you’ve got three or four CMSs, then it gets orders of magnitude complex, but keeping an eye on that stuff as real time as possible is a huge advantage. But beyond that, even if you have a perfect view of all that stuff, there’s still other questions that I think are important. The second leg of the table so to speak is quality. Does this content meet our standards? Has it gone through legal review? Does it use our brand and editorial guidelines? Is it accessible and audience appropriate? Are we using the right voice and tone? Did we decide we were going to do a bunch of amusing community building DevRel videos talking very casually to a particular audience about our new capabilities. But in actuality, what we produced were tutorial videos like, it’s not just good quality or bad quality, but is it actually doing what we think it ought to be doing in terms of quality tone voice standards?

Jeff:
That’s the quality question, and there’s loads of tools out there that can measure and interrogate that stuff, but that’s one pool of information. The third leg is health, and this is easy to confuse with quality, but we think of health as like, is the system in working order? Is the content working? Whether it’s low quality or high quality, is it showing up to users? Are there errors once it appears on the final website or in an RSS feed or whatever? Are there things that are out of date that are showing up? Are there pieces of information or are there broken images? Are we publishing this stuff? But the rest of our content system isn’t properly getting people to it thus they never see it and never benefit from it. Health can also be an important view of before it’s even published, how long is it taking us?

Jeff:
A lot of process and governance stuff I would say also touches on health like, how long has it been since this particular pool of content has been audited or reviewed? Do we need somebody to go back and check it? I think of health as the check engine light of the content system. It’s like it may not tell you that it’s excellent content, but it can tell you, “Whoa. Hey, something might be wrong here. You should check this.” Or, yeah, we’re on course to get this campaign published because based on how long it usually takes us to go from initial creation to final approval and publication, this cluster of different pieces of content usually takes six weeks. We’re two weeks from our deadline and we’re almost finished, so we’re on target. It’s possible to build some of those systems out with different CMSs, but you really have to be intentional about it. It’s very rare for that sort of thing to just be automatic, certainly not integrated into the CMS experience in most platforms.

Jeff:
Then the fourth leg of the table from the composition of this content and the overall ecosystem of content, the quality of it, the health of it, how it’s doing. Then the fourth one is what we touched on earlier, the effectiveness, like we’ve invested time and energy in making this stuff and putting it out there. Is it actually achieving its intended outcomes? Is it answering audience questions correctly? Is it building trust? Is our support content helping people actually accomplish the tasks, or is it just keeping them from calling our call center? Stuff like that. And that set of four things, the composition, the quality of the stuff that’s being created, the health of it, whether it’s on an individual piece of content level or in aggregates, like, oh, this department’s content is a lot slower to hit the streets than this department’s, why is that? To the effectiveness of like, hey, this particular type of stuff is doing its job or this stuff it gets a lot of quote “engagement” but it’s not really getting people to the other parts of our site.

Jeff:
Its job is really not just to be engaging, but to get people going and looking at other stuff that has other jobs. That big picture, those four things really support each other in maintaining that ongoing view of, are we doing the right thing? Is it working? What are we spending in terms of time and resources and energy to do this stuff and is it accomplishing what we expect? Those four pieces all feel really, really… They’re tightly interlinked once you start thinking about how they work together.

Larry:
And as you talk about them too, it’s not hard to imagine the metrics, the mechanisms, the ways you might evaluate each of those against the criteria you set up. And one thing before we went on the air, we were talking, I asked as we were talking about that, about ROI, and that’s really… Because I thought, is that part of effectiveness? But I think we decided that it’s a bigger thing, that it’s about the ROI of something like this would be about the system itself, not-

Jeff:
Yeah. I think one example we have is a client that had… I think I’ve used the example of trust-building content and problem-solving content and essentially sale-closing content as three complimentary kinds of things that work together and measuring individual types of content that way, you can come up with those metrics like, oh, well, if educational content were being successfully education-ey, what numbers would we see go up? And if our sale closing content is doing that, what would we see happen? But when you start looking at it as a system, then you start being able to come up with questions like, well, okay, are people who come in and are drawn in by our educational trust building content, are they actually moving over to our problem solving content that points them to products when they have a detailed question? Is our theory about the path people eventually take and that the way that these types of content complement each other? Is that big picture actually true?

Jeff:
Is this kind of content drawing people in and this kind of content is engaging them with our stuff long-term and then this kind of content to closing the deal? Is that story we’ve crafted about how they all work together, is that true in addition to the isolated metrics we’ve attached to each different type of thing? And some of that stuff is, that is legitimately difficult. You’ve got to think hard about, well, how would we measure that? Some of it’s click paths, some of it’s much easier if they are willing to sign up and create an account and you can keep careful track of what things they do from one visit to another. Other times it’s harder, but when you start thinking about your ROI questions and your effectiveness questions in terms of those end goals, the things that you want to occur and the expectation, the story that you’ve told yourself about how each one of these things we’re creating and producing will end up contributing to that goal. You can start building measurements around those theories a lot easier than saying, well, what kind of bounce rate would we expect for learning content?

Jeff:
Starting with a giant list of possible metrics and just trying to figure out what would tell us the right number. It’s lot less effective than saying, what is our theory about what’s going to occur if we do X, and how can we start measuring that?

Larry:
Well, that’s why I like your framework because you have a giant bucket of stuff and you just, okay, is this about composition or is this about quality or is this… You have some places to start hanging it and organizing it. But hey Jeff, I can’t believe we’re coming up close to the time already and I’m glad we got through the framework though because the thing I really wanted to share and the ROI I think is important if people want to try to convince their boss to do this. But before we wrap, is there anything last or anything else you want to revisit from the conversation before we close?

Jeff:
I want to be clear, it looks like we’ve put probably a few years of chewing on this into it, but only recently really started calling it observability. Fabrizio Benedetti, he’s the lead documentation engineer at Tinybird. He wrote a blog post in January 2024 about documentation observability, talking about this specifically in the realm of documentation and technical content and what it would be like to bring observability to it. And that was, I think one of the things that really made the observability label click for us, so looking at the stuff that he’s talked about is great. Seb Barre, he’s the digital platform product lead for TELUS, a Canadian telecom company. He’s actually done some speaking and a lot of amazing work inside of TELUS about what we would probably call the composition side of it in composable, headless, and decoupled content publishing platforms. How do you add a layer of observability to what content do we have? Where’s it getting used? What would break if we were to delete this stuff? He’s put a ton of work into building those kinds of systems.

Jeff:
For the composition side, his work has also been critical, particularly in the headless and decoupled space where that’s not as commonly… It’s not as common to have that kind of insight. But there’s all kinds of different voices that are talking about different pieces of this right now. Karen and I are actually going to be speaking at the IA Conference at the beginning of May on content observability doing the speed run of explaining this framework and what pieces can be tackled inside of most organizations. There’s a lot of interesting stuff going on, and I’m just personally excited that there’s a hook we can hang these conversations on now as we’re starting to call it content observability that brings together some of these diverse conversations. And again, at least for us, a lot of it boils down to that idea that strategy is critical, but strategy is a story we tell ourselves about what we think is going to happen if certain things are done and why.

Jeff:
And observability for content is all about figuring out how do we figure out if that’s true as quickly as possible so that we can course correct or put more energy behind something that’s really working when our theories are confirmed.

Larry:
I love that you emphasize the strategy because this is the Content Strategy Insights podcast, so thanks for tying it into that. Hey, one last thing, Jeff. If folks want to connect with you or follow you online, what’s the best place to find you?

Jeff:
Just check us out at autogram.is, that’s our website. We’re also active on BlueSky, LinkedIn, stuff like that. Those are great places to find out what we’re up to.

Larry:
Cool. I’ll put those in the show notes as well. Well, thanks Jeff. It’s always so much fun to talk with you, but this was really a cool one.

Jeff:
Always a pleasure.