Podcast: Play in new window | Download
Subscribe: Apple Podcasts | Spotify | Android | TuneIn | RSS
Manny SilvaTechnical documentation of software products is critical work that needs to accurately inform end users, regardless of how quickly and substantially the software might be changing.
Manny Silva has created the Docs as Tests, a system that integrates a set of well-developed practices and a collection of engineering tooling, to help technical communicators streamline and constantly improve their software documentation work.
We talked about:
- his work at Skyflow and his new book “Docs as Tests” and a new version of his Doc Detective software
- the variety of “docs as” strategies that have emerged in technical documentation
- the intent of Docs as Tests to validate the accuracy of documentation
- the principles that underlie his approach to documentation
- the truism that in software “something always breaks” and the way Docs as Tests addresses this issue
- the tools available to test documentation
- the origins of his Doc Detective tool, which integrates developer tools in a tech-writer-friendly package
- how he extracts content and code from documentation to test it
- the differences between his old and new ways of validating the documentation he creates
- some examples of his Docs as Tests workflows, as well as other organizations doing similar kinds of of testing
- a key insight in the development of how to present Docs as Code to other tech writers
- the support and excitement he has seen in the tech docs community for his tools and others’
Manny’s bio
Technical writer by day, engineer by night, and father everywhere in between, Manny wear many (figurative) hats. He’s passionate about intuitive and scalable developer experiences, and he likes diving into the deep end as the 0th user.
Here are a few things that keep him busy:
- Head of Docs at Skyflow, a data privacy vault company
- Author of Docs as Tests: A Strategy for Resilient Technical Documentation and the Docs as Tests blog
- Creator and maintainer of Doc Detective, an open-source doc testing framework
- AI development and experimentation
He’s always looking for collaborators on his projects, and he loves chatting with folks, so don’t hesitate to reach out.
Connect with Manny online
Video
Here’s the video version of our conversation:
Podcast intro transcript
This is the Content Strategy Insights podcast, episode number 213. As software has evolved from occasionally updated, shrink-wrapped artifacts to complex, continuously evolving online products, technical documentation practices have had to adapt. Manny Silva’s contribution to this evolving technical landscape is the idea of Docs as Tests, a set of practices and a collection of engineering tooling that helps technical communicators streamline, automate, and otherwise improve their software documentation work.
Interview transcript
Larry:
Hi everyone. Welcome to episode number 213 of the Content Strategy Insights podcast. I am really delighted today to welcome to the show Manny Silva. Manny is the author of the forthcoming book, Docs as Tests, and that’s what we’re going to talk about today. The subtitle of that book is A Strategy for Resilient Technical Documentation. Which he’s got a fascinating approach and we’ll talk in detail about that. He’s currently the head of Docs at Skyflow and he’s also the creator of a tool called Doc Detective that we’ll talk more about. So welcome, Manny. Tell the folks a little bit more about what you’re up to these days.
Manny:
Thank you for having me, Larry. Well, there’s the book that you already mentioned that’s going to be coming out shortly. I am working on a new version of the Doc Detective software to come out alongside the book. But otherwise I am handling things at Skyflow, new fun features coming out all the time. I own all of the documentation there, all of the error strings, really anything that’s text that it doesn’t come from marketing. But outside of job related things, I like to spend a lot of time with my kiddos and they keep me very busy. So lots of life happening all the time.
Larry:
Never a dull moment, that’s a good life to have. But you’ve chosen, in the midst of all that, to write a book. Which I just admire immensely that anybody who takes on that challenge. More power to you. So first I just want to set this up a little bit. A few years ago, I don’t know how long ago this came, the notion of Docs as Code that this theory that you should write documentation the same way you build the software ish. I probably am misrepresenting that. I assume that Docs as Tests might be a play on that, but tell me a little bit about the origin of the book and why you took all this time to write a book about it.
Manny:
Sure. So I’ll start with Docs as Code. Docs as Code is one of many Docs as strategies I’ve come across as a technical writer. Docs as Code means that we as writers adopt many of the engineering practices and tools for handling our documentation. We use version control to manage our content. We use continuous integration and continuous delivery to check that our content builds like it should and to push it live to our users. But there are other Docs as strategies, like Docs as Product is a way of interpreting documentation as a product that is delivered by the working group. Just like you would your web interface or given feature or whatever else it is that your company builds.
Manny:
Docs as Tests is another one of these that borrows from engineering. But instead of the underlying tools and practices for managing code porting over to managing content, it brings in the validation aspects. The ways that engineers make sure that their code does what they expect it to, we’re now porting over so that we can validate that documentation is describing what we intend it to, that it can work as written, that the procedures are correct and accurate against the products that they’re written to describe.
Larry:
Nice. And one of the things I really liked as I read the book was that this is like tech and platform agnostic, which makes sense because there’s so many ways that people can document software.
Manny:
Yeah.
Larry:
Like you just said, there’s a number of philosophies and theories and then there’s specific tools like DITA CCMSs or just a regular old CMS or probably a million spreadsheets out there that people are using. So that just points the need for a principles-based approach. Can you talk about the principles that underlie this approach to docs?
Manny:
Sure thing. I mean, you hit on one of them already. That Docs as Tests is platform agnostic, tool agnostic, format agnostic. If you use DITA, great, you can do Docs as Tests. If you do Docs as Code, you can do Docs as Tests. The tooling that you choose to implement Docs as Tests will vary by whatever constraints you have in your workflows. But Docs as Tests as a strategy works broadly. And as far as the application of it, there are a few core tenets that really help define what makes Docs as Tests work. The first is a pretty straightforward assumption that docs are inherently tests because docs are assertions that a tool is supposed to work a certain way, and I’ll give an example here in a moment, but assertions are inherently verifiable. They are testable. And so if docs are assertions and assertions are testable, that means that docs are tests.
Manny:
So when you reference a product behavior or an attribute like a button label, you make a literal, verifiable assertion about the product. When you write the procedural step click continue, you assert that there is a visible clickable button that has the label continue. This statement is either true or it is false, and it can be assessed by performing the procedure. In engineering, assertions are used in tests to verify code-based behaviors. Tests are assessed whenever they’re run, usually when the code changes. For content owners, our docs are our tests. Our product references are our assertions and our test runs are performing our procedures. And if we don’t test our docs ourselves, we are making our users test our docs for us. They get tested one way or another and it usually ends badly if your users test your docs.
Larry:
Yeah. No, I just had a thing yesterday where I was screaming at my computer, and screaming at probably your poor colleague at that software company. One of the things about this is everything changes so quickly now, you don’t buy software in a package and then a year later it’s updated. You already mentioned continuous integration, continuous deployment. So I assume that’s one of the drivers of this as well, just that relentless pace of change.
Manny:
Yes. Because I mean part of how this all came about was before Skyflow, I worked at Google. Before Google, I worked at Apple doing technical writing. And something always breaks, something always breaks. No matter how good your communication lines are with your colleagues, how frequently you check in, what procedures you have in place, inevitably something will slip through the cracks. A button label changes or your design team decides they want to change the shade of a banner from green to blue and it makes all of your screenshots go out of date. And so suddenly everything that was great before is no longer. And a user goes in to look at your docs and it no longer reflects the product interface that they’re interacting with, which undermines their trust in your docs, in your product, in your company as a whole.
Manny:
We’ve all had experiences with docs that were inaccurate. By making sure that our docs are accurate, by actively testing our docs against the product interfaces, we not only avoid undercutting trust, we actively are building trust with our users. Because as they see that, oh hey, time and again these docs work the way I expect them to, that continuously builds trust in your product and helps retain customers and gain new customers as prospects realize that they can trust your product helps with maturity and overall perception of everything that you’re building.
Larry:
Yeah. And trust seems like it’s a core thing that any product needs nowadays. So this is a tech-agnostic approach that you take. But what you just described that notion of, for example the color changing, there’s just sort of like a whole color palette in a product changing and then you have to update that. How much of it is communication between the product team and how much of it’s actually looking? Are there multiple parts to it, like both communication and actual testing of the product itself? Tell me a little bit about how you actually validate those assertions.
Manny:
So how you go about doing it depends on a few things. It depends on your product interface and it depends on what tooling you choose to adopt. So for example, if you have a web-based interface, then you have a UI. Great. You can do things like use a tool like Doc Detective, though there are other tools like Selenium, Playwright, Cypress that can be used for web automation, and you can step through your procedures. Write a test so that when you run one of your tools, either one-off or in a CI pipeline, it automatically spins up a web browser, it navigates to the correct web page, it tries to click the buttons, or to go with the color palette change, it takes a screenshot. And some of these tools can do visual regression testing. So great, it captured a screenshot. A screenshot for this procedure already existed, let’s compare the two and see if there are any differences. And if so, are they above an acceptable threshold of change? That way if green changed to blue pretty drastically, then, yeah, let’s flag that and let’s automatically update the screenshot for you.
Larry:
Nice. I’ve been doing this so long, I keep forgetting how powerful computers are these days. And all of these things like version testing or comparing two images or things like that, these little tools, are the technical implementations of this, are these kind of modular architectures where you have the little image comparison thing, the text evaluation thing, the other thing, or what does the tooling generically look like?
Manny:
Generically, frankly, it’s still mostly at the engineering level. There are small tools that can do each of these different things in isolation, but they’re intended for developer audiences more often than not. Engineers have been doing this sort of content, not content testing, they’ve been doing this code testing, this regression testing, one way or another for decades and they’ve been building the tools for themselves. But when I would complain to my engineering colleagues about, “Oh, hey, my docs went out of date because you changed something. You have this cool tooling, where is it for me?” And they’re like, “But, Manny, that’s why we have you here.” And it’s like, “No it isn’t.” That’s where I got the idea for Doc Detective in the first place. To be able to knit together these more developer-focused tools into something more cohesive that a non-developer would be able to use. And I’m not the only one starting to make tools like this. There are other tools that are crossing that threshold from developer to non-developer.
Manny:
But, by and large, mostly you either have to be a developer to use these individual tools or you have to find something like Azure Innovation Engine or Doc Detective that’s meant for more non-developer audiences.
Larry:
Right. I wonder, is that kind of thing more… Well, there’s a couple of things going on there. One, a lot of tech writers I know would just go, dang, he’s got those tools and there would just be some lingering envy and resentment or something like that. But you’re like, no, damn it, I’m just going to make this work for us. And so I love that you did that. But was that any easier now than it might’ve been like five or 10 years ago? Is there better tooling and engineering procedures that you could bring to bear or did you just have to plow through it like any software engineer?
Manny:
I mostly just plowed through it. It helped that I mostly did it to maintain my sanity. I was at home on paternity leave for my youngest and doing sleep training for my middle child, and that means that there was very little sleeping had by much of anyone. And so in the dead of night, five minutes between going back and comforting children, I had to have a project to work on. I couldn’t listen to anything. I couldn’t read anything that required synchronous thought. And so I decided I would try to tackle this problem. And in five minutes spurts over the course of five weeks, I came up with a MVP of Doc Detective and it actually worked. So I’ve been working on it now for about three years.
Larry:
Very cool. And was it pretty much like a parallel, the engineering tools that you had been envying, or was there a lot of extra work to be done?
Manny:
There was a lot of extra work to be done, especially as time went on. Because for some things there were pretty clear parallels. Like cool, I want to do web UI automation. Great, I can use Selenium and other tools to help me manage all of that. And it’s like, all right, cool. I want to click a button, let’s find the button, let’s click it. That’s easy. But then if you’re doing things like making API calls. Okay, well there are HTTP libraries to make these calls but that’s still really complicated. How do I simplify that? How can I make the interface easier for a non-developer so that they don’t feel overwhelmed by trying to use this? I mean, don’t get me wrong, there’s only so much you can simplify an API call.
Manny:
But similarly, if you’re trying to run just a CLI command, what does that look like? What’s the easiest possible interface to make that work? Or oh, hey, you have some hardware that you need to test. What does that look like? How do you test your hardware and its interfaces? And how do we simplify that process? So it’s been a lot of, cool, I’ll take this cool engineering tool from over here and that one from over there and that one from over there and I’ll simplify all of the interfaces and put it into a single package so that content authors of any stripes can use this toolkit to be more confident in their docs. And the next step is simplifying it further. Instead of forcing people to write tests, how can I take the content that they’ve already written, ingest it, and turn it into tests for them so that a literal procedure written in Markdown is testable on its own with minimal additional markup.
Larry:
Interesting. So like no-code for the end user, low-code implementations of this, that’s really… Well, I want to back up just a little bit because you mentioned at least three interfaces that I caught so far, like a web GUI kind of interface, an API call, and then a command line interface, a CLI. Those are each different sort of computing scenarios and I’ve seen plenty of documentation of each of those. But I’m trying to picture can you always, in good documentation, discern a one-to-one between an assertion in the docs into a piece of functionality that you want out of each of those interfaces. I’m picturing like a matrix now, I think, as I asked that question.
Manny:
The way that I think about it is how we write our content says a lot about how we interact with our products. If you are following a style guide that says, hey, use emphasis or bold text styling to represent on-screen text, like a button label, and you have the word click that appears before a string surrounded by markers for on-screen text, then that’s a pretty clear indication that you’re supposed to click whatever element is represented by that text. That’s not going to be a CLI command, that’s not going to be an API call. That’s pretty obviously a UI control of some sort. Whereas if you have a code block that is labeled as a bash command, then great, that’s going to be a CLI command that you need to run. If you have a code block that has a curl command in it, that gets a little bit more tricky because that could be run as a bash command but the intent of it is to make an API call so how do you interpret that one. That could go one way or the other.
Manny:
But then you have things like, great, you have a JavaScript block in your documentation. That’s pretty obviously something that you want users to run or you’re trying to provide as an example. And so one of the things that I’ve been able to do with recent versions of the software of Doc Detective is be able to identify like, cool, let’s get all of these code blocks, extract them programmatically, put them in a script and run the script to see if it runs as written. So there are a lot of ways looking at the structure, looking at the context in which we write our documentation that we can infer where and how it’s supposed to be run.
Larry:
Now I’m wondering if there’s a feedback loop from this philosophy back into how you write the docs in the first place.
Manny:
I mean, I think there is. Because if you have a consistent style guide that tells you how you are supposed to refer to certain elements and then you write a config with whatever tool you so choose to use that can pick up on this sort of markup, then not only do you have your style guide talking about how your content is discussed with verbiage, how it is represented visually, but you have it tied together with your testing functionality so that your tools know exactly what is operable and what is inoperable and know what to test. It all can work together harmoniously. It takes effort, just like maintaining any good style guide, but this is just another kind of tooling that can leverage it to lessen your workload.
Larry:
Yeah. Well, I mean that’s kind of the whole point of this is automation-
Manny:
Yes.
Larry:
… that’s the core benefit here. Can you talk a little bit about how has your workflow changed from five years ago or whenever, before you started doing this, how would you validate and test? Because you were probably doing the exact same thing, like, oh, I have asserted in this documentation that this will happen, let’s go check that, to automating that. And is it 100% automated, 90%? How has it changed over that time?
Manny:
Sure. So before what I would do is what most content authors do. I write a procedure, I write a guide, whatever, and then if there’s a procedure, I step through the procedure myself to make sure that it works. And when it works, I publish it and all is right with the world. And then I try to go in as often as I might to do content audits to make sure that everything is still properly representing the product. But most organizations do comprehensive content audits about once a year, if you’re lucky. And then usually if it’s in less than a year, I find out that something’s broken from a customer report, which is not great and takes a lot of investigation of who broke what, when, and why, and why wasn’t I notified of the upcoming change. Now I have almost all of my procedures and content, UI and API and SDK, tested every day. Every day via CI.
Manny:
On a daily basis, it runs all of my procedures against my testing environment in production, and if something breaks, if a screenshot goes out of date, it will automatically create an issue for me and give me as much contact as it has. So it shortcuts my research and investigation. And if it was a screenshot that went out of date, it captured a new screenshot and includes it with the report. That way, if I need to update the screenshot, I already have the new image. And that all happens before I wake up. So I’ll wake up in the morning with, oh, hey, here’s this new issue. Here’s the pull request to automatically merge any new media that was captured, if I so deem that I need to. And then I just get to evaluate what broke. I still have to do that investigation of is this a product issue, is this a docs issue, but it shortcuts so much time. And once a day is a lot more frequently than once a year.
Larry:
Yeah, no kidding. And the way you just said that, I’m kind of picturing… Did you say that it not only flags the issue but it sort of creates a ticket for you essentially, but it’ll also grab some of the assets or point to things that need attention? Is that…
Manny:
So Doc Detective specifically, which is what I use, which by the way is open source. It is not proprietary. Anyone can use this. Doc Detective can capture screenshots. And so if it captures a screenshot, it can automatically create a GitHub issue or it can create a pull request on a GitHub repository that includes any images that it captured, any changes that it detected. And so I can just say, cool, it created a new screenshot. Oh, it created a new recording of this step execution that it converted into a GIF that needs to be updated. Great. I can just hit merge PR and it goes. And my media is automatically updated for me. Updates content. Failed content tests like button labels, those usually take a little bit more investigation. But I mean, any sort of test breakage is cause for investigation.
Manny:
That’s the whole point. These things aren’t supposed to be breaking. And there are some times when you might anticipate breakages, like when a new build of the product is being released, but there are solutions to that. Every push into a staging environment, you can run all of your tests against your staging environment to find out what breaks, to make sure that you caught all of these intended changes before it goes to production and to customers. And if there’s any unintended changes, then it gives you the opportunity to flag that with your developers before anything ever gets released. If you run your tests against a development environment, you can catch any and all issues before it even gets to staging.
Larry:
And you’re working at all those stages of development of a product too. So it’s helping you across a lot of the span of your job it sounds like, not just live documentation.
Manny:
Yes. I run tests against production and staging today. I haven’t quite gotten to the point of running it against the shared development environment every day, but in the near future I hope so. And, yeah, it’s helped me catch subtle changes to APIs. It’s helped me catch unexpected changes in the UI. And it’s been very beneficial overall.
Larry:
Nice. And you mentioned a couple of times some of the screenshots it’ll grab and other assets that it’s pulling. I gather there’s a lot of content there, obviously, that it’s working with. I’m curious where all that stuff is. This is platform agnostic, so the source content can be any number of places. But how do you manage all of the screenshots and things that are associated with it? Is there something like a CMS or a DAM or digital asset management system or something that goes with this system?
Manny:
That’s just one of the tools that you can choose how you want to manage things? Personally, I run a Docs as Code set up. My images are part of my content, and so I’m able to just use GitHub issues and GitHub PRs to manage changes and merge changes. But depending on whatever tooling you use to do visual regression testing for your screenshots, you can make it work. Even if it’s just running it locally on your laptop and say, hey, there was just an update to the software, or I want to test this local build. Or, oh, hey, I have this new prototype of my hardware hooked up to my computer. Let me try running my tests against my product locally to see how it works. And then if it captures any necessary updates, then you have them locally and you can do with them what you will. But it’s not just screenshots.
Manny:
I can with Doc Detective make an API call and then store the result of that call to a file that I can then directly include into my documentation so I know for a fact that it’s accurate. And then if I make the same API call again in a subsequent test and the result that I get back is materially different, then it can flag that as an issue just like we could with a screenshot. Same thing with doing CLI commands. And so if I run a command like if I do Docker run “Hello World” and I store that output, I can include that in my docs. And then if I run it a second time and it gives me an error or it gives me something else, then it can flag that as a potential issue. There are people like Redpanda Data who do exactly that.
Manny:
They have Docker images that they test their CLI commands for, and that’s how they’re doing docs as tests today. They validate all of their CLI commands, and now they’re moving into doing UI tests to validate some of their UIs. Docker uses Playwright to step through some of their web-based UIs to make sure that all of their procedures are working as written. But there are other aspects to Docs as Tests too that people are doing out in the wild today. Like AWS tests all of, I’m not sure all but many, of their code snippets that they have for their SDK integrations to make sure that how they’re written, it runs. Raspberry Pi has done something very similar. But looking at the API sphere, there’s also contract testing. If you have something like an open API definition or an AsyncAPI definition, making sure that your examples actually match the schemas, that they match both a producer and consumer expectations of making and receiving API calls.
Manny:
But there’s also workflow testing, something like the Arato specification, and being able to test that the business cases that you are telling your users they can do, that you document in your content, work as expected. Whether it’s a series of API calls, whether it’s knitting together multiple features in your product end-to-end, or anything else. All of this is testable. People are testing it today, many times using these homegrown solutions because there hasn’t been anything comprehensive that worked for everybody because everyone has a little bit of developer expertise in one area or another. But we are just reaching an inflection point where tools are able to help the average non-developer content author be confident in their documentation. That people, without having to know how to use Playwright or Selenium or Cypress or Karate UI or whatever else, can still be able to validate their procedures. That their API calls can be testable and validatable regardless what ad hoc platform they use, whether it’s Postman or Bruno or whatever else. That there’s actually a CLI testing suite available that isn’t locked in to whatever testing platform their developers use.
Larry:
That’s cool. As you’re talking now, your reminding me … to circle all the way back to the start of the conversation that it feels like a lot of this was discovering what people are doing out there and just discerning and extracting and articulating the principles that guide all these things and then coming up with a methodology to tie it together. Does that make sense?
Manny:
Yes.
Larry:
Yeah.
Manny:
And that’s actually how Docs as Tests came about because I created Doc Detective first because that was my attempt at solving the problem. And then I had something that was a early but workable solution and I tried describing it to other tech writers, but they didn’t quite get it. They didn’t have a tool category that it fit into, so I had to take a step back. It’s like, no, this is bigger than my little project. How can I help people understand what this is? How do I even describe what this is? And that’s where Docs as Tests came from. I saw all of these cool things that people built, all of these wonderful ways in which people very creatively validated their content. And Docs as Tests came about. It’s like, no, people are testing their docs. They are treating their documentation as tests.
Manny:
That’s really what I’m doing here. And as soon as I started describing it to people in that way, then they got it. That immediately made sense because content folks, we learn from everyone around us. Whether we’re learning from engineers with Docs as Code, whether we are learning from product and product managers with Docs as Product, whether we are learning how to better present our content, learning from design, or trying to take a more educational and security-focused aspect, learning from our security friends, we learn from everyone around us. So by describing this as a testing solution and bringing forth the QA value that content authors already have, we often act as informal QA anyways, but more formalizing it, it really empowers people and they understand that empowerment. I see it in their eyes when I describe it to them. And it is wonderful to see.
Larry:
That’s awesome. If I could sum up this whole podcast, it would be that. People getting excited about each other’s work and helping each other. That’s what I’m here for. But, Manny, I can’t believe that we’re coming up on time, but before we wrap up, is there anything last, anything you want to revisit from the conversation or that you just want to make sure we share before we close?
Manny:
I just really want to reinforce that Docs as Tests, it’s format agnostic. It’s tool agnostic. You might have to be a little bit creative with how you apply it, depending on what tools are or aren’t available for how you do your work. But this could work if you’re using DITA. It can work if you’re in Word. It can work if you are in Docs as Code or anything in between. And there are lots of people who want to help. We all want to share our knowledge, we want to share our expertise. And everyone I have spoken to over the last three years as I have figured out this little corner of our world has been so supportive and excited about this. And so let’s talk, let’s collaborate. Let’s make this better for everyone. Part of how I’m doing that is Doc Detective is open source, many of these tools are open source, so if you’re interested in any of these communities, obviously you’re welcome in mine, but for any of these, feel free to get involved. The people there will be more than happy to have you and will be very welcoming, I’m sure.
Larry:
Yeah. This is what I love about this community is that that attitude is so pervasive, but hey, and to that point, if folks want to connect with you or follow you online, what’s the best place to find you?
Manny:
Please follow me on LinkedIn. I like to think I’m pretty friendly and I like to chat. That’s where I’m most active. You can also keep up with me at docdetective.com, which is where I have most of my content for that project. There’s also docsastests.com, which is a blog that I have to talk about all sorts of different ways to apply Docs as Tests. And there’s the book that’s coming out soon. So any and all of those channels, I would love if you could join me there.
Larry:
Fantastic. I’ll put all that in the show notes as well. Well, thank you so much, Manny. This was a really enjoyable and edifying conversation. Thanks.
Manny:
It was wonderful being here, Larry. Thank you for having me.
Leave a Reply