Eric Holscher - Posted in 2016

Questions after talks at conferences

2016-11-12T00:00:00+00:00

At many conferences, people allow the audience to ask questions after the talks. I want to argue that this is an anti-pattern in many ways, and some solutions that have worked that I recommend.

Issues with questions

There are two primary audiences that have issues with questions:

Speakers
The audience

Let’s start with speakers. Many first-time speakers that I know have an intense anxiety around having the audience ask questions. They think, “I am going to go up and give a talk, and then someone in the audience will contradict or embarrass me for lack of knowledge afterward.” Audience questions after talks are one of the biggest sources of stress for speakers.

Now for the audience. They have chosen to attend a talk to hear from a specific speaker about a topic they are knowledgeable on. If there are 250 people in the room, each minute of the talk is over 4 hours of combined time. When you offer up a microphone to anyone in the audience, you are now offering 4 hours of peoples life to an unaudited question and answer that likely only provides value to a small minority of attendees. This is not a good use of anyone’s time, and often audiences feel trapped in a talk room during Q&A time.

Better approaches

Here are a few different approaches that I recommend a lot more than “let anyone in the audience ask a question publicly”.

Speaker goes to the front of stage for questions

At my own conferences, Write the Docs, we have established the norm of not having full audience questions. After each talk we ask the speaker to come to the front of the stage, and then have a conversation with members of the audience with questions.

This achieves a couple beneficial results:

People are empowered to ask questions that are more specific to their situation, instead of trying to general them for a larger audience
The question asker isn’t given a “stage” to promote their own projects or ideas
The speaker isn’t worried about being “called out” in front of the full room
Everyone else in the audience is free to do whatever they want

I stole this idea from XOXO, but a lot of events do a version of this.

Questions to the speaker are moderated

Another approach I’ve seen work well is that the audience is allowed to ask questions, but they are moderated. This can be done in a couple different ways:

A #questions Slack or IRC channel where people can ask questions
Index cards handed out at the beginning of a talk and collected at the end

This allows one person to moderate the questions, and forces them to be asked in a direct way. It also removes the “I have a statement, not a question” problem, because all questions are filtered through an intermediary.

This has a few benefits as well:

People are still able to ask the speaker for clarifiaction/explanation on parts of their talk publicly, and it benefits everyone
The speaker knows they won’t get ambushed by the moderator
The moderator can blend the questions together into a narrative and group questions in a meaningful way

I’ve seen this work quite well at conferences like Django Under The Hood and PyDX.

Questions are your responsibility

As the organizer of an event, the way that you structure the event has a direct impact on people’s experience. Opening the room to questions and not doing any moderation is abdicating your responsibility as an organizer.

I highly recommend that you adopt a more equitable approach to questions at conferences, and make them more enjoyable for everyone involved.

Semantic Meaning in Authoring Documentation

2016-10-06T00:00:00+00:00

Semantic Meaning in documentation is the separation of what something is from what it looks like. What we mean and what we display are very different things.

We might want to warn someone in our writing, but we don’t want to think about how to display this. Writing with Semantic Meaning gives us this power.

Semantics

As an example, if we were writing documentation in HTML, we could warn a user with bold:

<b>
    Don't do this, it will break your system!
</b>

This has no semantics. Bold doesn’t mean “warning”, it is simply a way of displaying text. A better example in HTML is:

<span class="warning">
    Don't do this, it will break your system!
</span>

This allows you to write what something is, but not have to worry about what it looks like. Your designer might decide that warnings should be bold and red:

.warning {
    text-format: bold;
    color: red;
}

The important part is that you don’t need to think about what warnings look like, just that it’s a warning.

To go one step further, in reStructuredText we can do:

.. warning:: Don't do this, it will break your system!

Then Sphinx or some other tool will generate the proper HTML or PDF with styles for us. reStructuredText is abstracted away from all of the output formats, so you write in one format, and it transforms it properly into HTML, PDF, XML, or any other format it supports.

This approach allows your designers to work with a systematic and standardized set of class names, generated from the tooling. This keeps all your styles the same, and allows the tool to warn you if you try and represent something that doesn’t exist.

Value of Semantic Documentation

When you write documentation, you form a mental model in your head of the document you’re writing. When you use a powerful tool like reStructuredText, you can think in terms of warnings, references, classes, objects, and other powerful semantic constructs.

You start thinking in terms of nouns that can be represented in your problem domain. You can then encode this model into your document:

.. warning:: Make sure you :term:`instantiate`
             the Response objects before you use it.

You can read more about the :cls:`django.http.Response`
in our :doc:`/api/response` page.

In the above section, I thought Hey, maybe someone doesn’t know what instantiate means. I was able to link to the glossary with :term:. I didn’t have to look up the URL for our glossary and link to that. I didn’t have to think about how to style glossary references. I was able to simply write what I meant, and move on.

When you write with semantics, you can encode more of your mental state into the words you write. Conversely, if you write without semantics, valuable information about your writing is lost.

Semantic information also acts as a type of documentation for our writing. Similar to type systems in programming, they allow you to be explicit about what you’re talking about. When you write documentation about a Response object, it isn’t immediately obvious what that is. When you write about a :cls:`django.http.Response`, it is explicitly defined what you’re talking about.

Note

When you write documentation in Markdown, there is no clear way to represent semantic information. You can make something bold, but you can’t make something a warning.

Please don’t write documentation in Markdown.

Conclusion

Communicating with words is a much different skill than transferring communicating with design. In the process of producing documentation however, they are two sides of the same coin. We have to both write and display information for users, and make it easy for them to understand it.

As an author, you should only need to care about communicating knowledge with words. Writing with semantic meaning allows you to properly seperate communcation with words and design.

You should write in a format that gives you the most semantic meaning possible. This:

Allows you to focus on communicating information, not thinking about what HTML class you need for a concept
Expand your own ability to think about your writing in terms of semantic nouns, allowing you to better structure your thoughts
Allows tooling to raise errors when you try to reference semantic concepts that doesn’t exist (typos, etc.)
Give people updating your documents explicit information about what you’re documenting
Allows your documentation systems to crosslink information and provide a better experience for your user
Allows your designer to apply consistent styles to all types of information

When you have the ability to write with powerful semantic constructs, writing becomes easier and more powerful. If you want to be the most efficient and useful writer, you write in a way that preserves the most of your mental model while writing. You write with a tool that gives you semantic meaning.

A Selfish Appeal for Documentation

2016-09-24T00:00:00+00:00

Writing code is the act of building a mental model of a problem, and then translating that model into executable software.

Writing documentation about software is the act of serializing this mental state. You can think about documentation as JSON for your mental state while programming. Documentation allows you to dump the information in your brain, in a format that allows others to reload into theirs.

Reading source code is the most inefficient way of transferring mental state. It also only conveys information on what the code does, not why it does it. My favorite rule around code comments fits well here:

Document the why, not the how

By writing documentation, you are able to load the why of your code into someone elses brain. Or reload it back into your brain 6 months after you wrote it. This allows you to complete your job faster, and also stops you from making the same mistakes over again.

Do yourself a favor and write documentation for your software.

Funding Open Source with Marketing Money

2016-08-31T00:00:00+00:00

Often times as developers we see funding open source as a charity. We will give our personal money to projects we believe in. If we’re lucky, our company might have a matching program for our donations. This has proven not to be a sustainable way to support open source.

Funding Read the Docs

Over the years working on Read the Docs, we’ve tried a large number of ways to get sustainable funding. We’ve done:

Donations
Corporate fundraising drives
Support contracts
Contracting on documentation tooling
Training for documentation

However, at the end of the day doing this work takes away from the primary goal of providing a documentation hosting site. These are all side-businesses that take time and focus away from our primary goal as a project.

Charging money for our product doesn’t work, because our users are mostly working for free producing open source software. These are the last people we would want to charge money, because they are already working for free.

At the end of the day, we’ve settled on advertising as the way forward. We believe that advertising aligns with our incentives as publishers: the more projects and users we serve, the more money that we’re able to make. It also is the only viable business model that seems to work for large, free websites on the internet.

We aren’t building a traditional advertising business. We’re embarking to build a model that respects users, and we’re calling it Ethical Advertising.

Traditional Funding Sources

Saying that we’re building an advertising business to support open source isn’t overly interesting on it’s face. However, I want to look at this from a different perspective in terms of where the money comes from. This Ford Foundation report goes into much more depth here, but I want to cover it in broad strokes.

Historically, open source funding has come from:

Donation budgets from individuals, who are very likely to also produce open source themselves
Donation budgets from corporations, which are vanishingly small
Technical budgets from Engineering organizations within corporations

The last one is the least obvious. Fitting donations into your expenses forces OSS funding into something that appears to be a legitimate business expense. This has been in the form of “enterprise licenses” which provide no benefit, toothless “support” contracts, or other tricks to make the money come from another source in the budget.

You’ll notice a trend in the above funding sources:

They are generally one-time payments instead of recurring
They are the first expenses to be cut
They are generally less than 1% of total budget

Marketing Money

Open source funding has been attempting to scrape by with a tiny percentage of the overall company budget. In a lot of tech companies, the marketing and sales teams will account for up to half of the budget for the company. This money has been generally untouched by open source funding.

Focusing on doing advertising allows us to use this huge pile of money, and redirect it to fund open source software. We’re unlocking a source of funding that was traditionally closed off, and which has a lot more money.

Trying to create a new line item in a budget for “open source funding” is a pipe dream. We need to find a way to fit open source funding into existing budget items, and in a way that is legitimate and ongoing. We need to make sure that our funding isn’t on the chopping block at the first sign of a downturn at a company.

By providing value to marketers, we make sure that our budget isn’t the first thing to be cut. Instead of being a charity, we’re entering into a business relationship where both sides come out ahead.

Help Us Help You

If you work at a company that sells to developers and believe in the work that Read the Docs does, it would be great to send a link to your marketing folks. We have an audience of developers that almost certainly is interesting to them, and the money that might have historically gone to Google or Facebook will go to a worthwhile open source project.

We’re building an advertising platform that is ethical and open source. We hope that you will join us in this mission. We plan to lead by example and build a sustainable business model that respects user privacy.

Join us in this mission to make an advertising business model that works for open source, advertisers, and users.

The Power of Sphinx: Integrating Jinja with RST

2016-07-25T09:00:00+00:00

Sphinx is a super powerful tool. This has its upsides and downsides. One of the major downsides is that historically it has been built as a framework that allows users to do just about anything. This is great, except it also means that a lot of the specific value out of the modular design hasn’t been documented or made explicit to users. I’m hoping to address some of this power in a set of blog posts.

Today I’ll cover integrating the templating language Jinja inside your RST files. This is a really useful thing, and allows a large number of powerful display semantics inside of your RST files.

Jinja is also the templating engine that Sphinx uses internally for rendering HTML. This means you already have the library installed with Sphinx, so no external dependency is required.

The Extension

Most of the power of Sphinx comes from the ability to plug into any part of the documentation building process. Sphinx has an exhaustive list of Sphinx events, which I recommend reading up on. This extension will use the source-read hook, which fires when the actual RST file is read from the disk.

From there it is the simple matter of taking the source that has been read, and passing it through Jinja. Here is the full extension:

def rstjinja(app, docname, source):
    """
    Render our pages as a jinja template for fancy templating goodness.
    """
    # Make sure we're outputting HTML
    if app.builder.format != 'html':
        return
    src = source[0]
    rendered = app.builder.templates.render_string(
        src, app.config.html_context
    )
    source[0] = rendered

def setup(app):
    app.connect("source-read", rstjinja)

This simple extension gives us a ton of power. Let’s talk through one example.

Generating Docs from Data

A lot of times we have an external data set that we want to generate into our documentation. Let’s say this is a list of team members that is maintained in a remote system, and we want to include in our docs. If we can get that data into JSON, it’s pretty simple to add it into our project.

In our conf.py, or another Sphinx extension, we can do:

import json

staff = json.load(file('data/company-staff.json'))

html_context = {
    'company_staff': staff
}

This gives us the company_staff variable in our HTML context, which will be availabe from Jinja. Then we can generate our staff.rst file:

Staff
=====

{% for member in company_staff %}
{% if member.is_active %}
* `{{ member.name }} <{{ member.url }}>`_
{% else %}
* {{ member.name }} (Emeritus)
{% endif %}
{% endfor %}

Note

The Jinja templates will be rendered before the RST is processed.

Allowing ourselves to use Jinja inside of RST gives us a whole set of logic that isn’t available in the RST language itself.

This approach is incredibly powerful, but please make sure you don’t overdo it! Try to keep the Jinja logic simple, and only apply it to things that alter the display of the page.

Want more tips?

I love talking and thinking about the power of Sphinx. Along with blogging, I provide Sphinx documentation training for companies. We do half-day, full-day, and multi-day classes. Shoot me an email or check out our sphinx training website for more info.

An introduction to Sphinx and Read the Docs for Technical Writers

2016-07-01T00:00:00+00:00

Treating documentation as code is becoming a major theme in the software industry. This is coming from both sides, with developers starting to treat documentation as a priority alongside tests and code, and writers seeing a lot of value in integrating more into the development process. This marrying of cultures isn’t simple, but having the proper tools for the job makes both sides happy with the process and the results that get produced.

A lot of developer tools don’t work well for writers. Sphinx and Read the Docs are unique in this ecosystem, in that they have powerful features that both writers and developers want, in one convenient package.

Overview of the Ecosystem

Read the Docs is the largest open source documentation hosting site in the world. Open Source in this context means both that the code is open source, and the documentation hosted is for open source software. It is a developer focused platform, but it has most of the features that technical writers have come to expect in a tool as well. This blending of worlds makes it the best suited platform for teams that want both writers and developers contributing to product and API documentation.

Read the Docs is best thought of as a continuous documentation platform for Sphinx. Continuous documentation is analogous to continuous integration of code, which runs the tests on each commit. In practice it means that your documentation is built, tested, and updated on every commit.

Sphinx provides a documentation generator that is best-in-class for software docs. Sphinx documents are written in the reStructuredText markup language. reStructuredText is similar to Markdown, but much more powerful, as you’ll see in this article. Read the Docs provides a hosting platform for Sphinx, running a build on each commit of your repository.

As a writer who uses Sphinx, your day to day is writing reStructuredText in plain text files. You then build your documentation by running Sphinx on the command line. Generally it’s easiest to output HTML for local writing and testing, and then you can let Read the Docs generate PDF’s and other formats.

This article provides an overview of the features of Sphinx and Read the Docs, and enables you to evaluate them for use in your organization.

Introduction to Sphinx

Sphinx is the documentation tool of choice for the Python language community, and increasingly also for other programming languages and tools. It was originally created in 2008 to document the Python language itself.

Over the past eight years, it has turned into a robust and mature solution for software documentation. It includes a number of features that writers expect, such as:

Single Source Publishing
- Output HTML, PDF, ePub, and more
- Content reuse through includes
- Conditional includes based on content type and tags
Multiple mature HTML themes that provide great user experience on mobile and desktop
Referencing across pages, documents, and projects
Index and Glossary support
Internationalization support

It also has practical and powerful tools for documenting software specifically, including:

Semantic referencing of software concepts, including classes, functions, programs, variables, etc.
Including code comments in documentation output for many programming languages
Tools for documenting HTTP APIs
Extensions with the Python language
A vast array of third party extensions providing powerful new roles and directives

This article isn’t large enough to cover all of the power packed into this tool. But I hope to show enough to pique reader interest so that you can try these tools out and research their capacity for yourself.

Using Sphinx

reStructuredText is a powerful language primarily because the syntax can be extended. reStructuredText supports two types of extension:

Paragraph level (with Directives)
Inline level (with Interpreted Text Roles, or roles for short)

Paragraph Level Markup

Let’s see a basic example of a Directive in reStructuredText:

.. warning:: Here be dragons! This topic covers a number of options that
   might alter your database.

   Proceed with caution!

.. figure:: screenshot-control-panel.jpg
   :width: 50%

   An overview of the admin control panel.

Here warning acts as the name of the directive, and is the part you can extend for custom directives. In the figure, screenshot-control-panel.jpg is an argument, :width: is an option, and the rest is the content They enable customization of directives, and show the full power of reStructuredText.

Note

You’ll notice that Sphinx uses whitespace to denote where a directive ends. The “Proceed with caution!” is still part of the warning, and everything that continues to be indented will be part of that warning.

Sphinx ships with a number of powerful directives for documenting code. You can also write your own if you have someone on your team that knows Python.

Inline Markup

For extensibility inside of a paragraph Sphinx uses roles. Here is an example:

You can learn more about this in :rfc:`1984`.
It is implemented in our code at :class:`System.Security`

Here you can see rfc and class act as the name of the role, and then you can pass in a single argument. For the rfc role, it generates a link to the online reference for RFC 1984 with a text of RFC 1984.

The class role is where things get interesting. This acts as a reference to the class System.Security that is documented in your project, which is a hyperlink in the HTML output, but also a working link in PDF and other outputs as well.

You can implement your own roles with Python. This enables you to create powerful semantic constructs inside the actual markup you’re using to write documentation. If you worked building software at a toy factory, you can create a custom semantic sytnax for our own software:

Check out :jira:`199` for information on the :toy:`jump-rope`.
There is a fix in our :unit-test:`assert-jump-rope-length`.

These custom roles work in your .rst files, but also in any kind content that is pulled out of a comment in your source code too.

Now, let’s see how you take these sets of reStructuredText files and turn them into a document set.

Table of Contents Tree

Sphinx lets you combine multiple pages into a cohesive hierarchy. The toctree directive is a fundamental part of this structure.

.. toctree::
   :maxdepth: 2

   install
   support

The above example outputs a Table of Contents in the page where it occurs. The maxdepth argument tells Sphinx to include 2 levels of headers. This also tells Sphinx that the other pages are sub-pages of the current page, creating a “tree” structure of the pages:

index
├── install
├── support

The TOC Tree is also used for generating the global navigation inside of Sphinx. The index at the top of your project acts as the root of the navigation. The toctree is quite important, and one of the most powerful concepts in Sphinx.

More info on the toctree can be found here:

http://www.sphinx-doc.org/en/stable/markup/toctree.html

Working with Code

Showing code examples is a common task in all documentation. Sphinx is quite well prepared for this task. Let’s see how we can show a basic code example:

.. code-block:: python
   :linenos:

   import antigravity

   def main():
       antigravity.fly()

This markup displays the Python snippet with syntax highlighting and line numbers. The python in the above example is an argument to the code-block directive. The :linesnos: is an option that says to display line numbers for the code block. In the HTML output, it would look like this:

Sphinx doesn’t stop there though. It allows you to store your code examples in external files, and be included in your documentation for easier maintenance. This uses the literalinclude directive:

.. literalinclude:: example.rb
   :language: ruby
   :emphasize-lines: 12,15-18

The neat addition here is the :emphasize-lines:. This shows the lines highlighted in your output. This is quite useful for showing sets of code examples where a subset of the code changes. You can also specify just a subset of lines to show with the :lines: option, so you don’t have to manage multiple snippets.

There’s far more power to Sphinx directives than this article can show. You can see the full documentation for them on the Sphinx website:

http://www.sphinx-doc.org/en/stable/markup/code.html

Working with References

A powerful reference system is a large part of the power of Sphinx. You are able to reference arbitrary headings and paragraphs in your content, but also semantically reference a large number of programming concepts as well. On top of that, Sphinx includes intersphinx which allows referencing across Sphinx projects. This means that if you have multiple projects inside your company, you can still use semantic referencing across them!

A simple reference is defined like this:

.. _reference-target-name::

This is a bit of content.

This is how you point to the above reference, :ref:`reference-target-name`

Sphinx also includes a number of other semantic reference types. Examples of other semantic reference types that Sphinx provides:

:doc: for referencing documents
:cls: for referencing programming classes
:term: for referencing glossary terms

All references support intersphinx, which allows you to prefix your references with a specific project name. So if your user guide needs to reference your API documentation, you could do:

Check out the :cls:`api-ref:api.request`

Which would know to look in your api-ref project for the class. The name api-ref is arbitrary, and is defined in your project configuration’s intersphinx_mapping.

You see the complete set of references in the Sphinx documentation:

Cross Referencing Syntax - http://www.sphinx-doc.org/en/stable/markup/inline.html#cross-referencing-syntax
Intersphinx - http://www.sphinx-doc.org/en/stable/ext/intersphinx.html

Including comments form source code

Integration with code is one of the largest benefits of Sphinx. You can easily embed software comments from multiple languages, including Python, Java, and .NET.

Here is an example of embedding Python documentation for a class in your project:

Make a request
--------------

You can make requests using the :func:`api.request` module.
It makes HTTP requests against our website.
Here is the basic function signature:

.. autofunction:: api.request

As you can see, you can include generated content in the file that you’re writing by hand. This allows for building a narrative around generated code comments, instead of giving your users a separate User Guide and API Reference, which is often times just an alphabetical listing of code!

Documentation is best written by humans. Pulling comments from source code is valuable, but it should be a small part of a proper set of documentation.

You can read more about including code comments in the following documentation pages:

Domains, where the references are defined - http://www.sphinx-doc.org/en/stable/domains.html#what-is-a-domain
Autodoc, which generates docs from Python code - http://www.sphinx-doc.org/en/stable/ext/autodoc.html
Breathe, which bridges doxygen and Sphinx - https://breathe.readthedocs.io/en/latest/
AutoAPI, which bridges .NET and Sphinx - https://sphinx-autoapi.readthedocs.io/en/latest/

Tables

Working with tables can be the bane of anyone who uses plaintext markup languages. Most other languages require that you write them in the file with some arcane syntax. However with reStructuredText, you can use directives to make this much easier.

You can use either of two powerful list directives, csv-table and list-table. Here is an example of csv-table:

.. csv-table:: Frozen Delights!
   :header: "Treat", "Quantity", "Description"
   :widths: 15, 10, 30

   "Albatross", 2.99, "On a stick!"
   "Popcorn", 1.99, "Straight from the oven"

This shows the inline markup, however the CSV can also be managed in an external file. This allows you to manage your complex tables in a third party tool, and have your documentation consume them from a CSV which is a much nicer workflow.

Internationalization

Sphinx includes support for translating documentation into multiple languages. Since sphinx knows the structure of your documents, it is able to generate a translatable strings split by each paragraph, heading, or figure.

Sphinx internationalization works using the gettext system. It ships with a builder that allows you to generate a catalog:

make gettext

You can then use that catalog as the base translation, using any tool that supports gettext. I recommend using Transifex, which gives you a web-based system for translating the documentation:

You can then tell Sphinx what language to generate for its documentation when you build it by setting the language setting. Read the Docs also supports internationalization, allowing you to host multiple languages of your project documentation.

More information on Sphinx internationalization can be found here:

My blog

Sphinx is quite versatile, which means you can use it for a lot of different use cases. I use a package called ablog for hosting of my blog over at http://ericholscher.com.

The most basic usage allows you to specify that a document is a blog post with the post directive:

.. post:: 2016-03-15 09:00
   :tags: writing, stc, sphinx

post adds the document to the set of posts. You can show a post listing by using the postlist directive:

.. postlist::
   :tags: stc

This shows some of the magical things you can do with Sphinx’s extensibility. If you’re curious, this article was actually written in reStructuredText and then exported to Word for publishing. You can see the full source here: https://github.com/ericholscher/ericholscher.com/blob/master/site/blog/2016/jul/1/sphinx-and-rtd-for-writers.rst

Custom builders

Sphinx supports custom builders to perform tasks beyond providing basic output formats. Some examples that ship with Sphinx:

A linkcheck builder that tells you about broken URL’s
A builder that outputs all the changes in the latest version of your code for a changelog
An XML builder that outputs a representation of your documents in XML
A Man page builder that builds man pages from your documentation
JSON builder that outputs your pages as HTML inside of JSON, with some metadata, for embedding dynamically

You can get as creative as you like with custom builders, which is another place to extend Sphinx outside of the markup.

Tradeoffs with Sphinx

Every tool has its issues and limitations. I’d like to address some of the issues with Sphinx, so that you can go into it with eyes wide open.

The biggest issue is that it is originally a programmer tool. This means that a lot of the designs assume knowledge of code, especially for installation and extending the tools. Knowledge of the command line is definitely required.

The markup language, reStructuredText, is also a bit finicky. It depends on whitespace for separation of content, which is a natural concept for Python programmers, but not for most writers.

In general though, a lot of the complexity in the language comes from the extensibility and power. When compared to something like Markdown, reStructuredText can do so much more that it’s worth the complexity and somewhat steep learning curve.

Introduction to Read the Docs

Read the Docs is a hosting platform for Sphinx-generated documentation. It takes the power of Sphinx and adds version control, full-text search, and other useful features. It pulls down code and doc files from Git, Mercurial, or Subversion, then builds and hosts your documentation. We’ll use GitHub in this example as it’s the most commonly used system for accessing code.

To get started, you create a Read the Docs account, and link your GitHub account. Then you select the GitHub repository you’d like to build documentation for, at which point the magic happens.

Read the Docs will:

Clone your repository
Build HTML, PDF, and ePub versions of your documentation from your master branch.
Index your documentation to allow full-text-search
Create Version objects from each tag and branch in your repository, allowing you to optionally host those as well
Activate a webhook on your repository, so when you push code to any branch, your documentation is rebuilt

Now, whenever you commit new code or documentation to your repository, your documentation is kept up to date. This works with your master branch, as well as any other branches or tags you might have activated documentation for.

Read the Docs has two special versions:

latest which maps to the most up to date development version of your software
stable which maps to the latest tagged release of your software

These are version aliases that help maintain stable URLs for the most up-to-date commits or for the most stable released version of your software.

We built Read the Docs to be “set it and forget it”. Once you set your project up and activate the versions you want hosted, we sit downstream of your version control system and just keep your documentation up to date. It feels pretty magical once it’s set up, and takes the thankless task of deploying documentation out of your day.

Recommended Versioning System

Read the Docs only works with version control. This means that however you version your software, your documentation follows suit. This is great for integrating with your development team, but it also means you need to think about the proper strategy for versioning.

After working with a lot of open source projects, we generally recommend this workflow for projects:

A master branch that the next release of your software is developed on
Git branches for ongoing maintenance of each version of your software that is maintained
Git tags for specific released versions that users might be using

We recommend release branches because it allows you to update them over time. Git tags are static, so they are appropriate for specific versions that a user might have installed.

An example:

master is your 2.2 release that isn’t out yet
2.1.X is your release branch for the 2.1 version
2.1.1 is a similar tag of your 2.1 branch, with the latest release
2.1.0 is the first tag of your 2.1 branch

Additional Read the Docs Features

Read the Docs also provides the following features:

GitHub, Bitbucket, and Gitlab webhooks
Custom domain hosting
Full-text search for all your projects’ versions
Status badges to show your docs are up to date
Hosting of multiple languages for a specific project
Hosting of multiple projects on a single domain with “subprojects”

Feel free to email me or find me at a conference if you want to talk more about these concepts, or have ideas for other neat features.

Real Life Examples

Read the Docs has a large number of users across many different programming languages. This is in part because Sphinx is such a powerful and dynamic tool, and Read the Docs makes it easy and free to host docs for your open source project.

Here are some examples that show the wide range of projects using Sphinx & Read the Docs:

ASP.NET - Microsoft’s web framework: https://docs.asp.net
Julia - A language for scientific computing: http://docs.julialang.org
Jupyter - An interactive programming environment: http://jupyter.readthedocs.io
PHPMyAdmin - A web-based database editor: https://docs.phpmyadmin.net
Write the Docs - The community site for Write the Docs - http://www.writethedocs.org

As you can see, a wide range of projects are using the tools for many different uses. It’s a powerful tool for writing prose as well as documenting source code.

Going Forward

Sphinx is an incredibly powerful tool. Read the Docs builds on top to provide hosting for Sphinx documentation that keeps your docs up to date across versions. Together, they are a wonderful set of tools that developers and technical writers both enjoy using.

I firmly believe that the more integrated with the product development process technical writers get, the better our products get. Tools that integrate documentation and development workflows make it much easier for writers to become part of the larger development process. Sphinx and Read the Docs have been battle tested by hundreds of thousands of open source developers for years, and are a great choice for your software documentation project.

Note

Eric Holscher offers consulting and contracting on work related to Sphinx. If you want help implementing it in your organization, feel free to reach out and hopefully we can work together.

Why You Shouldn’t Use “Markdown” for Documentation

2016-03-15T09:00:00+00:00

Note

This post was written in 2016, and the landscape here has changed. I still mostly agree with what is important, but the “Markdown” ecosystem has evolved and gotten other capabilities that might change the calculus of what is the right tool for your organization.

“Markdown” is the most commonly used lightweight markup language on the internet. It is great for a subset of tasks, mainly blog posts and commenting. However, lately it has been adopted by the technical writing community as a solution for writing documentation.

I’d like to lay out the main arguments that I have against Markdown. Hopefully this will be useful in helping you decide whether it’s a good fit for your organization. If you are considering Markdown, I hope that you also look at Asciidoctor and Sphinx. I find them to be much better toolsets for writing documentation.

Markdown is often chosen because it’s viewed as a simple approach that handles the basic cases well. Developers prefer it because GitHub supports it, though GitHub supports 9 different markup languages, including Asciidoc and reStructuredText. As documentation grows from a few pages into a large set of documents, Markdown quickly falls over and becomes a liability instead of a benefit. I’d like to explain a bit more about why this is the case.

Lack of a standard

For a long time, Markdown was defined by the initial implementation written by John Gruber. There was no spec, and the “proper” behavior was whatever this script chose to do.

As Markdown got popular, more and more sites started to implement it. Those sites were written in other languages, so more implementations of Markdown were created. All of these implementations had slight variations in what was accepted.

One example is that some required a space before a heading and others didn’t:

# Heading 1

#Heading 1

There are a number of small issues that made it hard to port your Markdown between sites and versions.

In the last few years, Commonmark was developed as a standardized Markdown. This is great, and should solve lots of problems! Except that nobody has adopted it…

Flavors

The main reason for this lack of adoption is that people using Markdown haven’t been sitting still for all these years. Because the original Markdown is so limited, every popular tool built on top of Markdown implements what is called a “flavor” of Markdown. This sounds great, except that every tool implements a different flavor. Even tools that do similar things with the language use different syntax for it!

For example, in Markdown Extra code blocks look like this:

~~~ .python

import antigravity

~~~

This would apply a python class to the HTML block that is output.

However, with GitHub Flavored Markdown the same example would be:

```python

import antigravity

```

This would apply syntax highlighting to the actual rendered HTML output.

This is two different syntaxes for a similar concept, both called “Markdown”

Lack of Extensibility

With other markup languages, you can extend the language to provide the features that you need. They have mechanisms in the syntax to add new functionality, without breaking from the original spec and creating a new language.

reStructuredText for example, has both inline and block level markup:

.. contents:: All the stuff I'm going to talk about
   :depth: 2

Please look at :rfc:`1984` for more information.
This is implemented in our codebase at :class:`Example.Encryption`.

You can learn more about the rfc, class, and contents concepts.

As a developer working with rST or Asciidoctor, I can add new markup to the language in a simple, pluggable way. I don’t have to change how the language is parsed, and I can share those additions with other users via a standard extension mechanism.

There is no way of doing this in Markdown, in a way that would be portable across versions.

Note

CommonMark is working on an extensibility syntax, but it isn’t implemented yet.

Lack of Semantic Meaning

Though many people have added extensions to Markdown, almost none have any kind of semantic meaning. This means that you can’t write a Class or a Warning, you can only write text.

This leads people to embed HTML directly in their Markdown:

<div class="warning">

This is a Warning!

</div>

In reStructuredText for example, you can write:

.. warning:: This is a Warning!

This will be output as a warning properly in HTML, PDF, and any other output format you can generate.

Semantic markup firmly separates the words that you write from how they are displayed.

Writing without semantic markup is a problem for a few reasons:

Your Markdown is now dependent on specific CSS classes in your display, meaning your writers have to think about how your page will be designed
Your content is no longer portable to other output formats (PDF, etc.)
Conversion to other markup tools and page designs becomes much harder

Note

I have covered the ideas around semantics more in my post Semantic Meaning in Authoring Documentation.

Lock In and Lack of Portability

The explosion of flavors and lack of semantic meaning leads to lock in. Once you’ve built out a large set of Markdown documents, it’s quite hard to migrate them to another tool, even if that tool claims to support Markdown! You have a large set of custom HTML classes and weird flavor extensions that won’t work anywhere but the current set of tools and designs.

You also can’t migrate Markdown easily to another markup language (Asciidoc or RST), because Pandoc and other conversion tools won’t support your flavor’s extensions.

I think that a lot of people choose Markdown because they think they can migrate to another tool or markup later. Markdown is definitely the lowest common denominator, except that for any reasonably sized set of docs you’ll need things that aren’t in the basic language.

Once you start using markdown flavors, which is required for any non-trivial documentation, you lose all portability benefits.

Conclusion

I believe that CommonMark is a good step forward, and if it became more widely used, and added extension support, I could whole-heartedly recommend it as a solution to this problem. The current ecosystem we have around Markdown is not something that I can endorse, and believe that it’s actively holding back folks to want to make documentation better.

I hope that we can start to move forward with a more standardized set of languages, including CommonMark, reStructuredText, and Asciidoc, fully supporting them across the suite of tools that we use. For now, please investigate Sphinx and Asciidoctor as good alternatives. They come with a lot more extensibility built into the language, and are more complete tools for building sets of documentation today.

Markdown is a concept more than it is an implementation. It generally means “a set of incompatible extensions to something that looks kinda like Markdown”. When you are trying to author large sets of documents, it isn’t the correct tool.

Full Disclosure: I work on a product, Read the Docs, which is based on Sphinx, so my views are likely biased.