This post will describe how add translations (i18n), pdf/epub builds, and branch-specific versioned documentation to a Read-the-Docs-themed sphinx site hosted with GitHub Pages and built with GitHub’s free CI/CD tools.
This is part two of a two-part series. Before reading this, you should already be familiar with Continuous Documentation: Hosting Read the Docs on GitHub Pages (1/2).
ⓘ Note: If you don’t care about how this works and you just want to make a functional repo, you can just fork my ‘rtd-github-pages’ GitHub repo.
Adding “/en/latest/“
One of the first questions I had after setting-up a Read the Docs (rtd) site for my documentation on GitHub Pages was: how do I get it to build to language-and-version-specific directories?
Before deciding on sphinx to host my documentation, I had done a lot of research, and I was very impressed by the number of sites that were using it–and so many of them had such beautiful, complex, and unique site designs. One thing that I kept seeing: most of their URLs ended with ‘/<lang>/<version>/‘, so I expected my site would be the same (if, for nothing else, to future-proof the URLs):
- https://docs.phpmyadmin.net/en/latest/
- https://zulip.readthedocs.io/en/latest/
- https://cassandra.apache.org/doc/latest/
- https://book.cakephp.org/4/en/
- https://docs.readthedocs.io/en/latest/
- https://www.sphinx-doc.org/en/master/
I followed the sphinx guide to internationalization. I setup the `language` and `locale_dirs` in my sphinx ‘conf.py‘ file, but `make install` still spat-out all my html files straight into the `_build/html/` directory! Why wasn’t it putting it in a language-specific directory? And where does ‘/latest/‘ fit in?
Well, the undocumented truth is that sphinx doesn’t do this. In fact, sphinx i18n functionality stops-short of a workflow for building all your language-specific directories. It’s implicit that you have to design & wrap `sphinx-build` with your own scripts to set this up.
The ‘/en/latest/‘ that you see everywhere is actually done by rtd, not sphinx. Yes, even sphinx-doc.org is hosted by rtd.
Internationalization (i18n)
Later we’ll update our `buildDocs.sh` script that we created in part 1 of this guide, but first let’s demonstrate how to translate our helloWorld example rtd site from English to Spanish.
To translate your sphinx reST files to another language, you don’t have to update your ‘.rst‘ files themselves. Sphinx already understands what a block of text looks like, and it can divide-up heading strings, captions, double-newline-separated paragraphs, etc into unique “source strings” (msgid), and put them into ‘.pot‘ source-language files and ‘.po‘ destination-language files.
First, run `make gettext` from the ‘docs/‘ directory. This tells sphinx to parse your reST files and automatically find a bunch of strings-to-be-translated and give them a unique `msgid`.
user@host:~/rtd-github-pages$ cd docs/ user@host:~/rtd-github-pages/docs$ ls autodoc.rst buildDocs.sh conf.py.orig locales Makefile _build conf.py index.rst make.bat _static user@host:~/rtd-github-pages/docs$ user@host:~/rtd-github-pages/docs$ make gettext Running Sphinx v1.8.4 making output directory... building [gettext]: targets for 0 template files building [gettext]: targets for 2 source files that are out of date updating environment: 2 added, 0 changed, 0 removed Hello Worldrces... [ 50%] autodoc reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index writing message catalogs... [100%] index build succeeded. The message catalogs are in _build/gettext. user@host:~/rtd-github-pages/docs$
The above execution should create the following files
user@host:~/rtd-github-pages/docs$ ls _build/gettext/ autodoc.pot index.pot user@host:~/rtd-github-pages/docs$
Here’s a snippet from ‘_build/gettext/index.pot‘ showing two strings on our documentation’s main page that we’ll translate from English to Spanish.
user@host:~/rtd-github-pages/docs$ grep -m2 -A2 .rst _build/gettext/index.pot #: ../../index.rst:7 msgid "Welcome to helloWorld's documentation!" msgstr "" -- #: ../../index.rst:9 msgid "Contents:" msgstr "" user@host:~/rtd-github-pages/docs$
Next, let’s tell sphinx to prepare some Spanish destination-language ‘.po‘ files from our above-generated source-lananguage ‘.pot‘ files.
Before proceeding with this step, you’ll need to install `sphinx-intl` and the python `Stemmer` module. If you’re using a Debian-based distro, you can do so with the following command.
sudo apt-get install -y sphinx-intl python3-stemmer
Execute the following command to prepare our Spanish-specific translation files.
user@host:~/rtd-github-pages/docs$ sphinx-intl update -p _build/gettext -l es Create: locales/es/LC_MESSAGES/index.po Create: locales/es/LC_MESSAGES/autodoc.po user@host:~/rtd-github-pages/docs$
The above execution created two ‘.po‘ files: one for each of our ‘.pot‘ source-language files, which correlate directly to each of our two ‘.rst‘ files (index.rst and autodoc.rst). Perfect.
If we grep the new Spanish-specific ‘docs/locales/es/LC_MESSAGES/index.po‘ file, we see it has the same contents as the source ‘.pot‘ file.
user@host:~/rtd-github-pages/docs$ grep -m2 -A2 .rst locales/es/LC_MESSAGES/index.po #: ../../index.rst:7 msgid "Welcome to helloWorld's documentation!" msgstr "" -- #: ../../index.rst:9 msgid "Contents:" msgstr "" user@host:~/rtd-github-pages/docs$
These language-specific ‘.po‘ files are where we actually do the translating. If you’re a large project, then you’d probably want to use a special program or service to translate these files. But, for clarity, we’ll just edit the files directly.
user@host:~/rtd-github-pages/docs$ perl -pi -0e "s^(msgid \"Welcome to helloWorld's documentation\!\"\n)msgstr \"\"^\1msgstr \"¡Bienvenido a la documentación de helloWorld\!\"^" locales/es/LC_MESSAGES/index.po user@host:~/rtd-github-pages/docs$ perl -pi -0e "s^(msgid \"Contents:\"\n)msgstr \"\"^\1msgstr \"Contenidos:\"^" locales/es/LC_MESSAGES/index.po user@host:~/rtd-github-pages/docs$ user@host:~/rtd-github-pages/docs$ grep -m2 -A2 .rst locales/es/LC_MESSAGES/index.po #: ../../index.rst:7 msgid "Welcome to helloWorld's documentation!" msgstr "¡Bienvenido a la documentación de helloWorld!" -- #: ../../index.rst:9 msgid "Contents:" msgstr "Contenidos:" user@host:~/rtd-github-pages/docs$
As you can see, the above execution filled-in the contents of `msgstr ""` with the Spanish translation of the corresponding `msgid` line above it in the original (English) language.
Now let’s build two versions of our html static content: [1] in English and [2] in Spanish.
user@host:~/rtd-github-pages/docs$ sphinx-build -b html . _build/html/en -D language='en' Running Sphinx v1.8.4 loading translations [en]... done making output directory... building [mo]: targets for 0 po files that are out of date building [html]: targets for 2 source files that are out of date updating environment: 2 added, 0 changed, 0 removed Hello Worldrces... [ 50%] autodoc reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index generating indices... genindex py-modindex highlighting module code... [100%] helloWorld writing additional pages... search copying static files... done copying extra files... done dumping search index in English (code: en) ... done dumping object inventory... done build succeeded. The HTML pages are in _build/html/en. user@host:~/rtd-github-pages/docs$ user@host:~/rtd-github-pages/docs$ sphinx-build -b html . _build/html/es -D language='es' Running Sphinx v1.8.4 loading translations [es]... done making output directory... building [mo]: targets for 1 po files that are out of date writing output... [100%] locales/es/LC_MESSAGES/index.mo building [html]: targets for 2 source files that are out of date updating environment: 2 added, 0 changed, 0 removed Hello Worldrces... [ 50%] autodoc reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index generating indices... genindex py-modindex highlighting module code... [100%] helloWorld writing additional pages... search copying static files... done copying extra files... done dumping search index in Spanish (code: es) ... done dumping object inventory... done build succeeded. The HTML pages are in _build/html/es. user@host:~/rtd-github-pages/docs$ user@host:~/rtd-github-pages/docs$ firefox _build/html/en/index.html _build/html/es/index.html & [1] 12134 user@host:~/rtd-github-pages/docs$
The `firefox` command in the above execution should open your browser with two tabs: [1] in English and [2] in Spanish.
Our first tab is in English
Our second tab is in Spanish (partially)
As you can see, the two strings we edited above are now translated for the Spanish site in the ‘_build/html/es/‘ directory. There’s also a few other strings that got translated. I’m not sure why, but some words like “Module Index” may automatically get translated by sphinx while others like “Indices and tables” and “Next” do not.
Versioning
Now that we have the translation and the workflow for language-specific build directories worked-out, let’s look at building to version-specific directories.
Again, sphinx has no built-in support for the notion of building distinct sites based on distinct versions; it’s up to your team to wrap `sphinx-build` and setup your version-specific directories yourself.
For our helloWorld example, let’s link versions to branches. For every branch with a ‘docs/‘ directory, we’ll have a distinct version of our documentation inside each of our language-specific directories. Later we’ll update our `buildDocs.sh` script to iterate over our remote branches and execute `sphinx-build` for each, but first let’s create another branch!
Let’s assume that your project is only going to merge stable code into ‘master‘ and you’ll have a distinct branch called ‘dev‘ for active development. With this setup, our goal is to have four distinct documentation builds:
/en/master//en/dev//es/master//es/dev/
Execute the following to create a ‘dev‘ branch and switch to it.
user@host:~/rtd-github-pages/docs$ git branch -l * master user@host:~/rtd-github-pages/docs$ git checkout -b dev Switched to a new branch 'dev' user@host:~/rtd-github-pages/docs$ git branch -l * dev master user@host:~/rtd-github-pages/docs$
Now we should update our GitHub Actions ‘docs_pages_workflow‘ workflow so that our documentation is built when any branch is pushed to github.com, and make sure that the `buildDocs.sh` script has access to the other branches.
cat > ../.github/workflows/docs_pages_workflow.yml <<'EOF'
name: docs_pages_workflow
# execute this workflow automatically when a we push to master
on:
push:
jobs:
build_docs_job:
runs-on: ubuntu-latest
container: debian:buster-slim
steps:
- name: Prereqs
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
apt-get update
apt-get install -y git
git clone "https://token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" .
shell: bash
- name: Execute script to build our documentation and update pages
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: "docs/buildDocs.sh"
shell: bash
EOF
Let’s diff this new file with the old one that was created in part-one of this guide.
user@host:~/rtd-github-pages/docs$ git diff --unified=1 ../.github/workflows/docs_pages_workflow.yml
diff --git a/.github/workflows/docs_pages_workflow.yml b/.github/workflows/docs_pages_workflow.yml
index d063ac3..c85d816 100644
--- a/.github/workflows/docs_pages_workflow.yml
+++ b/.github/workflows/docs_pages_workflow.yml
@@ -5,3 +5,2 @@ on:
push:
- branches: [ master ]
@@ -21,3 +20,3 @@ jobs:
apt-get install -y git
- git clone --depth 1 "https://token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" .
+ git clone "https://token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" .
shell: bash
user@host:~/rtd-github-pages/docs$
The above diff shows that we removed the line restricting the workflow from running only when the ‘master‘ branch is pushed to github.com
And we’ve changed the `git clone` command so that we get a full clone of the repo, which will allow `buildDocs.sh` to see all branches.
Updating `buildDocs.sh`
Now let’s update our `buildDocs.sh` script to iterate over both languages & versions. And also build pdf & epub files.
Before proceeding with this step, you’ll need to install the `git` python module. If you’re using a Debian-based distro, you can do so with the following command.
sudo apt-get install -y python3-git
Execute the following to overwrite the `buildDocs.sh` script.
cat > buildDocs.sh <<'EEOOFF'
#!/bin/bash
set -x
################################################################################
# File: buildDocs.sh
# Purpose: Script that builds our documentation using sphinx and updates GitHub
# Pages. This script is executed by:
# .github/workflows/docs_pages_workflow.yml
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2020-07-17
# Updated: 2020-07-23
# Version: 0.2
################################################################################
###################
# INSTALL DEPENDS #
###################
apt-get update
apt-get -y install git rsync python3-sphinx python3-sphinx-rtd-theme python3-stemmer python3-git python3-pip python3-virtualenv python3-setuptools
python3 -m pip install --upgrade rinohtype pygments
#####################
# DECLARE VARIABLES #
#####################
pwd
ls -lah
export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
# make a new temp dir which will be our GitHub Pages docroot
docroot=`mktemp -d`
export REPO_NAME="${GITHUB_REPOSITORY##*/}"
##############
# BUILD DOCS #
##############
# first, cleanup any old builds' static assets
make -C docs clean
# get a list of branches, excluding 'HEAD' and 'gh-pages'
versions="`git for-each-ref '--format=%(refname:lstrip=-1)' refs/remotes/origin/ | grep -viE '^(HEAD|gh-pages)$'`"
for current_version in ${versions}; do
# make the current language available to conf.py
export current_version
git checkout ${current_version}
echo "INFO: Building sites for ${current_version}"
# skip this branch if it doesn't have our docs dir & sphinx config
if [ ! -e 'docs/conf.py' ]; then
echo -e "\tINFO: Couldn't find 'docs/conf.py' (skipped)"
continue
fi
languages="en `find docs/locales/ -mindepth 1 -maxdepth 1 -type d -exec basename '{}' \;`"
for current_language in ${languages}; do
# make the current language available to conf.py
export current_language
##########
# BUILDS #
##########
echo "INFO: Building for ${current_language}"
# HTML #
sphinx-build -b html docs/ docs/_build/html/${current_language}/${current_version} -D language="${current_language}"
# PDF #
sphinx-build -b rinoh docs/ docs/_build/rinoh -D language="${current_language}"
mkdir -p "${docroot}/${current_language}/${current_version}"
cp "docs/_build/rinoh/target.pdf" "${docroot}/${current_language}/${current_version}/helloWorld-docs_${current_language}_${current_version}.pdf"
# EPUB #
sphinx-build -b epub docs/ docs/_build/epub -D language="${current_language}"
mkdir -p "${docroot}/${current_language}/${current_version}"
cp "docs/_build/epub/target.epub" "${docroot}/${current_language}/${current_version}/helloWorld-docs_${current_language}_${current_version}.epub"
# copy the static assets produced by the above build into our docroot
rsync -av "docs/_build/html/" "${docroot}/"
done
done
# return to master branch
git checkout master
#######################
# Update GitHub Pages #
#######################
git config --global user.name "${GITHUB_ACTOR}"
git config --global user.email "${GITHUB_ACTOR}@users.noreply.github.com"
pushd "${docroot}"
# don't bother maintaining history; just generate fresh
git init
git remote add deploy "https://token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
git checkout -b gh-pages
# add .nojekyll to the root so that github won't 404 on content added to dirs
# that start with an underscore (_), such as our "_content" dir..
touch .nojekyll
# add redirect from the docroot to our default docs language/version
cat > index.html <<EOF
<!DOCTYPE html>
<html>
<head>
<title>helloWorld Docs</title>
<meta http-equiv = "refresh" content="0; url='/${REPO_NAME}/en/master/'" />
</head>
<body>
<p>Please wait while you're redirected to our <a href="/${REPO_NAME}/en/master/">documentation</a>.</p>
</body>
</html>
EOF
# Add README
cat > README.md <<EOF
# GitHub Pages Cache
Nothing to see here. The contents of this branch are essentially a cache that's not intended to be viewed on github.com.
If you're looking to update our documentation, check the relevant development branch's 'docs/' dir.
For more information on how this documentation is built using Sphinx, Read the Docs, and GitHub Actions/Pages, see:
* https://tech.michaelaltfield.net/2020/07/18/sphinx-rtd-github-pages-1
EOF
# copy the resulting html pages built from sphinx above to our new git repo
git add .
# commit all the new files
msg="Updating Docs for commit ${GITHUB_SHA} made on `date -d"@${SOURCE_DATE_EPOCH}" --iso-8601=seconds` from ${GITHUB_REF} by ${GITHUB_ACTOR}"
git commit -am "${msg}"
# overwrite the contents of the gh-pages branch on our github.com repo
git push deploy gh-pages --force
popd # return to main repo sandbox root
# exit cleanly
exit 0
EEOOFF
chmod +x buildDocs.sh
Let’s diff this new `buildDocs.sh` with the old one:
diff --git a/docs/buildDocs.sh b/docs/buildDocs.sh
index 91f01a8..0fd847e 100755
--- a/docs/buildDocs.sh
+++ b/docs/buildDocs.sh
@@ -10,4 +10,4 @@ set -x
# Created: 2020-07-17
-# Updated: 2020-07-17
-# Version: 0.1
+# Updated: 2020-07-23
+# Version: 0.2
################################################################################
@@ -19,3 +19,5 @@ set -x
apt-get update
-apt-get -y install git rsync python3-sphinx python3-sphinx-rtd-theme
+apt-get -y install git rsync python3-sphinx python3-sphinx-rtd-theme python3-stemmer python3-git python3-pip python3-virtualenv python3-setuptools
+
+python3 -m pip install --upgrade rinohtype pygments
@@ -29,2 +31,7 @@ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
+# make a new temp dir which will be our GitHub Pages docroot
+docroot=`mktemp -d`
+
+export REPO_NAME="${GITHUB_REPOSITORY##*/}"
+
##############
@@ -33,6 +40,54 @@ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
-# build our documentation with sphinx (see docs/conf.py)
-# * https://www.sphinx-doc.org/en/master/usage/quickstart.html#running-the-build
+# first, cleanup any old builds' static assets
make -C docs clean
-make -C docs html
+
+# get a list of branches, excluding 'HEAD' and 'gh-pages'
+versions="`git for-each-ref '--format=%(refname:lstrip=-1)' refs/remotes/origin/ | grep -viE '^(HEAD|gh-pages)$'`"
+for current_version in ${versions}; do
+
+ # make the current language available to conf.py
+ export current_version
+ git checkout ${current_version}
+
+ echo "INFO: Building sites for ${current_version}"
+
+ # skip this branch if it doesn't have our docs dir & sphinx config
+ if [ ! -e 'docs/conf.py' ]; then
+ echo -e "\tINFO: Couldn't find 'docs/conf.py' (skipped)"
+ continue
+ fi
+
+ languages="en `find docs/locales/ -mindepth 1 -maxdepth 1 -type d -exec basename '{}' \;`"
+ for current_language in ${languages}; do
+
+ # make the current language available to conf.py
+ export current_language
+
+ ##########
+ # BUILDS #
+ ##########
+ echo "INFO: Building for ${current_language}"
+
+ # HTML #
+ sphinx-build -b html docs/ docs/_build/html/${current_language}/${current_version} -D language="${current_language}"
+
+ # PDF #
+ sphinx-build -b rinoh docs/ docs/_build/rinoh -D language="${current_language}"
+ mkdir -p "${docroot}/${current_language}/${current_version}"
+ cp "docs/_build/rinoh/target.pdf" "${docroot}/${current_language}/${current_version}/helloWorld-docs_${current_language}_${current_version}.pdf"
+
+ # EPUB #
+ sphinx-build -b epub docs/ docs/_build/epub -D language="${current_language}"
+ mkdir -p "${docroot}/${current_language}/${current_version}"
+ cp "docs/_build/epub/target.epub" "${docroot}/${current_language}/${current_version}/helloWorld-docs_${current_language}_${current_version}.epub"
+
+ # copy the static assets produced by the above build into our docroot
+ rsync -av "docs/_build/html/" "${docroot}/"
+
+ done
+
+done
+
+# return to master branch
+git checkout master
@@ -45,5 +100,2 @@ git config --global user.email "${GITHUB_ACTOR}@users.noreply.github.com"
-docroot=`mktemp -d`
-rsync -av "docs/_build/html/" "${docroot}/"
-
pushd "${docroot}"
@@ -59,2 +111,16 @@ touch .nojekyll
+# add redirect from the docroot to our default docs language/version
+cat > index.html <<EOF
+<!DOCTYPE html>
+<html>
+ <head>
+ <title>helloWorld Docs</title>
+ <meta http-equiv = "refresh" content="0; url='/${REPO_NAME}/en/master/'" />
+ </head>
+ <body>
+ <p>Please wait while you're redirected to our <a href="/${REPO_NAME}/en/master/">documentation</a>.</p>
+ </body>
+</html>
+EOF
+
# Add README
The above diff shows the following changes:
- Lines 14-17: Adding some dependencies necessary for language translations, building PDFs, and building EPUBs
- Lines 35-83: This is our double-nested loop that iterates over the languages and versions (branches)
- Lines 57-73: Here is where we actually call `
sphinx-build` to generate our html, pdf, and epub files - Lines 93-105: Adding a redirect ‘
/index.html‘ file at the root of our docroot, since now all our builds are in language- and version-specific directories
Also note that we’re now setting and exporting the `current_version` and `current_language` environment variables in `updateDocs.sh`
export current_version
...
export current_language
We’ll see in the next section how these `current_version` and `current_language` environment variables are used by the ‘conf.py` file and made available to the rtd theme for preparing the lower-left menu — allowing the user to navigate between languages, versions, and downloads.
Updating “conf.py“
The outer-most wrapper that’s executed by our GitHub Actions workflow to build our documentation is the `buildDocs.sh` script we created above. That’s the file that iterates over the languages and the versions, and it executes `sphinx-build` to build out distinct language-and-version-specific html docs, pdf docs, and epub docs.
With every execution, ‘conf.py‘ is what sets-up the language and versions to be passed to the theme. And it’s what sets the build options for our pdf and ebpub files.
Let’s add that logic to our ‘conf.py‘ file.
cat >> conf.py <<'EOF'
############################
# SETUP THE RTD LOWER-LEFT #
############################
try:
html_context
except NameError:
html_context = dict()
html_context['display_lower_left'] = True
if 'REPO_NAME' in os.environ:
REPO_NAME = os.environ['REPO_NAME']
else:
REPO_NAME = ''
# SET CURRENT_LANGUAGE
if 'current_language' in os.environ:
# get the current_language env var set by buildDocs.sh
current_language = os.environ['current_language']
else:
# the user is probably doing `make html`
# set this build's current language to english
current_language = 'en'
# tell the theme which language to we're currently building
html_context['current_language'] = current_language
# SET CURRENT_VERSION
from git import Repo
repo = Repo( search_parent_directories=True )
if 'current_version' in os.environ:
# get the current_version env var set by buildDocs.sh
current_version = os.environ['current_version']
else:
# the user is probably doing `make html`
# set this build's current version by looking at the branch
current_version = repo.active_branch.name
# tell the theme which version we're currently on ('current_version' affects
# the lower-left rtd menu and 'version' affects the logo-area version)
html_context['current_version'] = current_version
html_context['version'] = current_version
# POPULATE LINKS TO OTHER LANGUAGES
html_context['languages'] = [ ('en', '/' +REPO_NAME+ '/en/' +current_version+ '/') ]
languages = [lang.name for lang in os.scandir('locales') if lang.is_dir()]
for lang in languages:
html_context['languages'].append( (lang, '/' +REPO_NAME+ '/' +lang+ '/' +current_version+ '/') )
# POPULATE LINKS TO OTHER VERSIONS
html_context['versions'] = list()
versions = [branch.name for branch in repo.branches]
for version in versions:
html_context['versions'].append( (version, '/' +REPO_NAME+ '/' +current_language+ '/' +version+ '/') )
# POPULATE LINKS TO OTHER FORMATS/DOWNLOADS
# settings for creating PDF with rinoh
rinoh_documents = [(
master_doc,
'target',
project+ ' Documentation',
'© ' +copyright,
)]
today_fmt = "%B %d, %Y"
# settings for EPUB
epub_basename = 'target'
html_context['downloads'] = list()
html_context['downloads'].append( ('pdf', '/' +REPO_NAME+ '/' +current_language+ '/' +current_version+ '/' +project+ '-docs_' +current_language+ '_' +current_version+ '.pdf') )
html_context['downloads'].append( ('epub', '/' +REPO_NAME+ '/' +current_language+ '/' +current_version+ '/' +project+ '-docs_' +current_language+ '_' +current_version+ '.epub') )
EOF
The above execution updates ‘conf.py‘ with 8 important changes:
- We update ‘
html_context‘ to enable the ‘display_lower_left‘ flag - We set the ‘
current_language‘ in ‘html_context‘ - We set the ‘
current_version‘ in ‘html_context‘ - We define a set of tuples for links to other ‘
languages‘ in ‘html_context‘ - We define a set of tuples for links to other ‘
versions‘ in ‘html_context‘ - We define some basic settings that tell sphinx how to build our pdf
- We set the destination filename prefix for our epub
- We define a set of tuples for links to other ‘
downloads‘ in ‘html_context‘
As we’ll see below, this ‘html_context‘ variable is a sphinx-specific python dict() that’s exposed to our rtd theme.
Let’s add a few more keys to the ‘html_context‘ dict that will change the “View page source” links in to top-right to “Edit on GitHub”.
cat >> conf.py <<EOF ########################## # "EDIT ON GITHUB" LINKS # ########################## html_context['display_github'] = True html_context['github_user'] = 'maltfield' html_context['github_repo'] = 'rtd-github-pages' html_context['github_version'] = 'master/docs/' EOF
You can read about some of the options that the rtd theme consumes from this ‘html_context‘ variable. Unfortunately, however, it appears that the theme used by readthedocs.io is distinct from the actual sphinx-rtd-theme that they publish on github.
Overriding “versions.html“
In order to get the functionality used on readthedocs.io on our rtd site (at least until my PR is merged), we have to override the rtd theme’s ‘versions.html‘ jinja template file.
Execute the following to create a new ‘_templates/‘ directory and ‘versions.html‘ file:
mkdir _templates
cat > _templates/versions.html <<'EOF'
{% if READTHEDOCS or display_lower_left %}
{# Add rst-badge after rst-versions for small badge style. #}
<div class="rst-versions" data-toggle="rst-versions" role="note" aria-label="versions">
<span class="rst-current-version" data-toggle="rst-current-version">
<span class="fa fa-book"> Read the Docs</span>
v: {{ current_version }}
<span class="fa fa-caret-down"></span>
</span>
<div class="rst-other-versions">
{% if languages|length >= 1 %}
<dl>
<dt>{{ _('Languages') }}</dt>
{% for slug, url in languages %}
{% if slug == current_language %} <strong> {% endif %}
<dd><a href="{{ url }}">{{ slug }}</a></dd>
{% if slug == current_language %} </strong> {% endif %}
{% endfor %}
</dl>
{% endif %}
{% if versions|length >= 1 %}
<dl>
<dt>{{ _('Versions') }}</dt>
{% for slug, url in versions %}
{% if slug == current_version %} <strong> {% endif %}
<dd><a href="{{ url }}">{{ slug }}</a></dd>
{% if slug == current_version %} </strong> {% endif %}
{% endfor %}
</dl>
{% endif %}
{% if downloads|length >= 1 %}
<dl>
<dt>{{ _('Downloads') }}</dt>
{% for type, url in downloads %}
<dd><a href="{{ url }}">{{ type }}</a></dd>
{% endfor %}
</dl>
{% endif %}
{% if READTHEDOCS %}
<dl>
<dt>{{ _('On Read the Docs') }}</dt>
<dd>
<a href="//{{ PRODUCTION_DOMAIN }}/projects/{{ slug }}/?fromdocs={{ slug }}">{{ _('Project Home') }}</a>
</dd>
<dd>
<a href="//{{ PRODUCTION_DOMAIN }}/builds/{{ slug }}/?fromdocs={{ slug }}">{{ _('Builds') }}</a>
</dd>
</dl>
{% endif %}
<hr/>
{% trans %}Free document hosting provided by <a href="http://www.readthedocs.org">Read the Docs</a>.{% endtrans %}
</div>
</div>
{% endif %}
EOF
We can diff the above file with the rtd theme’s ‘versions.html' to see what was changed:
@@ -1,2 +1,2 @@
-{% if READTHEDOCS %}
+{% if READTHEDOCS or display_lower_left %}
{# Add rst-badge after rst-versions for small badge style. #}
@@ -9,2 +9,13 @@
<div class="rst-other-versions">
+ {% if languages|length >= 1 %}
+ <dl>
+ <dt>{{ _('Languages') }}</dt>
+ {% for slug, url in languages %}
+ {% if slug == current_language %} <strong> {% endif %}
+ <dd><a href="{{ url }}">{{ slug }}</a></dd>
+ {% if slug == current_language %} </strong> {% endif %}
+ {% endfor %}
+ </dl>
+ {% endif %}
+ {% if versions|length >= 1 %}
<dl>
@@ -12,5 +23,9 @@
{% for slug, url in versions %}
+ {% if slug == current_version %} <strong> {% endif %}
<dd><a href="{{ url }}">{{ slug }}</a></dd>
+ {% if slug == current_version %} </strong> {% endif %}
{% endfor %}
</dl>
+ {% endif %}
+ {% if downloads|length >= 1 %}
<dl>
@@ -21,2 +36,4 @@
</dl>
+ {% endif %}
+ {% if READTHEDOCS %}
<dl>
@@ -30,5 +47,6 @@
</dl>
+ {% endif %}
<hr/>
{% trans %}Free document hosting provided by <a href="http://www.readthedocs.org">Read the Docs</a>.{% endtrans %}
The above diff shows the following changes:
- We add a ‘
display_lower_left‘ variable to the conditional so that we can display the lower-left menu without displaying the irrelevant “Read the Docs” section - We add a ‘
languages‘ section, which was entirely missing from the ‘sphinx-rtd-theme‘ template - We add a conditional to the ‘
versions‘ section to display the current version in bold - We add a conditional to the ‘
downloads‘ section to only display the downloads section if there’s a non-empty downloads list - We add a conditional to the “
Read the Docs‘ section to only be displayed if ‘READTHEDOCS‘ is ‘True‘
Now that we’ve updated our ‘conf.py‘ and ‘versions.html‘ files, we can run `make html` to test the lower-left menu.
Testing the lower-left rtd menu
While our `buildDocs.sh` script is meant to be run in a docker container on a GitHub Actions runner, we can still do a local build test with `make html`.
Running `sphinx-build` through `make` like this will only build one version and language, defaulting to the current branch and the English language.
user@host:~/rtd-github-pages/docs$ make clean && make html Removing everything under '_build'... Running Sphinx v1.8.4 making output directory... building [mo]: targets for 0 po files that are out of date building [html]: targets for 2 source files that are out of date updating environment: 2 added, 0 changed, 0 removed Hello Worldrces... [ 50%] autodoc reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index generating indices... genindex py-modindex highlighting module code... [100%] helloWorld writing additional pages... search copying static files... done copying extra files... done dumping search index in English (code: en) ... done dumping object inventory... done build succeeded. The HTML pages are in _build/html. user@host:~/rtd-github-pages/docs$ user@host:~/rtd-github-pages/docs$ firefox _build/html/index.html user@host:~/rtd-github-pages/docs$
After the `firefox` execution above, you should see a new “Read the Docs” icon in the lower-left of your documentation. If you click it, it will open the lower-left menu displaying the languages, versions, and downloads available.
Click the “Read the Docs” icon in the lower-left to open the lower-left menu
The lower-left rtd menu shows alternate languages, versions, and downloads
Push to github.com
At this point, everything is ready in your local repo; let’s push it up to github.com
user@host:~/rtd-github-pages/docs$ git add .
user@host:~/rtd-github-pages/docs$ msg="adding lower-left rtd menu support
>
> This commit adds language-specific builds (w/ basic spanish language translations), version-specific-builds, and pdf/epub downloads, and everything else needed to get the lower-left menu functional.
>
> For more info, see:
>
> * https://tech.michaelaltfield.net/2020/07/23/sphinx-rtd-github-pages-2> "
user@host:~/rtd-github-pages/docs$ git commit -am "${msg}"
[dev 90e4d96] adding lower-left rtd menu support
create mode 100644 docs/_templates/versions.html
create mode 100644 docs/locales/es/LC_MESSAGES/autodoc.mo
create mode 100644 docs/locales/es/LC_MESSAGES/autodoc.po
create mode 100644 docs/locales/es/LC_MESSAGES/index.mo
create mode 100644 docs/locales/es/LC_MESSAGES/index.po
8 files changed, 297 insertions(+), 12 deletions(-)
user@host:~/rtd-github-pages/docs$
user@host:~/rtd-github-pages/docs$ git push origin dev
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 4 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (8/8), 4.39 KiB | 4.39 MiB/s, done.
Total 8 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
remote:
remote: Create a pull request for 'dev' on GitHub by visiting:
remote: https://github.com/maltfield/rtd-github-pages/pull/new/dev
remote:
To github.com:maltfield/rtd-github-pages.git
* [new branch] dev -> dev
user@host:~/rtd-github-pages/docs$
Normally that would be the finish of our documentation workflow. But since we made changes to `updateDocs.sh` and our ‘docs_pages_workflow‘ is (intentionally) designed to execute the `updateDocs.sh` from the master branch, we’ll need to merge to master and push.
user@host:~/rtd-github-pages/docs$ git checkout master Switched to branch 'master' Your branch is up to date with 'origin/master'. user@host:~/rtd-github-pages/docs$ git pull remote: Enumerating objects: 64, done. remote: Counting objects: 100% (64/64), done. remote: Compressing objects: 100% (57/57), done. remote: Total 64 (delta 6), reused 64 (delta 6), pack-reused 0 Unpacking objects: 100% (64/64), done. From github.com:maltfield/rtd-github-pages + 0c3f1b3...2d435cf gh-pages -> origin/gh-pages (forced update) Already up to date. user@host:~/rtd-github-pages/docs$ git merge dev Updating 4013568..90e4d96 Fast-forward .github/workflows/docs_pages_workflow.yml | 3 +- docs/buildDocs.sh | 84 +++++++++++++++++++++++++++++++++++----- docs/conf.py | 81 ++++++++++++++++++++++++++++++++++++++ docs/_templates/versions.html | 55 ++++++++++++++++++++++++++++++++++++++++++ docs/locales/es/LC_MESSAGES/autodoc.mo | Bin 0 -> 844 bytes docs/locales/es/LC_MESSAGES/autodoc.po | 43 +++++++++++++++++++++++++++++++++ docs/locales/es/LC_MESSAGES/index.mo | Bin 0 -> 761 bytes docs/locales/es/LC_MESSAGES/index.po | 43 +++++++++++++++++++++++++++++++++ 8 files changed, 297 insertions(+), 12 deletions(-) user@host:~/rtd-github-pages/docs$ git checkout dev Switched to branch 'dev' user@host:~/rtd-github-pages/docs$ git push Total 0 (delta 0), reused 0 (delta 0) To github.com:maltfield/rtd-github-pages.git 4013568..90e4d96 master -> master user@host:~/rtd-github-pages/docs$
Viewing the updated site
After pushing your commits to github.com above, GitHub Actions will automatically begin to rebuild your site. In a few minutes, it should be live (note that this run will take a little longer than before, since now it’s running `sphinx-build` four times instead of one–but it should finish in less than two minutes).
helloWorld documentation live on github.io
Troubleshooting
This section will provide tips to fixing common issues with the above guide.
Exception occurred…ValueError: too many values to unpack (expected 1)
I got this error from rinoh, but only for Enlgish. The Spanish build worked fine.
user@host:~/rtd-github-pages$ sphinx-build -b rinoh docs/ docs/_build/rinoh -D language=en
Running Sphinx v1.8.4
loading translations [en]... done
making output directory...
'rinoh_documents' config variable not set, automatically converting from 'latex_documents'
'rinoh_elements/papersize' config variable not set, automatically converting from 'latex_elements/papersize'
'rinoh_elements/papersize' config variable not set, automatically converting from 'latex_elements/papersize'
'rinoh_logo' config variable not set, automatically converting from 'latex_logo'
'rinoh_logo' config variable not set, automatically converting from 'latex_logo'
'rinoh_domain_indices' config variable not set, automatically converting from 'latex_domain_indices'
'rinoh_domain_indices' config variable not set, automatically converting from 'latex_domain_indices'
building [mo]: targets for 2 po files that are out of date
writing output... [100%] locales/en/LC_MESSAGES/index.mo
building [rinoh]: all documents
updating environment: 2 added, 0 changed, 0 removed
Hello Worldrces... [ 50%] autodoc
reading sources... [100%] index
looking for now-outdated files... none found
pickling environment... 'rinoh_elements/papersize' config variable not set, automatically converting from 'latex_elements/papersize'
'rinoh_logo' config variable not set, automatically converting from 'latex_logo'
'rinoh_domain_indices' config variable not set, automatically converting from 'latex_domain_indices'
done
checking consistency... done
processing target... index autodoc resolving references...
rendering...
Exception occurred:
File "/home/user/.local/lib/python3.7/site-packages/rinoh/frontend/rst/nodes.py", line 183, in build_flowable
toc_id, = self.get('ids')
ValueError: too many values to unpack (expected 1)
The full traceback has been saved in /tmp/sphinx-err-l_ew0g06.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
user@host:~/rtd-github-pages$
I traced it back to the Topic class’ build_flowable() function in ‘/home/user/.local/lib/python3.7/site-packages/rinoh/frontend/rst/nodes.py‘.
For some reason `self.get('ids')` returns a list of two entries, when it expects only one.
['%autodoc#contents', '%autodoc#autodoc']
Meanwhile, the Spanish version returns this, which works fine:
['%autodoc#contenido']
I ended-up just bypassing the issue by removing the “contents” line from ‘autodoc.rst‘
@@ -2,4 +2,2 @@ -.. contents:: - helloWorld.py
It looks like the issue is that the target definition is followed by a directive without first being followed by a section header.
Another solution is to just move the ‘contents‘ directive after the section header:
@@ -1,7 +1,7 @@ .. _autodoc: - -.. contents:: helloWorld.py ============= + +.. contents::
I opened a bug report about this issue with rinohtype.
Limitations & Improvements
This guide is just a simple-as-possible example of how to get started with building a documentation site using rtd on GitHub Pages, and this suggestion will list some of its shortcomings and potential areas for improvement.
Local Docker Builds
The `buildDocs.sh` script is designed to run in Docker on a debian-slim image. In theory, that makes it more portable, but currently there’s no wrapper scripts in-place to run `buildDocs.sh` in a docker container locally.
PRs Welcome!
If you find any issues or would like to make improvements, please feel free to submit a PR for the rtd-github-pages repo.
Tips
This section will provide useful tips to help the reader customize their GitHub-Pages-powered rtd site.
Understanding Theme Templates
Very little documentation was available to help me build a language-specific and version-specific sphinx site with the lower-left rtd menu as described above. In order to get it working, I had to just roll up my sleeves and do a lot of digging through various Read the Docs github repos.
A great place to start this kind of search is the template files.
For example, the template files used by the rtd theme are installed to ‘/usr/share/sphinx_rtd_theme/‘. When trying to find more information about how to setup languages and versions, my first step was grep this dir, which revealed several relevant template files.
user@host:~/rtd-github-pages$ grep -irlE 'version|language|download|menu' /usr/share/sphinx_rtd_theme/ /usr/share/sphinx_rtd_theme/static/js/theme.js /usr/share/sphinx_rtd_theme/static/css/theme.css /usr/share/sphinx_rtd_theme/static/css/badge_only.css /usr/share/sphinx_rtd_theme/breadcrumbs.html /usr/share/sphinx_rtd_theme/theme.conf /usr/share/sphinx_rtd_theme/versions.html /usr/share/sphinx_rtd_theme/layout.html user@host:~/rtd-github-pages$
A quick opening of the above files made it quite clear that the ‘versions.html‘ file was what I needed. And from that file, you can see the jinja variable names used, which are invaluable keywords to help with future grepping and searching through github repos.
In this case, I was able to see that the ‘versions.html‘ file was using the following variables:
- current_version
- versions
- downloads
Read the Docs build logs
Another incredibly helpful tracing path was to find a readthedocs.io site that I wanted to mimic and check their build logs.
For example, one of the most popular sites on rtd that has several languages, versions, and downloads is phpMyAdmin:
If you open the lower-left menu on the above site, you’ll see a section titled “On Read the Docs” with a link to their “Builds”
Click the “Read the Docs” icon in the lower-left to open the lower-left menu
Click “Builds” in the lower-left menu for a list of rtd builds
Click their most recent build (the top-most one), and then click “View raw”
Click the top-most build
Click “View raw” to see the raw build log
The rtd build logs show the contents of conf.py
Among other things, this will show you the conf.py that was used in their build.
Related Posts
Hi, I’m Michael Altfield. I write articles about opsec, privacy, and devops ➡
HI Michael, First of thanks for this blog. I followed your steps and in my local when I `make html`, I see the languages and branches in the nav bar. But when I click dev branch, it says – The webpage at file://en/dev/ might be temporarily down or it may have moved permanently to a new web address.
I have also added you to my Git repo as a contributor and pinged you in Slack. I am seriously spending a lot of time on this and still not able to set up versioning for my project. Can you please help me with this?
Hi Sachin, I see that you have two repos, one that was forked from readthedocs/tutorial-template
* https://github.com/Sachin-Suresh/sphinx-versions-demo
And one that was forked from my rtd-github-pages repo:
* https://github.com/Sachin-Suresh/rtd-github-pages
Does it work on the later? If not, let me know. If so, then I’d recommend doing a diff of the contents of those two repos to isolate the issue.
Hi Michael, no for the https://github.com/Sachin-Suresh/sphinx-versions-demo repo, I dont get the language or versions working.
Hi Michael, thanks for your work on this. I adapted it for my software on a local web server. If I am not mistaken, you have to recompile all branches if a new one appears so that the switcher in older branches can link to recent branches. PyData https://pydata-sphinx-theme.readthedocs.io/en/stable/user_guide/version-dropdown.html uses a json file on a higher level that is read by all switcher instances to avoid recompiling older branches. If this is not an issue for your GitHub approach, please ignore this. I think that it is an important feature that might also improve your merge request with the rtd theme. My capabilities on the web programming are too limited to do this on my own. So I might switch to PyData although I prefer the style of RTD. I think your extension is very useful, and I am wondering why it was not merged into the rtd theme 1.2 yet.