A bit of documentation

(Most up-to-date version is located in the repository on bitbucket)

Sphinx comments/fixes web application

Sphinx provides you with a feature of building your documentation as a web application, which gives you a way to interact with your users.

Users are permitted to submit comments as well as their fixes for the documentation. Developers have additional rights of deleting comments/fixes and committing fixes to the documentation repository.

Building a webapp along with documentation

Building your documentation along with a web application, which will serve it, is almost as simple as building the documentation itself.

1. Start a new Sphinx project:

$ mkdir sphinx-project

$ cd sphinx-project

$ sphinx-quickstart

[ …usual sphinx-quickstart questions… ]

2. The documentation is empty as for now. Create at least one file with some content. Now, you can build a webapp which will serve your documentation:

$ make webapp


Build finished. The webapp HTML pages are in _build/webapp.

3. Run the server:

$ cd _build/webapp/

$ python server.py

Running at…

OK, there it goes. Now you can check it with web browser!

The webapp configuration file

The variables and values of webapp.conf file are based on conf.py file.

NOTE: Always change the values of variables in conf.py file.

  • licence – A name of the licence for fixes submitted by the users. Default is ‘BSD’.
  • reporoot – The name of a directory in which the repository files should be located in build directory. Default is ‘repo’.
  • repodir – If the documentation is located in a subdirectory of a repository, this should be a directory name of the documentation. For example, when the files for Sphinx are copied/cloned to reporoot directory, the directory for documentation in the repository is: _build/webapp/repo/doc/, because the documentation for Sphinx is held in sphinx/doc directory. Default is doc, which works well for Sphinx, but may fail for your project. Please, change it accordingly.
  • repopath – This variable is only visible in webapp.conf file. Its value is created by joining reporoot and repodir values.
  • reposums – A filename for MD5 sums of files from the repository. Default is reposums.pkl.
  • piddbfile – A filename for a database which sorts paragraphs by repository filename and paragraph ID. Default is piddb.pkl.

File structure of the build

After building your documentation, the _build/webapp/ directory gets populated with a number of directories and files.


  • comments – a database for all the comments added to any paragraph.
  • fixes – a database for all the fixes added to any paragraph.
  • openidstore – OpenID logging database
  • html – HTML templates used by the webapp
  • public – your documentation generated by Sphinx
  • repo – source files of your documentation in a repository
  • xapian_db – a database for Xapian search engine


  • server.py – a simple WSGI server, useful for testing purposes
  • webapp.conf – a configuration file for the webapp (based on values from Sphinx’s conf.py file
  • developers.txt – a list of OpenIDs of users who should be granted developer’s rights
  • piddb.pkl – a database of paragraphs sorted by filename and paragraph ID
  • reposums.pkl – a database of MD5 sums for files in repo directory. Whenever a sum is not matching reality, the source file is rebuilt.

All for now, folks!


Full picture

To make it easier to imagine what’s my project about, I’ve created a chart which presents the main idea of the Sphinx’s webapp in action:

It’s not, how it works at the moment – it’s how it will look at the end of GSoC. What I need to add is version control system integration and a cache system for the HTML docs. And that’s what I am working on now.

A bit of update

Basics for theming is already there. I still have to improve it and rethink its API to make it easier to replace SinglefileHTMLBuilder (*not StandaloneHTMLBuilder) with user’s own theming class (I guess some advanced users would like to have a choice in this regard. I also have to move this class somewhere else, there’s no need to keep it in a webapp module which is responsible for handling actions associated with comments/fixes.

Some other changes: a file for OpenID logins of developers was added. Some actions (like deleting comments/fixes) will be available only for logged in users whose logins are listed in this file. Also, cookies were added for comments rating.

Now I am diving into anyvc since the last coding-based phase of my project has just started. It’s integrating diffs view with the VCSs. After this task is done there are two weeks more left of GSoC. In my application I reserved this time for writing documentation and testing application but I think that I will spend a part of this time on coding anyway.

Theming headache

Mid-term evaluation is long gone (I passed! :-)) and new code is being written. In my last post I said that next items on my TODO list are: threading for comments (done), displaying diffs in fixes view (done) and adding a configuration file for webapp level. The last item on the list is still undone. Instead of working on it, I was spending my time on theming issue.

Sphinx’s webapp builder is not building HTML files. Instead, it creates pickled dictionaries for most of the files. Every pickled file contains only the contents (in ‘body’ key of the dictionary), htmlized text itself, but it does not contain a full HTML code. It is a user’s choice how to transform these pieces of text and HTML into a fully functional HTML page.

To make it possible to produce a full HTML page on webapp level I have to create an environment for StandaloneHTMLBuilder. To make it possible, I need to create an instance of the main Sphinx’s class called – unsurprisingly – Sphinx. Even if I do it, there are many details that still need to be handled (Sphinx class and StandaloneHTMLBuilder were created to work on the source files of a project and not with – looking from Sphinx’s perspective – built project).

Some basic code on this issue was just comitted to the repository, but it’s not even a half of the work that needs to be done with theming. Long and challenging way to go, but with mid-term money from Google I can afford as much coffee as I want to, so it’s not that bad 🙂

Mid-term summary

It’s mid-term evaluation time, so a bit of summarization seems like a good idea.

First, I learned a whole lot during first phase of coding. The greatest discovery for me is jQuery. If I knew it before I would construct my schedule in a bit other fashion. Next, I learned how to use Xapian and OpenID. I also learned Pylons and liked it, but decided to drop it at this stage and use WSGI + WebOb instead.

Second, I improved my skills in using Mercurial (and Bitbucket!). 41 commits since May 30, a few problems in the meantime, and I am quite experienced in using it now.

Third, I learned more about Sphinx (and hopefully improved my Python coding style too while reading its code base), and have spent some time with Pocoo team and really enjoyed it, thanks guys!

Now, a few words about the project. I think I am on time with my schedule. The only problem is that there are some unscheduled tasks that I have to/would like to finish. First, I need to add theming support on webapp level. Second, the comments/fixes views and a post form should be customizable on webapp level. Basically, it’s about moving some functionality from Sphinx’s builder level to webapp.

Xapian and OpenID support is done. When it comes to talk about comments/fixes, currently it is possible to add comments and fixes to two separate databases, display submitted comments/fixes, sort them and rate them. When one wants to fix a paragraph, a reST’s rawsource is displayed, so one is changing exactly the text which lies in the repository. VCS support is not added yet, though. It’s scheduled for July 23 – July 31.

The closest items on my TODO lists are:

  • adding threading to comments and fixes views,
  • adding displaying diffs instead of the text for submitted paragraphs (it’s easier to compare the changes then),
  • adding some configuration file on webapp level, because the number of things which are configureable is growing steadily.

That’s all for now, all I can do is waiting for the results 🙂

Scoring/sorting comments

The more I work with jQuery the more I like it. It made it possible to add basics for scoring comments (+1/-1) and sorting them by score or date quite quickly. Now it’s time for post form validation (should be easy with jQuery Form) and a view for diffs.

Two screenshots (right click and ‘show picture’ for fullsize):

a) sort by date:

b) sort by score:


All for now!

Adding/displaying comments

Yup, it’s really far from being perfect, but basic adding comments under the paragraph and displaying them is done. At the same time I am switching from StandaloneHTMLBuilder (produces HTML files) to SerializingHTMLBuilder (produces pickled files) and thus it’s hard to show how it really works. The switch from one builder to another caused some problems with theming support and with producing a bunch of interconnected html files.

Nevertheless, some screenshots from the battlefield (click right button on the picture and then ‘show picture’ to see it in full size):


Now, let’s try to add a comment to second paragraph:



Now we can fill out the form:



…and after pressing the ‘submit’ button the comment is added to the page and to the database:



Now it’s time for a bit of form validation. Cheers!