A bit of documentation

(Most up-to-date version is located in the repository on bitbucket)

Sphinx comments/fixes web application

Sphinx provides you with a feature of building your documentation as a web application, which gives you a way to interact with your users.

Users are permitted to submit comments as well as their fixes for the documentation. Developers have additional rights of deleting comments/fixes and committing fixes to the documentation repository.

Building a webapp along with documentation

Building your documentation along with a web application, which will serve it, is almost as simple as building the documentation itself.

1. Start a new Sphinx project:

$ mkdir sphinx-project

$ cd sphinx-project

$ sphinx-quickstart

[ …usual sphinx-quickstart questions… ]

2. The documentation is empty as for now. Create at least one file with some content. Now, you can build a webapp which will serve your documentation:

$ make webapp

[…]

Build finished. The webapp HTML pages are in _build/webapp.

3. Run the server:

$ cd _build/webapp/

$ python server.py

Running at 127.0.0.1:8000…

OK, there it goes. Now you can check it with web browser!

The webapp configuration file

The variables and values of webapp.conf file are based on conf.py file.

NOTE: Always change the values of variables in conf.py file.

• licence – A name of the licence for fixes submitted by the users. Default is ‘BSD’.

• reporoot – The name of a directory in which the repository files should be located in build directory. Default is ‘repo’.

• repodir – If the documentation is located in a subdirectory of a repository, this should be a directory name of the documentation. For example, when the files for Sphinx are copied/cloned to reporoot directory, the directory for documentation in the repository is: _build/webapp/repo/doc/, because the documentation for Sphinx is held in sphinx/doc directory. Default is doc, which works well for Sphinx, but may fail for your project. Please, change it accordingly.

• repopath – This variable is only visible in webapp.conf file. Its value is created by joining reporoot and repodir values.

• reposums – A filename for MD5 sums of files from the repository. Default is reposums.pkl.

• piddbfile – A filename for a database which sorts paragraphs by repository filename and paragraph ID. Default is piddb.pkl.

File structure of the build

After building your documentation, the _build/webapp/ directory gets populated with a number of directories and files.

Directories:

• comments – a database for all the comments added to any paragraph.
• fixes – a database for all the fixes added to any paragraph.
• openidstore – OpenID logging database
• html – HTML templates used by the webapp
• public – your documentation generated by Sphinx
• repo – source files of your documentation in a repository
• xapian_db – a database for Xapian search engine

Files:

• server.py – a simple WSGI server, useful for testing purposes
• webapp.conf – a configuration file for the webapp (based on values from Sphinx’s conf.py file
• developers.txt – a list of OpenIDs of users who should be granted developer’s rights
• piddb.pkl – a database of paragraphs sorted by filename and paragraph ID
• reposums.pkl – a database of MD5 sums for files in repo directory. Whenever a sum is not matching reality, the source file is rebuilt.

All for now, folks!

Full picture

To make it easier to imagine what’s my project about, I’ve created a chart which presents the main idea of the Sphinx’s webapp in action:

It’s not, how it works at the moment – it’s how it will look at the end of GSoC. What I need to add is version control system integration and a cache system for the HTML docs. And that’s what I am working on now.

A bit of update

Basics for theming is already there. I still have to improve it and rethink its API to make it easier to replace SinglefileHTMLBuilder (*not StandaloneHTMLBuilder) with user’s own theming class (I guess some advanced users would like to have a choice in this regard. I also have to move this class somewhere else, there’s no need to keep it in a webapp module which is responsible for handling actions associated with comments/fixes.

Some other changes: a file for OpenID logins of developers was added. Some actions (like deleting comments/fixes) will be available only for logged in users whose logins are listed in this file. Also, cookies were added for comments rating.

Now I am diving into anyvc since the last coding-based phase of my project has just started. It’s integrating diffs view with the VCSs. After this task is done there are two weeks more left of GSoC. In my application I reserved this time for writing documentation and testing application but I think that I will spend a part of this time on coding anyway.

Theming headache

Mid-term evaluation is long gone (I passed! :-)) and new code is being written. In my last post I said that next items on my TODO list are: threading for comments (done), displaying diffs in fixes view (done) and adding a configuration file for webapp level. The last item on the list is still undone. Instead of working on it, I was spending my time on theming issue.

Sphinx’s webapp builder is not building HTML files. Instead, it creates pickled dictionaries for most of the files. Every pickled file contains only the contents (in ‘body’ key of the dictionary), htmlized text itself, but it does not contain a full HTML code. It is a user’s choice how to transform these pieces of text and HTML into a fully functional HTML page.

To make it possible to produce a full HTML page on webapp level I have to create an environment for StandaloneHTMLBuilder. To make it possible, I need to create an instance of the main Sphinx’s class called – unsurprisingly – Sphinx. Even if I do it, there are many details that still need to be handled (Sphinx class and StandaloneHTMLBuilder were created to work on the source files of a project and not with – looking from Sphinx’s perspective – built project).

Some basic code on this issue was just comitted to the repository, but it’s not even a half of the work that needs to be done with theming. Long and challenging way to go, but with mid-term money from Google I can afford as much coffee as I want to, so it’s not that bad 🙂

Mid-term summary

It’s mid-term evaluation time, so a bit of summarization seems like a good idea.

First, I learned a whole lot during first phase of coding. The greatest discovery for me is jQuery. If I knew it before I would construct my schedule in a bit other fashion. Next, I learned how to use Xapian and OpenID. I also learned Pylons and liked it, but decided to drop it at this stage and use WSGI + WebOb instead.

Second, I improved my skills in using Mercurial (and Bitbucket!). 41 commits since May 30, a few problems in the meantime, and I am quite experienced in using it now.

Third, I learned more about Sphinx (and hopefully improved my Python coding style too while reading its code base), and have spent some time with Pocoo team and really enjoyed it, thanks guys!

Now, a few words about the project. I think I am on time with my schedule. The only problem is that there are some unscheduled tasks that I have to/would like to finish. First, I need to add theming support on webapp level. Second, the comments/fixes views and a post form should be customizable on webapp level. Basically, it’s about moving some functionality from Sphinx’s builder level to webapp.

Xapian and OpenID support is done. When it comes to talk about comments/fixes, currently it is possible to add comments and fixes to two separate databases, display submitted comments/fixes, sort them and rate them. When one wants to fix a paragraph, a reST’s rawsource is displayed, so one is changing exactly the text which lies in the repository. VCS support is not added yet, though. It’s scheduled for July 23 – July 31.

The closest items on my TODO lists are:

• adding threading to comments and fixes views,
• adding displaying diffs instead of the text for submitted paragraphs (it’s easier to compare the changes then),
• adding some configuration file on webapp level, because the number of things which are configureable is growing steadily.

That’s all for now, all I can do is waiting for the results 🙂

Scoring/sorting comments

The more I work with jQuery the more I like it. It made it possible to add basics for scoring comments (+1/-1) and sorting them by score or date quite quickly. Now it’s time for post form validation (should be easy with jQuery Form) and a view for diffs.

Two screenshots (right click and ‘show picture’ for fullsize):

a) sort by date:

b) sort by score:

.

All for now!

Adding/displaying comments

Yup, it’s really far from being perfect, but basic adding comments under the paragraph and displaying them is done. At the same time I am switching from StandaloneHTMLBuilder (produces HTML files) to SerializingHTMLBuilder (produces pickled files) and thus it’s hard to show how it really works. The switch from one builder to another caused some problems with theming support and with producing a bunch of interconnected html files.

Nevertheless, some screenshots from the battlefield (click right button on the picture and then ‘show picture’ to see it in full size):

.

Now, let’s try to add a comment to second paragraph:

.

.

Now we can fill out the form:

.

.

…and after pressing the ‘submit’ button the comment is added to the page and to the database:

.

.

Now it’s time for a bit of form validation. Cheers!

The file structure of my project

At the moment the directory structure of my project built upon Sphinx’s dir structure looks like this:

builders/webapp/ -> webapp builder used by Sphinx to build the docs and the webapp,
builders/webapp/templates/ -> the *.py template files rendered by webapp builder (at the moment one file – server.py – lies here, but actually this file is simply copied, not rendered, but it might change in the future),
builders/webapp/templates/html/ -> the templates of HTML code rendered by the middleware (imported by server.py) when serving the docs,

web/middleware/ -> middleware for Xapian, OpenID and serving html files in general,

themes/basic/web/ -> templates for full HTML files. These files are used as templates twice. First, when Sphinx is building the docs, it extends these files with “layout.html”, and then, for a second time, when the files are accessed through the browser. That’s why these files are separated from those located in builders/webapp/templates/html/ directory, which are rendered only once, by the middleware.

A bit of mess, isn’t it? Comments are appreciated 🙂

The structure for generated docs and webapp looks like this (to be found in _build/webapp):

html/ -> files from builders/webapp/templates/html and themes/basic/web directories are located here. We don’t have to separate them anymore, because they are all used as templates by middleware now.

openidstore/ -> OpenID directory. It is created and administered by python-openid library.

public/ -> files built by Sphinx go here. This is the directory which should be served

xapian_db/ -> Xapian’s database.

server.py -> a script which imports the middleware and serves the docs. The idea is that: by adapting the code from this file to one’s framework, one should be able to use this webapp in his python web framework of choice. I plan to change it soon, so all server.py logic will be available as a middleware, because the middleware (Xapian, OpenID, etc.) pieces are actually unseparable, so there is no need for an ordinary user to care about importing them all.
Nevertheless, that’s how it looks like now. It may all change in the future, but if we are about to discuss it, it’s good to describe it as it is first.

Next step: OpenID logging

OK, it is now possible to login with one’s OpenID login after the docs and webapp for serving the documentation were generated. It took me some time to make it work, mainly because Paste’s module for OpenID logging – paste.auth.open_id – is rather out of date.

What I needed to do to make it work was to put together the WSGI architecture and the OpenID logging logic. The examples from python-openid package were very helpful, though. It is working already, but there are some problems I still need to resolve. To name a few: add a way to distinguish admins from ordinary users (anyone can log in, but only choosen users have special rights), add a “logged in as…” line to the templates (the problem here is that the docs are used as templates twice: when Sphinx generates the docs and later on when webapp is displaying them).

Nevertheless, you can try it just as described in earlier posts. Also, until June 22 a form for posting comments should be ready. Seems like busy weekend! 🙂

Integration of Xapian

It already works. I still can point to some problems, but the Xapian search engine is quite usable and it acknowledges themes choosen by the user.

A “search” form for Xapian is placed under a traditional “quick search” form. You can already build your documentation with this addition. Simple “make webapp” makes the job.