This post touches Python only indirectly, but a server written in Python still plays a major role. The topic is how to create a proxy in Apache for a WebSocket server built on the Tornado Python framework.

The first actor is a WebSocket server built on Tornado. Tornado is strong when hundreds of open connections need to be handled with non-blocking I/O in one or more processes. We chose it mainly because it makes creating a WebSocket server in Python simple.

Creating the WS server means implementing a handler derived from WebSocketHandler. From that class we can handle the basic WS primitives: open, on_close, on_message, and write_message. The implemented handler is then attached to a Tornado HTTP application, for example:

application = tornado.web.Application([
    (r'/tts/mp3/(.*)', ResultHandler),
    (r'/tts/wav/(.*)', ResultHandler),
    (r'/tts', WSHandler),
])

For our purposes, ResultHandler is a descendant of StaticFileHandler and serves static files from disk, while WSHandler is the WebSocket handler.

The application worked well, connected to the HTML front end without trouble, and nothing suggested future problems.

A Few Months Later

A few months later, we needed SSL, or HTTPS, on the server. Because we wanted to run HTTPS for all services on the domain on a single port, we decided to proxy everything through Apache listening on port 443. Part of the traffic would be served directly, and part would be passed to the Tornado server on localhost port 8001.

A quick search showed that Apache 2.4 from Debian testing already contained mod_proxy_wstunnel, which can proxy WebSockets in almost the same way as ordinary HTTP traffic. The Apache configuration could contain:

ProxyPass /tts-ws ws://127.0.0.1:8001/

At first everything worked. Requests to ws://server/tts-ws/tts were handled by the Tornado back end exactly as needed. But once a non-trivial number of WebSocket connections existed, Apache started behaving unpredictably. It even seemed that this proxy configuration was attracting surrounding traffic, and Tornado returned 404 errors for pages that existed and worked after a refresh or in another window.

Where Did the Bug Probably Happen?

Finding the answer took almost two whole afternoons. The solution was trivial and completely logical once found, but it was not written down in the documentation or forums. The problem was that the /tts-ws proxy path also handled ordinary HTTP traffic, namely /tts-ws/tts/mp3 and /tts-ws/tts/wav. Because WebSockets are designed as an HTTP upgrade, this works at first. But when Apache starts reusing proxy workers and connections start alternating between them, unpredictable things happen.

The solution is to proxy HTTP traffic on the /tts-ws/tts/* paths explicitly:

ProxyPass /tts-ws/tts/mp3 http://127.0.0.1:8001/tts-ws/tts/mp3
ProxyPass /tts-ws/tts/wav http://127.0.0.1:8001/tts-ws/tts/wav
ProxyPass /tts-ws ws://127.0.0.1:8001/

The HTTP proxy rules for the subpaths must be listed before the WebSocket proxy for the parent path.

When the cause was found and fixed, I mentally rewarded myself for the brilliant idea of designing the WebSocket server so that the parent path pointed to the WebSocket handler while its children were served through ordinary HTTP.

The fix again belongs to the “easy once you know it” category, but the road to it was thorny and long, full of experiments and testing. It was also a road with no debuggers or debug prints, much like the first article Where Did the Bug Probably Happen?.