Week 2: IPython - Proof of Concept

Hello reader,

I have done an another proof of concept using IPython widgets.

In this proof of concept, we aimed to re-implement the same thing that we did before. But of course with IPython notebook.

The main idea was to have a widget with a string (Unicode) trait attribute. This trait should contain the base64 encoded representation of the PNG image. This widget captures the screen on every update and saves this image. Then encodes this image with base64. As soon as it updates its value with encoded representation of the PNG, the JavaScript part also updates itself simultaneously. This is the power of the IPython's widget machinery.

We have one problem though. We are capturing the screen with glReadPixels(). This works fine, but in the end we have numpy.ndarray in our hands. So, in order to have a PNG image, we were using PIL image library and its Image.fromarray() function to convert the ndarray into a PNG. Then, we were saving this image onto cStringIO buffer and encoding with base64. Unfortunately, this approach is very slow. We were also having this issue on the past proof of concepts, however we solved this problem using threading. But then, after a discussion we have agreed upon not using threads.

In order to solve this problem, one of the Vispy-devs, Luke Campagnola, has done a great job, wrote a pure python PNG writer. This helped us remove the PIL dependency and the need of saving onto a buffer. After some benchmarks we saw that it was the zlib.compress() function which slows down the process. But since this is still a proof of concept, we decided to let it go.

I think this proof of concept also went well. What we need now is some interactivity. We will listen user events with JavaScript and send them to Vispy to handle them. For this purpose, we are going to design a protocol (Wow, I am going to design a protocol!). In this protocol we will define the JSON representations of user events.

Next stop: Design of a JSON protocol
See you next week!