posted on December 03, 2015 in drupan, project, sakebowl

Drupan 2.2.0 - the panda gets a bowl of Sake

It has been a long time since I talked about drupan. I put a lot of time into thinking about the best way to prepare drupan for Sakebowl. This was not as straight forward as it sounds. But the solution I found works quite well, improves different parts of the system and is also fully compatible with the 2.x branch, so no need for a new major release.

The biggest change was to decouple all plugins. In its current form the filesystem writer required to know about the template directory. S3 deployment wanted to know the directory the filesystem writer wrote the site to. On its own it was straight forward and not too different than what other static site generators do. But, and this is an important but, that does not work anymore when Sakebowl joins the game.

Sakebowl will store the whole site in the database. When you hit deploy, a new instance of drupan will be create, the template and content directory will be populated, the site generated and the result will be deployed. One thing that will not be used is the filesystem writer - no more temporary directories, no slow spinning disks, just the whole site in memory.

With recent changes you do not need any reader or writer, you just set the relevant dictionaries yourself and generate your site. Everything will happen in memory, which was something I initially wanted to avoid. When you run drupan locally it will read all assets, images and template files and store them in memory. So if you decide to generate a 20GB photo portfolio you better have enough RAM or swap. At least for now, changing this is at the top of my to-do list.

After talking to users who actually use drupan for photo portfolios I discarded all worries. The average size of a drupan site I am aware of is roughly 1.3MB. Sites that use lots of photos go up to 156MB average. The biggest site that came up is 1.6GB. Most people who have videos or lots of photos already store them on S3 or a third party hoster, so they will never be held in memory when the site is generated. And no one with a site bigger than 5MB generated it on an RaspberryPI. So at least the users I know will be fine. Please let me know if you see any problems with the new design or if you have a use case where this will be problematic.

Another thing that changes is that no MD5 files will be written anymore. Creating the hash happens in memory and it will be compared to the etag (S3s MD5 sum with unnecessary quotation marks) that is part of the S3 key. This seems to work pretty reliably so far. At the same time this puts more work on the deployment plugin since it will be in charge of the decision if an entity is uploaded or not.

One thing most first time users will care about is the fact that the configuration got a lot easier and more straight forward than before. The number of required configuration options was reduced a little bit and since drupan now also supports optional configuration options some more can be avoided.

Speed, speed, speed

Those things combined also improve the time needed to generate deploy a site quite a bit. Let us look at a normal use case where you write a new post, you add an image (200kb), you write the site to your disk, you deploy to S3 and invalidate CloudFront.

python ~/projects/drupan/drupan.py config.yaml 0,36s user 0,09s system 4% cpu 9,397 total

Using Sakebowl this will even be a bit faster. The limiting factor will likely be the upload and invalidation request.

Most of the time you probably do not care about the speed of this process. At least not if we do not talk about minutes to publish 50 words. But it was a nice side effect and also allows me to work on a preview feature rendering the content using the correct template, not just interpreting markdown.

To sit a panda into a bowl of sake

I will not make any promises when something you can actually have a look at will be ready. I am still preparing a drupan 2.3 release. Once that is done I will start on a basic implementation of Sakebowl, likely using stock bootstrap and no fancy features.

Once everything works two things will happen:

  1. It will get a nice UI (do not worry, I will not design it myself)
  2. Some usability features allowing non tech-savvy users to edit a page will be added

Point two is something I am especially interested in. Allowing anyone to edit a site, no matter what background they got, while having all the awesomeness of a static site generator is something I consider a great thing - and while I am aware of some projects that try to do that, all of them (at least in my opinion) do not hit the sweet spot of usability, features and performance.

This will also take priority over the features I and most readers of this blog would likely like to see - posting via email, pushing content via git, a nice API and all the stuff we all love to play around with on a daily basis. But it will not be delayed for too long. Pinky swear.

I would love to talk to you about this post, your ideas or awesome projects.

I am @fallenhitokiri on Twitter and GitHub or you can send me a mail.