screaming at my screen updates for screaming at my screen - the private domain of Timo Zimmermann en-en Timo Zimmermann Tue, 30 Dec 2014 20:26:43 +0200 Never trust an environment you do not fully control <p>This is usually a suggestion you hear when it comes to sensitive information, personal data and security related tasks. I learned this again over Christmas while working on a small side project and while giving a service a friend of mine is working on a try. <!--MORE--></p> <p>Before we start let me warn you: Do not take this as a guide on how to troubleshoot a problem. It is not a good idea and you never should do it this way if you are working on a serious project. What I did here was a mix of laziness, not being at home for 4 days and we are talking about Christmas, so I was running from one family event to the next one and time to troubleshoot unimportant stuff was pretty rare - I had 2 or 3 tries a day without scarifying any family time, but I really wanted to know if it works.</p> <p>The service I was trying sounded pretty nice - a kind of Heroku alternative, based on what you would expect (docker etc.), but with the difference that you bring your own API keys for a cloud provider and all servers etc. belong to you - they just provision them and charge you some $ a month for the service which means you get a lot saner pricing. I was told <em>everything</em> is documented and there would never be any wired behavior since it is just one process running and not doing much.</p> <p>I hope when she reads this she will not be too mad because I likely messed up her pitch and made it sound a lot lamer and useless than it does when she is pitching you. Since she does not even have a name or anything I also cannot link you to the obligatory big-header-no-content marketing site. I will let you know when there is one.</p> <p>She set me up with a test environment and I requested a 512MB Digital Ocean box for "workers". Everything was pretty painless, pushing to the services git endpoint worked and some minutes later I got a "deployment succeeded" notification. The wired stuff began when my management command ran out of memory.</p> <p>The management command is quite trivial. It is iterating over all keys in an S3 bucket, checks if one of them changed and if this is true downloads the content, unzips and stores the content in the database.</p> <p>The script takes quite a bit of time for the full production dataset (roughly 26 hours), so I only used 2% of the dataset as test data. The code is fully test covered and used real data for tests, so I didn't expect too many stupid errors to make it to the test server. One of the reckless assumptions that turned out to be true - lucky shot :)</p> <p>This is the first time I am using <a href="">boto</a> with lots of keys and in memory fetching and unzipping, so I made a quick list with my first thoughts what could possibly be wrong</p> <ul> <li>a file that is broken / different / too big</li> <li>me messing up the in memory fetching and unzipping</li> </ul> <p>My first attempt was to run the script a few times and look at the key on which it is failing. It was always a different one and the one that failed on the second run passed on the first one.</p> <p>Next I just tried running it from top to bottom always doing only one thing at a time. The first test was to iterate over all keys and print the name and etag, which worked fine and was even pretty fast compared to the expected runtime.</p> <p>Next I fetched all keys and checked if they changed - and the script was killed, again. So my second theory was not even worth testing, something broke way earlier.</p> <pre><code> def logfile_changed(self, name, etag): logfile, created = LogFile.objects.get_or_create(, name=name, ) if created is True: logfile.hashed = etag return True if logfile.hashed != etag: logfile.hashed = etag return True return False </code></pre> <p>Quickly running it wit everything after <code>get_or_create</code> commented out showed that it is still crashing. To get boto out of the picture I replace it with a for loop</p> <pre><code> for x in range(1,9999999999): self.logfile_changed(self, x, x) </code></pre> <p>Guess what - out of memory. So we have a for loop which just runs <code>get_or_create</code> that runs out of memory. Anyone using Django for more than the duration of a bootcamp will likely scream "set debug to False you idiot". But guess what - debug was set to False. At least I thought it was.</p> <p>But from here it was pretty obvious - it has to be the debug flag. Local testing didn't uncover any problems, my server at home completed the import of the full production dataset without going above 150MB for the Python process and suspecting that Django is broken is nearly an as good guess as thinking of a compiler bug.</p> <p>After a short mail describing the problem it turned out that no matter what environment variables I set, the system just refuse to overwrite the ones set by the service and somehow everything ends up in one big pile of environment variable mess. Please do not ask me how they do it, I was too scared to ask. In this case they use the presence of <code>DEBUG</code>, too, to activate the debug mode. Which actually explains why the script was running out of memory. Hardcoding <code>DEBUG=False</code> in <code></code> fixed it. It was a bit troublesome to figure out what was happening, but at least she got a real world example why it is a bad idea to toy with the customers environment without documenting it or showing warnings.</p> <p>The lesson is the same as always: If you do not have full control over an environment - in this case it was not even possible to get a shell or anything like that - you are on the best way to a world of pain when you run into a problem. Even if you think you know and understand the problem, you will eventually not be able to troubleshoot or fix it by yourself. While this bug was relatively easy, there are harder ones out there you can run into. But hey, obscure bugs are the funny ones :)</p> Tue, 30 Dec 2014 20:01:00 +0200 What Start-Ups Can Learn From Blizzard Entertainment <p>I spent a good part of my weekend playing Blizzards Heroes of the Storm Alpha. It is a great game, I really enjoy it and it proves, once again, that Blizzard is really good in bringing new concepts, improvements and innovation to an already existing genre. And they are successful with it. So what can a start-up learn from Blizzard? <!--MORE--></p> <p>If you are a fan of a game I am not talking nicely about please just ignore it and do not fall into a nerd-rage. There is certainly a huge discrepancy how people see certain games and I can only talk about my experience with them. You may certainly disagree and that is totally fine, but this will not change how I and many others, too, experienced those games. So please just ignore the things that make you mad and concentrate on the actual idea behind the article. :)</p> <p>First of all let me make a bold statement: Blizzard is so incredibly successful because they are managing to take a genre and make it fun for everyone, not just people who enjoy it so much that they have the urge to become top players. They do not invent, they improve.</p> <p>Ultima Online was nice. It certainly destroyed grades in school, relationships and many lives, but hey, it was a nice game. And then came World of Warcraft. You could just create an account and play. You run around a bit, do some quests, maybe party with other players. You were not as free as in Ultima Online, you could not learn every profession possible and many things just looked easy, but in exchange you got a lot better story. And people liked it. Not being forced to learn <em>everything</em> about a character and professions for example allowed you to focus more on the game and the gameplay. Compared to today Classic World of Warcraft was garbage, but still a lot better than all the competitors.</p> <p>Real-time strategy games existed before Blizzard released Warcraft: Orcs vs. Humans and Dune certainly helped the genre gaining traction, but Warcraft and later on StarCraft were hit titles. They got most of the pathfinding right, which is extremely valuable when it comes to a fun gameplay, the characters, the story - they just did an amazing job to pull the players into their world (as with WoW). And players stayed. There were surely other successful RTS games, but as far as I know none that came close to the fan base of Warcraft and Starcraft. Maybe Command and Conquer had a good shot.</p> <p>Fast forward - Hearthstone. A trading card game you play on your computer or iPad. I played Magic, I was skeptical and I was right, it is not for me. Too trivial, no depth and not many ways that actual skill determines if you win a match or not. But I have seen my significant other starting to play a genre she never cared about, just because it looked fun, was easy to get in and provided enough longterm motivation to stay and come back when expansions are released. Hearthstone is the most watered down, trivial kind of trading card game I am aware of and at the same time the one with one of the biggest player bases.</p> <p>I predict that the same thing will happen with Heroes of the Storm. It is slower, you have more abilities from the beginning and the gameplay is easier than in Leagues of Legend and DOTA2. But is is fun. You do not have such a big penalty when you mess up a little and have to fall back to your base. You can actually do something the first few levels but pressing one button to harass your opponent and farm. The small side games you play are funny and add to the gameplay and you do not have to consult guides and read all your abilities twice to find a good combination of hero skills, summoner skills and build orders. Confused by all the different things I just mentioned? Many people who do not know the genre would be just as confused as you are. Know what? You do not have to care about all this stuff when playing HotS. Just play. The tutorial will explain everything you need to know and you will be fine after some matches.</p> <h2>Execution Is Everything</h2> <p>There were other phones and other smartphones before the iPhone. There were other car manufacturers before Mercedes and BMW. There were other publishers in the game genres I just talked about. Facebook and Twitter were not the first social networks. And still, companies who were not the first defined the field.</p> <p>I often hear people complaining that an idea, a start-up or a project sucks because someone else did it already. You know what? Nearly every publisher would sell their whole executive team including families to some ancient demon only to get half of the player base Blizzard got. It does not matter if the idea is new. It does not matter if you are late to the game, if others did it and failed, the only thing that matters is your execution. And a certain amount of luck, but that is always true, even if you are the first one.</p> <p>So instead of trying to come up with the best of all ideas to disrupt a market you will first have to define since it does not exist - just do something people really want, but done right. Your chances are far better to succeed and build a successful business with a great product people enjoy than with a product no one knew they needed.</p> <p>I am not saying you should not follow you not-existing-market-disrupting idea. If it is a good idea and you are the first one to execute it right you just win. But if someone asks you what you think of an idea and you hear one that was done so many times - do not shoot it down just because it is not new. It does not need to be new to make it to the top of the market segment it is in.</p> <p>You can discuss this post on <a href="">HackerNews</a>.</p> Wed, 10 Dec 2014 21:40:00 +0200 LeeroyCI - taking down production with automatic deploys since last week <p>Let me give you a quick update about a new feature that made it to <a href="">Leeroys master branch</a>. Deployment. You can now specify a script that will run after a build for a branch was successful. Initially I planned to focus on deployment when working on version 2 or a late 1.x release - for me this is a scary feature and I would have preferred to focus on testing right now. But at FlightCar we decided we want to start leveraging continuous deployment and so I began to work it. <!--MORE--></p> <p>Why did I say this is a scary feature? Leeroy is not only used by me. Some people already trust it with running tests for their start-up or hobby projects. Hopefully more people will start using Leeroy and some of them eventually decide to let Leeroy automatically deploy branches. Deployment is working as everything else, by calling an external script or program. So, in the end, if a deploy is not working, eventually taking down production, you could blame the person who wrote the test or deployment script. In my opinion it is not that easy.</p> <p>I see it as my responsibility to make deploying through Leeroy as secure and convenient as possible. There will be a grace period of likely 30 seconds in which you can cancel a deploy by just clicking on a link. It will involve a lot notifications, in its first incarnation eventually too many. But at the end of the day I want to be sure this feature will, once we hit 1.0, be so rock solid that if a deploy is ever failing or messing up something everyone can easily agree it was not Leeroys fault. </p> <p>The trick will be to keep things simple. One of the reasons why I started working on Leeroy was that the existing CIs are either complicated or impotent. I want to keep Leeroys simplicity even while offering features like automatically deploying your latest changes. I am satisfied with its current state - or it would not be in master. It is deploying two Django based environments under my watch right now, I use it to build Leeroy branches and upload the binaries to S3 and I am about to configure it for drupan this weekend to publish changes in master to pypi.</p> <p>Okay, that is enough whining. Give it a try, the <a href="">configuration</a> is simple and if you are on Heroku or Elastic Beanstalk a deployment script can look as trivial as this</p> <pre><code>#! /bin/bash cd /home/ec2-user/test-deploy git fetch get reset --hard origin/master git aws.push </code></pre> <p>So not much magic involved here. Do not worry about deploying broken code, it will only deploy if all tests pass.</p> <h3>Preview</h3> <p>Some things you will see happening shortly:</p> <ul> <li>custom templates (this is already in testing)</li> <li>simplified notification system (eventually with custom notifications)</li> <li>more notifications (deployment announcements e.x.)</li> </ul> <p>And after I got those done the next two bigs steps are</p> <ul> <li>admin interface (including user support)</li> <li>notifications about successful deploys</li> </ul> <p>This is what I am planning to get done this year. Ignoring the GitLab and BitBucket integration for a moment (this does not mean it will not be done), the <a href="">roadmap for 1.0</a> looks pretty good and suggests a 1.0 beta between Christmas and New Year.</p> Thu, 30 Oct 2014 20:13:00 +0200 Why I am writing a web interface for a static site generator <p>Let us talk about drunken pandas, sake and why I am trying to tie a static site generator to a SQL database and a Django process. There are actually two simple reasons: The first one is that I want more comfort while blogging without losing the advantages of drupan. The second one is that I hope to enable more people to use a static site generator and own their content without losing any comfort or having to put up with managing a small, likely virtual, server. <!--MORE--></p> <p>While <a href="">drupan hit 2.0</a> some weeks ago and is doing pretty well I decided to take on a project <a href="">I already talked about</a>. Making it easier to create content, bringing features of full blown content management systems to the world of static site generators and still keeping the whole thing simple enough that lots of people will be able to set it up and use it. As I already mentioned one of the reasons is that I want more comfort, but let us focus on the other one for now.</p> <p>There is certainly a trend to use a service that provides you a hosted blog, no matter if Blogger, Svbtle or Medium, according to rumors there is even one person left publishing stuff on LiveJournal. But in my opinion you give up too much just for the convenience of not having to take care of a bit software. It varies from service to service, sometimes you do not own your content, sometimes you have to live with the fact that it could be required at one point to publish things under your real name, no matter what the consequences would be, sometimes the service tries everything so the content you create is not associated with you but the service, sometimes the service is just a mess and the downtime is higher than the uptime. There are so many reasons why hosting your own content is a good idea. But why do more and more people favor hosted services?</p> <p>One word: Convenience. You enter your email and a password and start typing. That is it. If it would be a bit more trivial someone would create and send you your credentials the moment you are born. A self hosted solution will never be able to provide this level of convenience. But there are certainly ways to get closer to it than what we have right now.</p> <h3>Uptime, performance and security</h3> <p>Those three points are likely the three biggest offenders when it comes to convenience. How many blogs did we see going down because there was an article making it to the front page of HackerNews or reddit? How often do you have to update your system and your blogging software to have a vague chance of keeping everything secure? How often do cheap, and sadly also sometimes expensive, VPS providers fail when it comes to reliability?</p> <p>I can understand anyone who does not want to administrate yet another VPS. There are only two options to get around this: use a hosted blogging service or a managed server or platform. The first and most important goal for drupan and Sakebowl is to solve those three problems. Let us take a look at the idea.</p> <p>Drupan deploys your site to AWS S3 + CloudFront. CloudFront and S3 are doing an awesome job keeping static files online. And I believe not many people are willing to try to achieve the same level of reliability than those services offer.</p> <p>You can run Sakebowl wherever you want. On your laptop? No problem! One click deployment to Heroku using the Heroku button? Sure. On the server in your basement or a VPS? Despite trying to solve the problem of having to take care of a VPS it is certainly possible and painless. With Sakebowl you get an intuitive web interface, your data stored in a database and with one click you generate and deploy your site using drupan. Sakebowl does not have to be online or reachable by the whole world to do its job.</p> <h3>How far did I get and what will Sakebowl look like?</h3> <p>At least that is the idea. I started working on Sakebowl, making sure it can import existing drupan sites and generate and deploy them. It is hard to start with the minimal necessary functionality and keep a certain target audience in mind - websites and blogs - since drupan is designed to get out of our way no matter what you create with it. But it is currently focusing on exactly this, websites and blogs. Features like multiple users with roles are something I already thought about, but they will have to wait. I always appreciate feature requests, but anything beyond the basics would end in many blog posts and no results.</p> <p>The primary focus for the beginning will be deployment and setup. The setup process should look like this</p> <ul> <li>click on Heroku button</li> <li>enter AWS credentials</li> <li>answer some simple questions (site name etc.)</li> <li>write your first blog post</li> </ul> <p>The setup will be 100% automatic, you will not have to configure the whole AWS stack yourself. Sure, it involves a Heroku and AWS account. This is not targeting end users who have trouble singing up for both services. The target audience is the technical crowd who is fine with following an easy tutorial that involves ~ 20 steps to get their blog online, knowing that they do not have to put up with the usual problems that occur <em>after</em> the setup.</p> <p>I want to use Sakebowl, so it will happen. Eventually it will also help other people to own their content without the usual troubles, at least I hope so. If you have any ideas how the process should look like, what features it will have to ship to be usable or if you want to discuss that I should stop wasting your time with blog posts about projects I am working on that sound like sales pitches - yeah, after reading it I am aware of that - just mail me, I'd love to hear your thoughts.</p> Mon, 06 Oct 2014 20:00:00 +0200 Roadmap to Leeroy 1.0 <p>Leeroy CI is evolving just fine and in the timeframe I expected it to move forward. I have not seen any problems in production and it is already helping some developers and startups with automated builds and tests and harassing them to make sure all tests are passing before merging a pull request. Let me give you an update how far we are from the magical 1.0 release. <!--MORE--></p> <p>If you have not already seen it, I opened an <a href="">issue</a> on GitHub where I list and check off all features that will make it in 1.0.</p> <h2>Notifications</h2> <p>First of all I am happy that the major notification systems are completed. With Slack, HipChat, Campfire and Email the most important tools should be covered. There is one thing missing though and I am still thinking about the best way to implement it: IRC.</p> <h3>3rd Party Libraries</h3> <p>Currently Leeroy only uses Golangs standard library. This is pretty nice considering that dependency management in Golang is still suboptimal and that many libraries you find are not always that well maintained or stable. I would prefer keeping it that way.</p> <p>Integrating with the services I mentioned was no big deal. They all provide a nice API and you only have to make one request and send your API key together with the message to display - or in the case of mail, well, the big advantage of Email is that it is supported by basically everything, including the standard library.</p> <p>Writing an IRC client would be a bigger task than just making a request to an URL and drastically increase one part of the system for a feature with questionable value. To solve this I am considering adding some kind of webhook system. It does not matter which service is notified by the webhook and it can focus on doing one job well.</p> <h2>Integrate All The Things</h2> <p>GitLab, compared to GitHub, is a pain to integrate. You do not always get the SHA1 of a commit, the branch or other important information. This means the integration will likely have to do a lot requests compared to GitHub where you just have to parse the payload the webhook sends you.</p> <p>BitBucket on the other hand should be a lot easier. But with the BitBucket integration I also want to make sure that the Mercurial is supported properly, eventually refactoring the existing code a bit and keeping the naming of the variables and structs either the same across all implementations or, more likely, creating an interface so it does not matter which VCS is triggering a build.</p> <h3>GitBucket, Gogs and Gitorious</h3> <p>I am not only considering supporting <a href="">GitBucket</a>, <a href="">Gogs</a> and <a href="">Gitorious</a>, I am actively looking into possible integrations. But as far as I am concerned GitLab and BitBucket are the most important services to support so I will start with them and move the other three to a later release. Of course, this depends on the level of webhook support those services will offer when I start working on the integration.</p> <p>Currently it looks like Gogs will be the first one to be fully supported. I am considering to use it for personal projects I do not want on GitHub, so this would be a good reason.</p> <h2>Configuration and Authentication</h2> <p>Those two features are the biggest ones and will result in the most refactoring and it will, with nearly 100% certainty, end in the current configuration syntax changing. The end result will be a web interface so you do not always have to SSH to your CI box to configure something. But to allow web based configuration we first should have a concept of users.</p> <p>So while being the two biggest features they also depend on each other to some degree. The most likely thing to happen is:</p> <ul> <li>new configuration syntax / style</li> <li>configuration validation</li> <li>user account support</li> <li>web based configuration</li> </ul> <p>The new configuration style, which will break the current one, will bring keys and tokens for services on a per repository basis, named repositories getting rid of the hex representation in the URL and other nice things. Most of my thoughts are currently related to this feature. I would prefer to never change the configuration format again. Of course this is unlikely to happen if I do not abandon the project but I would prefer to keep the configuration the same as long as possible.</p> <h2>Overall</h2> <p>I am really satisfied with how Leeroy turned out. It is working great from what I can tell and it never got in the way - that is more than I can say for other software and tools I use daily.</p> <p>There are some parts I will refactor to become more idiomatic Golang and easier to extend. I actually choose to move this back as long as possible to see what changes are required to support all services, makes the whole process and design a lot easier. So there is still some work to do before starting with the major features.</p> <p>I cannot give you a fixed date when 1.0 will be released. I am still working on some features for <a href="">drupan</a>, including the webinterface to make posting a pure joy and letting you believe in Santa and unicorns again, but since I also do not like the "it is done when it is done" statements let me give you a vague idea: I am giving my best to release 1.0 with all promised features by the end of 2014.</p> Mon, 08 Sep 2014 23:10:00 +0200