screaming at my screen updates for screaming at my screen - the private domain of Timo Zimmermann en-en Timo Zimmermann Wed, 14 Oct 2015 21:27:18 +0200 Bringing LeeroyCI to the next level <p>LeeroyCI started out as an simple CI that gets out of your way and at the same time is powerful enough to cover the needs of small to medium sized organizations. And you can see that in some design decisions, like not using a database but only plain files and JSON. While LeeroyCI has proven to be the right tool for the job, it was lacking some features a „modern CI“(tm) should provide. Let me tell you about all the upcoming changes and why LeeroyCI will be better than ever while still being simple. <!--MORE--></p> <p>Before starting to add features I talked to other users I know of, in different company sizes. It ranged from start ups to mid sized agencies and the largest one I know of is a contractor for an airline (rest assured: not the critical stuff that keeps planes in the air). There are some setups I would not have imagined to see, the two most surprising being Xcode builds and .Net. There were some common pain points I wanted to address with the next release. To be able to do that it was time to ditch files and move to a database.</p> <h2>Choosing The Right Data Store</h2> <p>It is 2015. There are more data stores than you can possible remember (and/or imagine). And all of them, of course, solve all your problems, scale infinitely, disprove CAP, laugh at ACID and solve world hunger. Thankfully I only need to store a bit of configuration, jobs and results. So basically all of them work, without challenging their crazy marketing statements.</p> <p>I would have preferred one that I can just integrate in Leeroy, like <a href="">bolt</a>, but all I found were either really complex to integrate, just simple key/value stores or had other shortcomings. So I took the easy way out: SQL - and to keep the dependencies as minimal as possible: SQLite.</p> <p>The biggest inconvenience is that I now have to use the target platforms compiler toolchain to build a binary. So easy cross-compilation from OSX to Linux is gone. But since I already run Leeroy on a Linux box I just added a build step when all tests pass.</p> <p>As for the ORM I decided to go with <a href="">gorm</a>. There will now be many devs reading this and think I am crazy for not using SQLx or using an "ORM" at all in Golang. But honestly? It works and it saves me time. And it is nicely encapsulated in model methods, if it ever gets in my way replacing it is stupidly simple.</p> <p>The only open question is if I want to support different databases than SQLite. Currently the Postgres drivers are part of the developer branch, but I am not sure I'll keep them around. The advantage would be that you can reuse the database you eventually already have and you do not have to backup another file to keep your CI statuses. I also got a rough idea how to use a database to coordinate a farm of Leeroys, so you run one instance on Windows, one on Linux, one on OSX and point them to the same DB and viola: your builds only pass when <em>all</em> Leeroys report that the code is good.</p> <h2>Interface</h2> <p>Beside a new UI, which is still WIP, there are some nice features that will hopefully make your life easier. The UI will undergo a redesign once my fiancee got some free time. Meanwhile it should still be an improvement. To add assets and easily modifiable templates while only shipping one binary without any archive to extract I added <a href="">rice</a>.</p> <h4>Search, Rerun and Cancel</h4> <p>You are now able to search for a branch or commit.</p> <p>You can rerun previous jobs. Have a race condition in your tests you did not fix yet but a failed build is blocking your deploy? Just rerun the tests, cross your fingers and question yourself for not fixing the tests.</p> <p>Got the same branch scheduled 5 times? Need to get a production build out and you do not want to wait for other tests to finish? Cancel everything that is in your way - as long as it is not running.</p> <p><a href="leeroy-new-webinterface.png"><img src="leeroy-new-webinterface.png" alt="new interface" /></a></p> <h4>Admin Interface</h4> <p>The biggest visible change is likely the admin interface. You can now configure commands, notifications, repositories through your browser. In 2015. Isn't it amazing?! Jokes aside: After introducing the database and user accounts this was the next logical step to make using Leeroy simpler.</p> <iframe src="" width="500" height="357" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe> <iframe src="" width="500" height="577" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe> <p>But we are still not there yet. Configuring a notification for example is still a bit ugly.</p> <p>A command still relies on a script stored on the CIs instance. One of the next things to do will be moving the build, test and deploy scripts to the database.</p> <p>But overall the admin interface should help everyone getting started with LeeroyCI while still not getting in the way if you configure the 10th repository.</p> <h2>Websockets</h2> <p>You can not connect via a websocket and get <em>all</em> events pushed to your client. At the same time I introduced the concept of access keys. I implemented a <a href="">POC</a> to show how it works in practice.</p> <iframe src="" width="500" height="413" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe> <p>It is a native OSX app, written in Swift, which shows a notification when an event happened. The next step will be filtering by the email address of the commit author. From there this could turn out to be quite usable.</p> <h2>Parallel builds</h2> <p>Leeroy can now also run builds in parallel. The default configuration is 1 build at a time, but it can easily be changed in the admin interface. This is a two edged sword. Running builds in parallel can be nice, but it also adds a bit complexity. If you are using Django e.x. you have to make sure your test script uses a different database for each test runner. To support this LeeroyCI now passes a third argument to the build script, a number indicating which internal task number the build got. In your build script you can now just change the environment variable for the database - you are using an environment variable and don't hardcode, right? - and you are good to go.</p> <p>Leeroy will not build the same branch in parallel. This is a limitation to not increase the complexity of the way deployments are handled. I am not sure if this will change in future, it most likely depends on the feedback I get.</p> <h2>Next Release</h2> <p>There are a few things I have to take care of before merging the current development branch into master. I want to improve the test coverage a bit and use websockets to refresh the browser to see when a build is done and hopefully add some UI improvements.</p> <p>In its current state the <a href="">development branch</a> should be stable. We are using it at FlightCar and I am using it privately, as well as 3 other companies who made the transition. I do not think you will see any surprises when you migrate, but if you prefer to only use production releases you have to wait for another week or two.</p> Wed, 14 Oct 2015 22:00:00 +0200 Upgrading Django Projects - Introduction <p>One questions I am asked quite often is how to upgrade larger projects to a newer Django release. While upgrading to a newer minor release is usually really easy and does not require much work it can become a bit harder upgrading to a new major release. My oldest project points back to 0.9x, so it has seen some upgrades, and there are certain patters that proved to work pretty well. <!--MORE--></p> <p>Upgrading a Django project is not harder or easier than upgrading any other project of the same size which is using a big third party framework as base and smaller libraries for certain parts of the system. I will try to start with general and simple principles on upgrading projects and move to more Python and Django specific topics later on.</p> <p>Before upgrading you should ask yourself if you have to. Let us assume your application is running fine and you are on a release that still gets bugfix and security updates. You could upgrade to get new features that eventually help you down the road. Or you could implement a new feature that separates your application from all competitors. The answer in that case is pretty obvious, isn’t it? </p> <p>You should keep in mind that the longer you wait with upgrading the more work it <em>can</em> become. There are three situations where you should <em>always</em> upgrade:</p> <ol> <li>No more bugfix and security patches for your Django version</li> <li>There is a new feature that will help you getting stuff done</li> <li>No more bugfix and security patches for your Django version</li> </ol> <p>My general advice would be: try to stay as close to the latest release as possible. Maybe not on the first day of the release, maybe not before the first minor release of a new major release. But try to stay up to date with the Django release cycle. I have to admit that personally I have a hard time breaking out of the old „never upgrade to a .0 release“ habit, but the Django dev team does an amazing job with releases and proved that this is eventually not the best strategy for a Django code base.</p> <h2>RemovedIn Warnings</h2> <p>Start to love them, they will save you a lot of pain. Django is telling you pretty early when a feature will be removed. If you take care of those warnings when they first show up, upgrading will be so much easier.</p> <p>Let us say you are currently running Django 1.7. Once in a while you should run your tests and server with the <code>-Wd</code> option to show silent warnings. If you happen to work on a part of the code where such a warning is raised - fix it. If you are not working on any code related to this warning just ignore it. It means the feature you are using will be removed in Django 1.9, so there is plenty of time.</p> <p>A different story are RemovedIn warnings which are shown when you do not use <code>-Wd</code>. This means the feature will be removed in Django 1.8. Fix the code immediately, no matter on which part of the system you are working.</p> <p>If you are not using any features that have been removed when upgrading you already took care of a major pain point. And fixing deprecation warnings when they first show up is usually pretty painless.</p> <h2>Deprecation Timeline</h2> <p>The same process as for RemovedIn warnings applies to the <a href="">deprecation timeline</a> you can find on the official website. If you happen to work on a part of the codebase that uses a feature that will be deprecated in current release + 2 then update your code if possible. If you see anything that will be deprecated in the next release always update your code, even if you are not directly working on this specific part of the system.</p> <p>Updating things that will be deprecated in the next release can become part of your upgrade process. If you are upgrading from Django 1.6 to 1.7 you could, as a part of the process, just update the code that is flagged to be deprecated in 1.8.</p> <h2>Forking Dependencies</h2> <p>Sooner or later the following will happen: Your application depends on a third party package to work with certain, usually the one you are using, Django version, but it is not updated to support the newest Django release yet or it is not actively maintained anymore. If you do not want to reimplement the whole package you always have the option to fork or vendor / bundle it. Do not be afraid of forking a package and bundling it with your source, nearly everyone will have to do this at one point, just get used to it.</p> <p>Usually this means just putting it in your project directory if you are lazy and you are done - this works because you can directly import packages from there, so you do not have to change any code. Putting it in a dedicated directory would be the better option though - most of the time this directory is called „vendor“ or „third_party“.</p> <p>From there on you can freely edit, change and update the package to make sure it is compatible with the latest version. Do not forget to remove it from your <code>requirements.txt</code>, no need to install it if you ship it with your codebase.</p> <p>When doing those changes, like fixing compatibility or updating something, please consider opening a pull request if the project is hosted on GitHub, ButBucket or another VCS hoster that supports pull requests or sending the maintainer of the package a patch. There are likely other people who need those updates, too. Relying on other people to do this is tricky, most of the time changes are made, forks are created, but nothing is contributed back to the original project with the result that not one or two developers but eventually hundreds waste their time changing exactly the same two lines of code.</p> <p>From time to time a package is not maintained anymore or the maintainer refuses to fix obvious bugs or does not even acknowledge them as such. If this happens it is sometimes easier to just create a fork you will maintain yourself or make it part of the your project. If this really makes sense is hard to tell - depending on the size of the package you put a lot of work on yourself, eventually forcing yourself to maintain a package with many bugs you did not notice yet. Sometimes it is easier and in the longterm the better solution to just search for an alternative package. It is rare that there is only one package providing a certain functionality.</p> <h2>Release Notes</h2> <p>Once you decided to upgrade, took care of the dependencies and warnings, ran <code>pip install -U django</code> and made sure your code is working it is time to read release notes. Too often have I seen complex, weird, scary (or all three together) code that kind of tries to solve what Django already ships as a working, well tested util, function or method. New stuff is added constantly, try to stay current with your knowledge about the different parts of the framework, especially the view layer and the ORM.</p> <p>Sometimes it is something trivial like being able to give migrations a name - making it easier to identify them when browsing the files - but in the long run things and stuff accumulate and provide real value.</p> <h2>Conclusion</h2> <p>While this is a pretty high level overview of the whole process it should answers the, in my opinion, most important questions I was asked. It is by no means a complete or Django specific guide - I am planning to go into more details in future posts walking through some more complex scenarios I encountered.</p> <p>The most important advice I can give you is that you should always upgrade when there are no more security and bugfix releases for your Django version. Maybe it is scary if you are doing it for the first time. Maybe you will mess up a little bit. Maybe it will take some time. But that should never be the reason to stay on an outdated version, eventually jeopardizing your or your customers data.</p> Sat, 27 Jun 2015 19:25:00 +0200 Never trust an environment you do not fully control <p>This is usually a suggestion you hear when it comes to sensitive information, personal data and security related tasks. I learned this again over Christmas while working on a small side project and while giving a service a friend of mine is working on a try. <!--MORE--></p> <p>Before we start let me warn you: Do not take this as a guide on how to troubleshoot a problem. It is not a good idea and you never should do it this way if you are working on a serious project. What I did here was a mix of laziness, not being at home for 4 days and we are talking about Christmas, so I was running from one family event to the next one and time to troubleshoot unimportant stuff was pretty rare - I had 2 or 3 tries a day without scarifying any family time, but I really wanted to know if it works.</p> <p>The service I was trying sounded pretty nice - a kind of Heroku alternative, based on what you would expect (docker etc.), but with the difference that you bring your own API keys for a cloud provider and all servers etc. belong to you - they just provision them and charge you some $ a month for the service which means you get a lot saner pricing. I was told <em>everything</em> is documented and there would never be any wired behavior since it is just one process running and not doing much.</p> <p>I hope when she reads this she will not be too mad because I likely messed up her pitch and made it sound a lot lamer and useless than it does when she is pitching you. Since she does not even have a name or anything I also cannot link you to the obligatory big-header-no-content marketing site. I will let you know when there is one.</p> <p>She set me up with a test environment and I requested a 512MB Digital Ocean box for "workers". Everything was pretty painless, pushing to the services git endpoint worked and some minutes later I got a "deployment succeeded" notification. The wired stuff began when my management command ran out of memory.</p> <p>The management command is quite trivial. It is iterating over all keys in an S3 bucket, checks if one of them changed and if this is true downloads the content, unzips and stores the content in the database.</p> <p>The script takes quite a bit of time for the full production dataset (roughly 26 hours), so I only used 2% of the dataset as test data. The code is fully test covered and used real data for tests, so I didn't expect too many stupid errors to make it to the test server. One of the reckless assumptions that turned out to be true - lucky shot :)</p> <p>This is the first time I am using <a href="">boto</a> with lots of keys and in memory fetching and unzipping, so I made a quick list with my first thoughts what could possibly be wrong</p> <ul> <li>a file that is broken / different / too big</li> <li>me messing up the in memory fetching and unzipping</li> </ul> <p>My first attempt was to run the script a few times and look at the key on which it is failing. It was always a different one and the one that failed on the second run passed on the first one.</p> <p>Next I just tried running it from top to bottom always doing only one thing at a time. The first test was to iterate over all keys and print the name and etag, which worked fine and was even pretty fast compared to the expected runtime.</p> <p>Next I fetched all keys and checked if they changed - and the script was killed, again. So my second theory was not even worth testing, something broke way earlier.</p> <pre><code> def logfile_changed(self, name, etag): logfile, created = LogFile.objects.get_or_create(, name=name, ) if created is True: logfile.hashed = etag return True if logfile.hashed != etag: logfile.hashed = etag return True return False </code></pre> <p>Quickly running it wit everything after <code>get_or_create</code> commented out showed that it is still crashing. To get boto out of the picture I replace it with a for loop</p> <pre><code> for x in range(1,9999999999): self.logfile_changed(self, x, x) </code></pre> <p>Guess what - out of memory. So we have a for loop which just runs <code>get_or_create</code> that runs out of memory. Anyone using Django for more than the duration of a bootcamp will likely scream "set debug to False you idiot". But guess what - debug was set to False. At least I thought it was.</p> <p>But from here it was pretty obvious - it has to be the debug flag. Local testing didn't uncover any problems, my server at home completed the import of the full production dataset without going above 150MB for the Python process and suspecting that Django is broken is nearly an as good guess as thinking of a compiler bug.</p> <p>After a short mail describing the problem it turned out that no matter what environment variables I set, the system just refuse to overwrite the ones set by the service and somehow everything ends up in one big pile of environment variable mess. Please do not ask me how they do it, I was too scared to ask. In this case they use the presence of <code>DEBUG</code>, too, to activate the debug mode. Which actually explains why the script was running out of memory. Hardcoding <code>DEBUG=False</code> in <code></code> fixed it. It was a bit troublesome to figure out what was happening, but at least she got a real world example why it is a bad idea to toy with the customers environment without documenting it or showing warnings.</p> <p>The lesson is the same as always: If you do not have full control over an environment - in this case it was not even possible to get a shell or anything like that - you are on the best way to a world of pain when you run into a problem. Even if you think you know and understand the problem, you will eventually not be able to troubleshoot or fix it by yourself. While this bug was relatively easy, there are harder ones out there you can run into. But hey, obscure bugs are the funny ones :)</p> Tue, 30 Dec 2014 20:01:00 +0200 What Start-Ups Can Learn From Blizzard Entertainment <p>I spent a good part of my weekend playing Blizzards Heroes of the Storm Alpha. It is a great game, I really enjoy it and it proves, once again, that Blizzard is really good in bringing new concepts, improvements and innovation to an already existing genre. And they are successful with it. So what can a start-up learn from Blizzard? <!--MORE--></p> <p>If you are a fan of a game I am not talking nicely about please just ignore it and do not fall into a nerd-rage. There is certainly a huge discrepancy how people see certain games and I can only talk about my experience with them. You may certainly disagree and that is totally fine, but this will not change how I and many others, too, experienced those games. So please just ignore the things that make you mad and concentrate on the actual idea behind the article. :)</p> <p>First of all let me make a bold statement: Blizzard is so incredibly successful because they are managing to take a genre and make it fun for everyone, not just people who enjoy it so much that they have the urge to become top players. They do not invent, they improve.</p> <p>Ultima Online was nice. It certainly destroyed grades in school, relationships and many lives, but hey, it was a nice game. And then came World of Warcraft. You could just create an account and play. You run around a bit, do some quests, maybe party with other players. You were not as free as in Ultima Online, you could not learn every profession possible and many things just looked easy, but in exchange you got a lot better story. And people liked it. Not being forced to learn <em>everything</em> about a character and professions for example allowed you to focus more on the game and the gameplay. Compared to today Classic World of Warcraft was garbage, but still a lot better than all the competitors.</p> <p>Real-time strategy games existed before Blizzard released Warcraft: Orcs vs. Humans and Dune certainly helped the genre gaining traction, but Warcraft and later on StarCraft were hit titles. They got most of the pathfinding right, which is extremely valuable when it comes to a fun gameplay, the characters, the story - they just did an amazing job to pull the players into their world (as with WoW). And players stayed. There were surely other successful RTS games, but as far as I know none that came close to the fan base of Warcraft and Starcraft. Maybe Command and Conquer had a good shot.</p> <p>Fast forward - Hearthstone. A trading card game you play on your computer or iPad. I played Magic, I was skeptical and I was right, it is not for me. Too trivial, no depth and not many ways that actual skill determines if you win a match or not. But I have seen my significant other starting to play a genre she never cared about, just because it looked fun, was easy to get in and provided enough longterm motivation to stay and come back when expansions are released. Hearthstone is the most watered down, trivial kind of trading card game I am aware of and at the same time the one with one of the biggest player bases.</p> <p>I predict that the same thing will happen with Heroes of the Storm. It is slower, you have more abilities from the beginning and the gameplay is easier than in Leagues of Legend and DOTA2. But is is fun. You do not have such a big penalty when you mess up a little and have to fall back to your base. You can actually do something the first few levels but pressing one button to harass your opponent and farm. The small side games you play are funny and add to the gameplay and you do not have to consult guides and read all your abilities twice to find a good combination of hero skills, summoner skills and build orders. Confused by all the different things I just mentioned? Many people who do not know the genre would be just as confused as you are. Know what? You do not have to care about all this stuff when playing HotS. Just play. The tutorial will explain everything you need to know and you will be fine after some matches.</p> <h2>Execution Is Everything</h2> <p>There were other phones and other smartphones before the iPhone. There were other car manufacturers before Mercedes and BMW. There were other publishers in the game genres I just talked about. Facebook and Twitter were not the first social networks. And still, companies who were not the first defined the field.</p> <p>I often hear people complaining that an idea, a start-up or a project sucks because someone else did it already. You know what? Nearly every publisher would sell their whole executive team including families to some ancient demon only to get half of the player base Blizzard got. It does not matter if the idea is new. It does not matter if you are late to the game, if others did it and failed, the only thing that matters is your execution. And a certain amount of luck, but that is always true, even if you are the first one.</p> <p>So instead of trying to come up with the best of all ideas to disrupt a market you will first have to define since it does not exist - just do something people really want, but done right. Your chances are far better to succeed and build a successful business with a great product people enjoy than with a product no one knew they needed.</p> <p>I am not saying you should not follow you not-existing-market-disrupting idea. If it is a good idea and you are the first one to execute it right you just win. But if someone asks you what you think of an idea and you hear one that was done so many times - do not shoot it down just because it is not new. It does not need to be new to make it to the top of the market segment it is in.</p> <p>You can discuss this post on <a href="">HackerNews</a>.</p> Wed, 10 Dec 2014 21:40:00 +0200 LeeroyCI - taking down production with automatic deploys since last week <p>Let me give you a quick update about a new feature that made it to <a href="">Leeroys master branch</a>. Deployment. You can now specify a script that will run after a build for a branch was successful. Initially I planned to focus on deployment when working on version 2 or a late 1.x release - for me this is a scary feature and I would have preferred to focus on testing right now. But at FlightCar we decided we want to start leveraging continuous deployment and so I began to work it. <!--MORE--></p> <p>Why did I say this is a scary feature? Leeroy is not only used by me. Some people already trust it with running tests for their start-up or hobby projects. Hopefully more people will start using Leeroy and some of them eventually decide to let Leeroy automatically deploy branches. Deployment is working as everything else, by calling an external script or program. So, in the end, if a deploy is not working, eventually taking down production, you could blame the person who wrote the test or deployment script. In my opinion it is not that easy.</p> <p>I see it as my responsibility to make deploying through Leeroy as secure and convenient as possible. There will be a grace period of likely 30 seconds in which you can cancel a deploy by just clicking on a link. It will involve a lot notifications, in its first incarnation eventually too many. But at the end of the day I want to be sure this feature will, once we hit 1.0, be so rock solid that if a deploy is ever failing or messing up something everyone can easily agree it was not Leeroys fault. </p> <p>The trick will be to keep things simple. One of the reasons why I started working on Leeroy was that the existing CIs are either complicated or impotent. I want to keep Leeroys simplicity even while offering features like automatically deploying your latest changes. I am satisfied with its current state - or it would not be in master. It is deploying two Django based environments under my watch right now, I use it to build Leeroy branches and upload the binaries to S3 and I am about to configure it for drupan this weekend to publish changes in master to pypi.</p> <p>Okay, that is enough whining. Give it a try, the <a href="">configuration</a> is simple and if you are on Heroku or Elastic Beanstalk a deployment script can look as trivial as this</p> <pre><code>#! /bin/bash cd /home/ec2-user/test-deploy git fetch get reset --hard origin/master git aws.push </code></pre> <p>So not much magic involved here. Do not worry about deploying broken code, it will only deploy if all tests pass.</p> <h3>Preview</h3> <p>Some things you will see happening shortly:</p> <ul> <li>custom templates (this is already in testing)</li> <li>simplified notification system (eventually with custom notifications)</li> <li>more notifications (deployment announcements e.x.)</li> </ul> <p>And after I got those done the next two bigs steps are</p> <ul> <li>admin interface (including user support)</li> <li>notifications about successful deploys</li> </ul> <p>This is what I am planning to get done this year. Ignoring the GitLab and BitBucket integration for a moment (this does not mean it will not be done), the <a href="">roadmap for 1.0</a> looks pretty good and suggests a 1.0 beta between Christmas and New Year.</p> Thu, 30 Oct 2014 20:13:00 +0200