screaming at my screen updates for screaming at my screen - the private domain of Timo Zimmermann en-en Timo Zimmermann Sun, 28 Jun 2015 10:04:01 +0200 Upgrading Django Projects - Introduction <p>One questions I am asked quite often is how to upgrade larger projects to a newer Django release. While upgrading to a newer minor release is usually really easy and does not require much work it can become a bit harder upgrading to a new major release. My oldest project points back to 0.9x, so it has seen some upgrades, and there are certain patters that proved to work pretty well. <!--MORE--></p> <p>Upgrading a Django project is not harder or easier than upgrading any other project of the same size which is using a big third party framework as base and smaller libraries for certain parts of the system. I will try to start with general and simple principles on upgrading projects and move to more Python and Django specific topics later on.</p> <p>Before upgrading you should ask yourself if you have to. Let us assume your application is running fine and you are on a release that still gets bugfix and security updates. You could upgrade to get new features that eventually help you down the road. Or you could implement a new feature that separates your application from all competitors. The answer in that case is pretty obvious, isn’t it? </p> <p>You should keep in mind that the longer you wait with upgrading the more work it <em>can</em> become. There are three situations where you should <em>always</em> upgrade:</p> <ol> <li>No more bugfix and security patches for your Django version</li> <li>There is a new feature that will help you getting stuff done</li> <li>No more bugfix and security patches for your Django version</li> </ol> <p>My general advice would be: try to stay as close to the latest release as possible. Maybe not on the first day of the release, maybe not before the first minor release of a new major release. But try to stay up to date with the Django release cycle. I have to admit that personally I have a hard time breaking out of the old „never upgrade to a .0 release“ habit, but the Django dev team does an amazing job with releases and proved that this is eventually not the best strategy for a Django code base.</p> <h2>RemovedIn Warnings</h2> <p>Start to love them, they will save you a lot of pain. Django is telling you pretty early when a feature will be removed. If you take care of those warnings when they first show up, upgrading will be so much easier.</p> <p>Let us say you are currently running Django 1.7. Once in a while you should run your tests and server with the <code>-Wd</code> option to show silent warnings. If you happen to work on a part of the code where such a warning is raised - fix it. If you are not working on any code related to this warning just ignore it. It means the feature you are using will be removed in Django 1.9, so there is plenty of time.</p> <p>A different story are RemovedIn warnings which are shown when you do not use <code>-Wd</code>. This means the feature will be removed in Django 1.8. Fix the code immediately, no matter on which part of the system you are working.</p> <p>If you are not using any features that have been removed when upgrading you already took care of a major pain point. And fixing deprecation warnings when they first show up is usually pretty painless.</p> <h2>Deprecation Timeline</h2> <p>The same process as for RemovedIn warnings applies to the <a href="">deprecation timeline</a> you can find on the official website. If you happen to work on a part of the codebase that uses a feature that will be deprecated in current release + 2 then update your code if possible. If you see anything that will be deprecated in the next release always update your code, even if you are not directly working on this specific part of the system.</p> <p>Updating things that will be deprecated in the next release can become part of your upgrade process. If you are upgrading from Django 1.6 to 1.7 you could, as a part of the process, just update the code that is flagged to be deprecated in 1.8.</p> <h2>Forking Dependencies</h2> <p>Sooner or later the following will happen: Your application depends on a third party package to work with certain, usually the one you are using, Django version, but it is not updated to support the newest Django release yet or it is not actively maintained anymore. If you do not want to reimplement the whole package you always have the option to fork or vendor / bundle it. Do not be afraid of forking a package and bundling it with your source, nearly everyone will have to do this at one point, just get used to it.</p> <p>Usually this means just putting it in your project directory if you are lazy and you are done - this works because you can directly import packages from there, so you do not have to change any code. Putting it in a dedicated directory would be the better option though - most of the time this directory is called „vendor“ or „third_party“.</p> <p>From there on you can freely edit, change and update the package to make sure it is compatible with the latest version. Do not forget to remove it from your <code>requirements.txt</code>, no need to install it if you ship it with your codebase.</p> <p>When doing those changes, like fixing compatibility or updating something, please consider opening a pull request if the project is hosted on GitHub, ButBucket or another VCS hoster that supports pull requests or sending the maintainer of the package a patch. There are likely other people who need those updates, too. Relying on other people to do this is tricky, most of the time changes are made, forks are created, but nothing is contributed back to the original project with the result that not one or two developers but eventually hundreds waste their time changing exactly the same two lines of code.</p> <p>From time to time a package is not maintained anymore or the maintainer refuses to fix obvious bugs or does not even acknowledge them as such. If this happens it is sometimes easier to just create a fork you will maintain yourself or make it part of the your project. If this really makes sense is hard to tell - depending on the size of the package you put a lot of work on yourself, eventually forcing yourself to maintain a package with many bugs you did not notice yet. Sometimes it is easier and in the longterm the better solution to just search for an alternative package. It is rare that there is only one package providing a certain functionality.</p> <h2>Release Notes</h2> <p>Once you decided to upgrade, took care of the dependencies and warnings, ran <code>pip install -U django</code> and made sure your code is working it is time to read release notes. Too often have I seen complex, weird, scary (or all three together) code that kind of tries to solve what Django already ships as a working, well tested util, function or method. New stuff is added constantly, try to stay current with your knowledge about the different parts of the framework, especially the view layer and the ORM.</p> <p>Sometimes it is something trivial like being able to give migrations a name - making it easier to identify them when browsing the files - but in the long run things and stuff accumulate and provide real value.</p> <h2>Conclusion</h2> <p>While this is a pretty high level overview of the whole process it should answers the, in my opinion, most important questions I was asked. It is by no means a complete or Django specific guide - I am planning to go into more details in future posts walking through some more complex scenarios I encountered.</p> <p>The most important advice I can give you is that you should always upgrade when there are no more security and bugfix releases for your Django version. Maybe it is scary if you are doing it for the first time. Maybe you will mess up a little bit. Maybe it will take some time. But that should never be the reason to stay on an outdated version, eventually jeopardizing your or your customers data.</p> Sat, 27 Jun 2015 19:25:00 +0200 Never trust an environment you do not fully control <p>This is usually a suggestion you hear when it comes to sensitive information, personal data and security related tasks. I learned this again over Christmas while working on a small side project and while giving a service a friend of mine is working on a try. <!--MORE--></p> <p>Before we start let me warn you: Do not take this as a guide on how to troubleshoot a problem. It is not a good idea and you never should do it this way if you are working on a serious project. What I did here was a mix of laziness, not being at home for 4 days and we are talking about Christmas, so I was running from one family event to the next one and time to troubleshoot unimportant stuff was pretty rare - I had 2 or 3 tries a day without scarifying any family time, but I really wanted to know if it works.</p> <p>The service I was trying sounded pretty nice - a kind of Heroku alternative, based on what you would expect (docker etc.), but with the difference that you bring your own API keys for a cloud provider and all servers etc. belong to you - they just provision them and charge you some $ a month for the service which means you get a lot saner pricing. I was told <em>everything</em> is documented and there would never be any wired behavior since it is just one process running and not doing much.</p> <p>I hope when she reads this she will not be too mad because I likely messed up her pitch and made it sound a lot lamer and useless than it does when she is pitching you. Since she does not even have a name or anything I also cannot link you to the obligatory big-header-no-content marketing site. I will let you know when there is one.</p> <p>She set me up with a test environment and I requested a 512MB Digital Ocean box for "workers". Everything was pretty painless, pushing to the services git endpoint worked and some minutes later I got a "deployment succeeded" notification. The wired stuff began when my management command ran out of memory.</p> <p>The management command is quite trivial. It is iterating over all keys in an S3 bucket, checks if one of them changed and if this is true downloads the content, unzips and stores the content in the database.</p> <p>The script takes quite a bit of time for the full production dataset (roughly 26 hours), so I only used 2% of the dataset as test data. The code is fully test covered and used real data for tests, so I didn't expect too many stupid errors to make it to the test server. One of the reckless assumptions that turned out to be true - lucky shot :)</p> <p>This is the first time I am using <a href="">boto</a> with lots of keys and in memory fetching and unzipping, so I made a quick list with my first thoughts what could possibly be wrong</p> <ul> <li>a file that is broken / different / too big</li> <li>me messing up the in memory fetching and unzipping</li> </ul> <p>My first attempt was to run the script a few times and look at the key on which it is failing. It was always a different one and the one that failed on the second run passed on the first one.</p> <p>Next I just tried running it from top to bottom always doing only one thing at a time. The first test was to iterate over all keys and print the name and etag, which worked fine and was even pretty fast compared to the expected runtime.</p> <p>Next I fetched all keys and checked if they changed - and the script was killed, again. So my second theory was not even worth testing, something broke way earlier.</p> <pre><code> def logfile_changed(self, name, etag): logfile, created = LogFile.objects.get_or_create(, name=name, ) if created is True: logfile.hashed = etag return True if logfile.hashed != etag: logfile.hashed = etag return True return False </code></pre> <p>Quickly running it wit everything after <code>get_or_create</code> commented out showed that it is still crashing. To get boto out of the picture I replace it with a for loop</p> <pre><code> for x in range(1,9999999999): self.logfile_changed(self, x, x) </code></pre> <p>Guess what - out of memory. So we have a for loop which just runs <code>get_or_create</code> that runs out of memory. Anyone using Django for more than the duration of a bootcamp will likely scream "set debug to False you idiot". But guess what - debug was set to False. At least I thought it was.</p> <p>But from here it was pretty obvious - it has to be the debug flag. Local testing didn't uncover any problems, my server at home completed the import of the full production dataset without going above 150MB for the Python process and suspecting that Django is broken is nearly an as good guess as thinking of a compiler bug.</p> <p>After a short mail describing the problem it turned out that no matter what environment variables I set, the system just refuse to overwrite the ones set by the service and somehow everything ends up in one big pile of environment variable mess. Please do not ask me how they do it, I was too scared to ask. In this case they use the presence of <code>DEBUG</code>, too, to activate the debug mode. Which actually explains why the script was running out of memory. Hardcoding <code>DEBUG=False</code> in <code></code> fixed it. It was a bit troublesome to figure out what was happening, but at least she got a real world example why it is a bad idea to toy with the customers environment without documenting it or showing warnings.</p> <p>The lesson is the same as always: If you do not have full control over an environment - in this case it was not even possible to get a shell or anything like that - you are on the best way to a world of pain when you run into a problem. Even if you think you know and understand the problem, you will eventually not be able to troubleshoot or fix it by yourself. While this bug was relatively easy, there are harder ones out there you can run into. But hey, obscure bugs are the funny ones :)</p> Tue, 30 Dec 2014 20:01:00 +0200 What Start-Ups Can Learn From Blizzard Entertainment <p>I spent a good part of my weekend playing Blizzards Heroes of the Storm Alpha. It is a great game, I really enjoy it and it proves, once again, that Blizzard is really good in bringing new concepts, improvements and innovation to an already existing genre. And they are successful with it. So what can a start-up learn from Blizzard? <!--MORE--></p> <p>If you are a fan of a game I am not talking nicely about please just ignore it and do not fall into a nerd-rage. There is certainly a huge discrepancy how people see certain games and I can only talk about my experience with them. You may certainly disagree and that is totally fine, but this will not change how I and many others, too, experienced those games. So please just ignore the things that make you mad and concentrate on the actual idea behind the article. :)</p> <p>First of all let me make a bold statement: Blizzard is so incredibly successful because they are managing to take a genre and make it fun for everyone, not just people who enjoy it so much that they have the urge to become top players. They do not invent, they improve.</p> <p>Ultima Online was nice. It certainly destroyed grades in school, relationships and many lives, but hey, it was a nice game. And then came World of Warcraft. You could just create an account and play. You run around a bit, do some quests, maybe party with other players. You were not as free as in Ultima Online, you could not learn every profession possible and many things just looked easy, but in exchange you got a lot better story. And people liked it. Not being forced to learn <em>everything</em> about a character and professions for example allowed you to focus more on the game and the gameplay. Compared to today Classic World of Warcraft was garbage, but still a lot better than all the competitors.</p> <p>Real-time strategy games existed before Blizzard released Warcraft: Orcs vs. Humans and Dune certainly helped the genre gaining traction, but Warcraft and later on StarCraft were hit titles. They got most of the pathfinding right, which is extremely valuable when it comes to a fun gameplay, the characters, the story - they just did an amazing job to pull the players into their world (as with WoW). And players stayed. There were surely other successful RTS games, but as far as I know none that came close to the fan base of Warcraft and Starcraft. Maybe Command and Conquer had a good shot.</p> <p>Fast forward - Hearthstone. A trading card game you play on your computer or iPad. I played Magic, I was skeptical and I was right, it is not for me. Too trivial, no depth and not many ways that actual skill determines if you win a match or not. But I have seen my significant other starting to play a genre she never cared about, just because it looked fun, was easy to get in and provided enough longterm motivation to stay and come back when expansions are released. Hearthstone is the most watered down, trivial kind of trading card game I am aware of and at the same time the one with one of the biggest player bases.</p> <p>I predict that the same thing will happen with Heroes of the Storm. It is slower, you have more abilities from the beginning and the gameplay is easier than in Leagues of Legend and DOTA2. But is is fun. You do not have such a big penalty when you mess up a little and have to fall back to your base. You can actually do something the first few levels but pressing one button to harass your opponent and farm. The small side games you play are funny and add to the gameplay and you do not have to consult guides and read all your abilities twice to find a good combination of hero skills, summoner skills and build orders. Confused by all the different things I just mentioned? Many people who do not know the genre would be just as confused as you are. Know what? You do not have to care about all this stuff when playing HotS. Just play. The tutorial will explain everything you need to know and you will be fine after some matches.</p> <h2>Execution Is Everything</h2> <p>There were other phones and other smartphones before the iPhone. There were other car manufacturers before Mercedes and BMW. There were other publishers in the game genres I just talked about. Facebook and Twitter were not the first social networks. And still, companies who were not the first defined the field.</p> <p>I often hear people complaining that an idea, a start-up or a project sucks because someone else did it already. You know what? Nearly every publisher would sell their whole executive team including families to some ancient demon only to get half of the player base Blizzard got. It does not matter if the idea is new. It does not matter if you are late to the game, if others did it and failed, the only thing that matters is your execution. And a certain amount of luck, but that is always true, even if you are the first one.</p> <p>So instead of trying to come up with the best of all ideas to disrupt a market you will first have to define since it does not exist - just do something people really want, but done right. Your chances are far better to succeed and build a successful business with a great product people enjoy than with a product no one knew they needed.</p> <p>I am not saying you should not follow you not-existing-market-disrupting idea. If it is a good idea and you are the first one to execute it right you just win. But if someone asks you what you think of an idea and you hear one that was done so many times - do not shoot it down just because it is not new. It does not need to be new to make it to the top of the market segment it is in.</p> <p>You can discuss this post on <a href="">HackerNews</a>.</p> Wed, 10 Dec 2014 21:40:00 +0200 LeeroyCI - taking down production with automatic deploys since last week <p>Let me give you a quick update about a new feature that made it to <a href="">Leeroys master branch</a>. Deployment. You can now specify a script that will run after a build for a branch was successful. Initially I planned to focus on deployment when working on version 2 or a late 1.x release - for me this is a scary feature and I would have preferred to focus on testing right now. But at FlightCar we decided we want to start leveraging continuous deployment and so I began to work it. <!--MORE--></p> <p>Why did I say this is a scary feature? Leeroy is not only used by me. Some people already trust it with running tests for their start-up or hobby projects. Hopefully more people will start using Leeroy and some of them eventually decide to let Leeroy automatically deploy branches. Deployment is working as everything else, by calling an external script or program. So, in the end, if a deploy is not working, eventually taking down production, you could blame the person who wrote the test or deployment script. In my opinion it is not that easy.</p> <p>I see it as my responsibility to make deploying through Leeroy as secure and convenient as possible. There will be a grace period of likely 30 seconds in which you can cancel a deploy by just clicking on a link. It will involve a lot notifications, in its first incarnation eventually too many. But at the end of the day I want to be sure this feature will, once we hit 1.0, be so rock solid that if a deploy is ever failing or messing up something everyone can easily agree it was not Leeroys fault. </p> <p>The trick will be to keep things simple. One of the reasons why I started working on Leeroy was that the existing CIs are either complicated or impotent. I want to keep Leeroys simplicity even while offering features like automatically deploying your latest changes. I am satisfied with its current state - or it would not be in master. It is deploying two Django based environments under my watch right now, I use it to build Leeroy branches and upload the binaries to S3 and I am about to configure it for drupan this weekend to publish changes in master to pypi.</p> <p>Okay, that is enough whining. Give it a try, the <a href="">configuration</a> is simple and if you are on Heroku or Elastic Beanstalk a deployment script can look as trivial as this</p> <pre><code>#! /bin/bash cd /home/ec2-user/test-deploy git fetch get reset --hard origin/master git aws.push </code></pre> <p>So not much magic involved here. Do not worry about deploying broken code, it will only deploy if all tests pass.</p> <h3>Preview</h3> <p>Some things you will see happening shortly:</p> <ul> <li>custom templates (this is already in testing)</li> <li>simplified notification system (eventually with custom notifications)</li> <li>more notifications (deployment announcements e.x.)</li> </ul> <p>And after I got those done the next two bigs steps are</p> <ul> <li>admin interface (including user support)</li> <li>notifications about successful deploys</li> </ul> <p>This is what I am planning to get done this year. Ignoring the GitLab and BitBucket integration for a moment (this does not mean it will not be done), the <a href="">roadmap for 1.0</a> looks pretty good and suggests a 1.0 beta between Christmas and New Year.</p> Thu, 30 Oct 2014 20:13:00 +0200 Why I am writing a web interface for a static site generator <p>Let us talk about drunken pandas, sake and why I am trying to tie a static site generator to a SQL database and a Django process. There are actually two simple reasons: The first one is that I want more comfort while blogging without losing the advantages of drupan. The second one is that I hope to enable more people to use a static site generator and own their content without losing any comfort or having to put up with managing a small, likely virtual, server. <!--MORE--></p> <p>While <a href="">drupan hit 2.0</a> some weeks ago and is doing pretty well I decided to take on a project <a href="">I already talked about</a>. Making it easier to create content, bringing features of full blown content management systems to the world of static site generators and still keeping the whole thing simple enough that lots of people will be able to set it up and use it. As I already mentioned one of the reasons is that I want more comfort, but let us focus on the other one for now.</p> <p>There is certainly a trend to use a service that provides you a hosted blog, no matter if Blogger, Svbtle or Medium, according to rumors there is even one person left publishing stuff on LiveJournal. But in my opinion you give up too much just for the convenience of not having to take care of a bit software. It varies from service to service, sometimes you do not own your content, sometimes you have to live with the fact that it could be required at one point to publish things under your real name, no matter what the consequences would be, sometimes the service tries everything so the content you create is not associated with you but the service, sometimes the service is just a mess and the downtime is higher than the uptime. There are so many reasons why hosting your own content is a good idea. But why do more and more people favor hosted services?</p> <p>One word: Convenience. You enter your email and a password and start typing. That is it. If it would be a bit more trivial someone would create and send you your credentials the moment you are born. A self hosted solution will never be able to provide this level of convenience. But there are certainly ways to get closer to it than what we have right now.</p> <h3>Uptime, performance and security</h3> <p>Those three points are likely the three biggest offenders when it comes to convenience. How many blogs did we see going down because there was an article making it to the front page of HackerNews or reddit? How often do you have to update your system and your blogging software to have a vague chance of keeping everything secure? How often do cheap, and sadly also sometimes expensive, VPS providers fail when it comes to reliability?</p> <p>I can understand anyone who does not want to administrate yet another VPS. There are only two options to get around this: use a hosted blogging service or a managed server or platform. The first and most important goal for drupan and Sakebowl is to solve those three problems. Let us take a look at the idea.</p> <p>Drupan deploys your site to AWS S3 + CloudFront. CloudFront and S3 are doing an awesome job keeping static files online. And I believe not many people are willing to try to achieve the same level of reliability than those services offer.</p> <p>You can run Sakebowl wherever you want. On your laptop? No problem! One click deployment to Heroku using the Heroku button? Sure. On the server in your basement or a VPS? Despite trying to solve the problem of having to take care of a VPS it is certainly possible and painless. With Sakebowl you get an intuitive web interface, your data stored in a database and with one click you generate and deploy your site using drupan. Sakebowl does not have to be online or reachable by the whole world to do its job.</p> <h3>How far did I get and what will Sakebowl look like?</h3> <p>At least that is the idea. I started working on Sakebowl, making sure it can import existing drupan sites and generate and deploy them. It is hard to start with the minimal necessary functionality and keep a certain target audience in mind - websites and blogs - since drupan is designed to get out of our way no matter what you create with it. But it is currently focusing on exactly this, websites and blogs. Features like multiple users with roles are something I already thought about, but they will have to wait. I always appreciate feature requests, but anything beyond the basics would end in many blog posts and no results.</p> <p>The primary focus for the beginning will be deployment and setup. The setup process should look like this</p> <ul> <li>click on Heroku button</li> <li>enter AWS credentials</li> <li>answer some simple questions (site name etc.)</li> <li>write your first blog post</li> </ul> <p>The setup will be 100% automatic, you will not have to configure the whole AWS stack yourself. Sure, it involves a Heroku and AWS account. This is not targeting end users who have trouble singing up for both services. The target audience is the technical crowd who is fine with following an easy tutorial that involves ~ 20 steps to get their blog online, knowing that they do not have to put up with the usual problems that occur <em>after</em> the setup.</p> <p>I want to use Sakebowl, so it will happen. Eventually it will also help other people to own their content without the usual troubles, at least I hope so. If you have any ideas how the process should look like, what features it will have to ship to be usable or if you want to discuss that I should stop wasting your time with blog posts about projects I am working on that sound like sales pitches - yeah, after reading it I am aware of that - just mail me, I'd love to hear your thoughts.</p> Mon, 06 Oct 2014 20:00:00 +0200