http://ericholscher.comEric Holscher - Posted in 20102024-02-28T16:22:28.602297+00:00ABloghttp://ericholscher.com/blog/2010/nov/4/running-hudson-matrix-builds-multiple-machines/Running Hudson matrix builds on multiple machines2010-11-04T16:00:00+00:00<section id="running-hudson-matrix-builds-on-multiple-machines">
<p>When I was setting up the Django Hudson instance, I ran into a
problem that seems like it should be pretty easy to solve. However,
I couldn’t figure out a way. So at this point it’s looking like
we’re going to have to use buildbot to build out what we want
instead of Hudson. Wondering if I missed something obvious, or if
this is a missing feature.</p>
<section id="our-setup">
<h2>Our setup</h2>
<p>We have our builds segmented currently by Django version and
database. So to get something up and going, we have a Django trunk
and 1.2.X build going against sqlite. We use a matrix build to run
this against python 2.4-2.6, which means 3 builds in total for each
Django version.</p>
<p>However,
<strong>I can’t find a way to make the matrix build choose the slave to run on based on what version of python it supports</strong>.
I want to be able to randomly add slaves to the Hudson
configuration, and have them pick up builds based off of their
capabilities.</p>
</section>
<section id="the-problem">
<h2>The problem</h2>
<p>Hudson supports the idea of tagging, so we came up with a
<a class="reference external" href="http://code.djangoproject.com/wiki/BuildFarm#Desiredconfigs">tagging scheme</a>
for the slaves. So it seems that we should be able to tell a test
to run on any slave tagged with the versions of python we want.
Hudson also supports this, but it runs all the tests across all the
different slaves each time. I need it to
<strong>have a pool of workers capable of running a set of tests, but run each test on only one of the members of the pool</strong>.</p>
<p>I was wondering if we’re doing it wrong, or if other people know
the correct way to run tests across a set of slaves, based on the
capabilities of the slave? This seems like a pretty basic
requirement for people running any kind of sizable Hudson
configuration.</p>
</section>
</section>
When I was setting up the Django Hudson instance, I ran into a
problem that seems like it should be pretty easy to solve. However,
I couldn’t figure out a way. So at this point it’s looking like
we’re going to have to use buildbot to build out what we want
instead of Hudson. Wondering if I missed something obvious, or if
this is a missing feature.2010-11-04T16:00:00+00:00http://ericholscher.com/blog/2010/nov/5/using-znc-irc-bouncer/Using ZNC, an IRC bouncer2010-11-05T17:00:00+00:00<section id="using-znc-an-irc-bouncer">
<p>I use IRC for work, play, and generic open source questions and
support. I think it’s a pretty integral part of my existence as a
developer. Today I’m going to write about why using an IRC bouncer
makes IRC a ton better and show you how to get one setup.</p>
<section id="id1">
<h2><a class="reference external" href="http://en.znc.in/wiki/ZNC">ZNC</a></h2>
<aside class="system-message">
<p class="system-message-title">System Message: INFO/1 (<span class="docutils literal">/home/docs/checkouts/readthedocs.org/user_builds/ericholschercom/checkouts/latest/site/blog/2010/nov/5/using-znc-irc-bouncer.rst</span>, line 12); <em><a href="#id1">backlink</a></em></p>
<p>Duplicate implicit target name: “znc”.</p>
</aside>
<p>ZNC is the bouncer that I was recommended when I started, and I
only have good things to say about it. On ubuntu installing ZNC is
really simple.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">sudo</span> <span class="n">apt</span><span class="o">-</span><span class="n">get</span> <span class="n">install</span> <span class="n">znc</span><span class="o">-</span><span class="n">extra</span>
</pre></div>
</div>
<p>This will pull down ZNC and it’s extra modules which offer you some
nice features. So now that you have ZNC installed, you need to make
a configuration. You can do this with the <code class="docutils literal notranslate"><span class="pre">znc</span> <span class="pre">--makeconf</span></code>
command, which I’ve pasted a session of at the bottom of this post.
However, here is the config that I run now.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Listen</span> <span class="o">=</span> <span class="o">+</span><span class="mi">6666</span>
<span class="n">ConnectDelay</span> <span class="o">=</span> <span class="mi">30</span>
<span class="o"><</span><span class="n">User</span> <span class="n">eric</span><span class="o">></span>
<span class="n">Pass</span> <span class="o">=</span> <span class="n">md5</span><span class="c1">#blah</span>
<span class="n">Nick</span> <span class="o">=</span> <span class="n">ericholscher</span>
<span class="n">AltNick</span> <span class="o">=</span> <span class="n">ericholscher_</span>
<span class="n">Ident</span> <span class="o">=</span> <span class="n">eric</span>
<span class="n">RealName</span> <span class="o">=</span> <span class="n">Eric</span>
<span class="n">QuitMsg</span> <span class="o">=</span> <span class="n">Peace</span><span class="o">.</span>
<span class="n">StatusPrefix</span> <span class="o">=</span> <span class="o">*</span>
<span class="n">ChanModes</span> <span class="o">=</span> <span class="o">+</span><span class="n">stn</span>
<span class="n">Buffer</span> <span class="o">=</span> <span class="mi">5000</span>
<span class="n">KeepBuffer</span> <span class="o">=</span> <span class="n">false</span>
<span class="n">MultiClients</span> <span class="o">=</span> <span class="n">true</span>
<span class="n">BounceDCCs</span> <span class="o">=</span> <span class="n">true</span>
<span class="n">DenyLoadMod</span> <span class="o">=</span> <span class="n">false</span>
<span class="n">Admin</span> <span class="o">=</span> <span class="n">true</span>
<span class="n">DenySetVHost</span> <span class="o">=</span> <span class="n">false</span>
<span class="n">DCCLookupMethod</span> <span class="o">=</span> <span class="n">default</span>
<span class="n">TimestampFormat</span> <span class="o">=</span> <span class="p">[</span><span class="o">%</span><span class="n">H</span><span class="p">:</span><span class="o">%</span><span class="n">M</span><span class="p">:</span><span class="o">%</span><span class="n">S</span><span class="p">]</span>
<span class="n">AppendTimestamp</span> <span class="o">=</span> <span class="n">false</span>
<span class="n">PrependTimestamp</span> <span class="o">=</span> <span class="n">true</span>
<span class="n">TimezoneOffset</span> <span class="o">=</span> <span class="mf">0.00</span>
<span class="n">JoinTries</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">MaxJoins</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">LoadModule</span> <span class="o">=</span> <span class="n">chansaver</span>
<span class="n">LoadModule</span> <span class="o">=</span> <span class="n">log</span>
<span class="n">Server</span> <span class="o">=</span> <span class="n">irc</span><span class="o">.</span><span class="n">freenode</span><span class="o">.</span><span class="n">net</span> <span class="o">+</span><span class="mi">7000</span>
<span class="o"><</span><span class="n">Chan</span> <span class="c1">#django></span>
<span class="o"></</span><span class="n">Chan</span><span class="o">></span>
<span class="o"></</span><span class="n">User</span><span class="o">></span>
</pre></div>
</div>
<p><strong>Note</strong>: This might be from a slightly older version of ZNC, so
you might have to modify this to work on the newest version of ZNC
in the ubuntu 10.10 repos. Using the –makeconf option with the
same answers will also make an up to date conf.</p>
</section>
<section id="important-options">
<h2>Important options</h2>
<p>The <code class="docutils literal notranslate"><span class="pre">Buffer</span></code> setting is the number of lines to keep in the buffer
when you’re not connected. If you have your ZNC not keep your
buffer, with <code class="docutils literal notranslate"><span class="pre">KeepBuffer</span> <span class="pre">=</span> <span class="pre">false</span></code> setting, I tend to keep the
Buffer setting pretty high. You don’t want to miss messages when
you’re disconnected, so it will just stream these messages back to
you when you reconnect.</p>
<p>The other use of ZNC that I know people have is to connect from
multiple clients, like checking IRC on an iPhone. So long as you
have the <code class="docutils literal notranslate"><span class="pre">MultiClients</span></code> setting set, you can do this and it is
transparent to anyone else in your channels. However, when you
connect, you always want a scrollback of the context of the last
things said. In this case, you’d want to have
<code class="docutils literal notranslate"><span class="pre">KeepBuffer</span> <span class="pre">=</span> <span class="pre">true</span></code>, but have a small <code class="docutils literal notranslate"><span class="pre">Buffer</span></code> playback, so you
don’t spam your phone.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">+6666</span></code> on the Listen portion of the config means that it is
listening on port 6666, using SSL. So make sure if you turn on SSL,
your client is connecting with SSL as well.</p>
</section>
<section id="good-modules">
<h2>Good Modules</h2>
<p>The modules that I have enabled are the chansaver and log modules.
The chansaver module writes out your ZNC config every time you join
or part a channel, so when you restart your bouncer it will have
the rooms you’re in. The log module logs all the channels your in,
which is nice because it allows you to have comprehensive logs of
your work channels.</p>
<p>ZNC also comes with a <a class="reference external" href="http://en.znc.in/wiki/Webadmin">webadmin</a>
that I have never used, but hear is quite nice. It allows you to
configure everything through a web interface. ZNC has a lot more
power than I’ve shown here, including connecting to multiple
backend servers, having multiple users, inter-user chat, and lots
of other interesting things. Once you’ve gotten hooked, you can
share your server with your friends, and play around with modules.</p>
</section>
<section id="using-it">
<h2>Using it</h2>
<p>So now to connect to your proxy, in your IRC client, instead of
connecting to your normal server (eg. irc.freenode.net 6667), you
would connect to your bouncer instead. This will be running on the
IP and port that you have configured in your ZNC config. My
<a class="reference external" href="http://limechat.net/mac/">Limechat</a> config looks like this:</p>
<figure class="align-center" id="id2">
<img alt="Limechat Config" src="../../../_images/limechat-config.png" />
<figcaption>
<p><span class="caption-text">Limechat Config</span></p>
</figcaption>
</figure>
<p>You should then be able to just connect to that server, and your
client will show all the channels you’re connected to. You can try
logging off and back on a couple of minutes later and see that it
plays back what you’ve missed.</p>
<p>This really changes how you interact with IRC I find, because you
can keep tabs on everything that is going on when you aren’t
connected. I can be in the middle of a conversation, disconnect and
move to a meeting room downstairs, and pick right back up where
I’ve left off.</p>
<section id="znc-makeconf-session">
<h3>znc –makeconf session</h3>
<p>Here is a copy of the makeconf session I talked about earlier.
Where there is no visual input, it’s just me accepting the
defaults.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>eric@Chimera:~$ znc --makeconf
[ ** ] Building new config
[ ** ]
[ ** ] First lets start with some global settings...
[ ** ]
[ ?? ] What port would you like ZNC to listen on? (1 to 65535): 6666
[ ?? ] Would you like ZNC to listen using SSL? (yes/no) [no]: yes
[ ** ] Unable to locate pem file: [/home/eric/.znc/znc.pem]
[ ?? ] Would you like to create a new pem file now? (yes/no) [yes]: yes
[ ?? ] hostname of your shell (including the '.com' portion): irc.ericholscher.com
[ ok ] Writing Pem file [/home/eric/.znc/znc.pem]...
[ ?? ] Would you like ZNC to listen using ipv6? (yes/no) [no]:
[ ?? ] Listen Host (Blank for all ips):
[ ** ]
[ ** ] -- Global Modules --
[ ** ]
[ ?? ] Do you want to load any global modules? (yes/no): yes
[ ** ] +-----------+----------------------------------------------------------+
[ ** ] | Name | Description |
[ ** ] +-----------+----------------------------------------------------------+
[ ** ] | partyline | Internal channels and queries for users connected to znc |
[ ** ] | webadmin | Web based administration module |
[ ** ] +-----------+----------------------------------------------------------+
[ ** ] And 10 other (uncommon) modules. You can enable those later.
[ ** ]
[ ?? ] Load global module <partyline>? (yes/no) [no]: no
[ ?? ] Load global module <webadmin>? (yes/no) [no]: yes
[ ** ]
[ ** ] Now we need to setup a user...
[ ** ]
[ ?? ] Username (AlphaNumeric): eric
[ ?? ] Enter Password:
[ ?? ] Confirm Password:
[ ?? ] Would you like this user to be an admin? (yes/no) [yes]:
[ ?? ] Nick [eric]:
[ ?? ] Alt Nick [eric_]:
[ ?? ] Ident [eric]:
[ ?? ] Real Name [Got ZNC?]:
[ ?? ] VHost (optional):
[ ?? ] Number of lines to buffer per channel [50]: 500
[ ?? ] Would you like to keep buffers after replay? (yes/no) [no]:
[ ?? ] Default channel modes [+stn]:
[ ** ]
[ ** ] -- User Modules --
[ ** ]
[ ?? ] Do you want to automatically load any user modules for this user? (yes/no): yes
[ ** ] +-------------+-------------------------------------------------------------------+
[ ** ] | Name | Description |
[ ** ] +-------------+-------------------------------------------------------------------+
[ ** ] | admin | Dynamic configuration of users/settings through irc |
[ ** ] | chansaver | Keep config up-to-date when user joins/parts |
[ ** ] | keepnick | Keep trying for your primary nick |
[ ** ] | kickrejoin | Autorejoin on kick |
[ ** ] | nickserv | Auths you with NickServ |
[ ** ] | perform | Keeps a list of commands to be executed when ZNC connects to IRC. |
[ ** ] | simple_away | Auto away when last client disconnects |
[ ** ] +-------------+-------------------------------------------------------------------+
[ ** ] And 33 other (uncommon) modules. You can enable those later.
[ ** ]
[ ?? ] Load module <admin>? (yes/no) [no]: yes
[ ?? ] Load module <chansaver>? (yes/no) [no]: yes
[ ?? ] Load module <keepnick>? (yes/no) [no]: yes
[ ?? ] Load module <kickrejoin>? (yes/no) [no]:
[ ?? ] Load module <nickserv>? (yes/no) [no]:
[ ?? ] Load module <perform>? (yes/no) [no]:
[ ?? ] Load module <simple_away>? (yes/no) [no]: yes
[ ** ]
[ ** ] -- IRC Servers --
[ ** ]
[ ?? ] IRC server (host only): irc.freenode.net
[ ?? ] [irc.freenode.net] Port (1 to 65535) [6667]:
[ ?? ] [irc.freenode.net] Password (probably empty):
[ ?? ] Does this server use SSL? (probably no) (yes/no) [no]:
[ ** ]
[ ?? ] Would you like to add another server? (yes/no) [no]:
[ ** ]
[ ** ] -- Channels --
[ ** ]
[ ?? ] Would you like to add a channel for ZNC to automatically join? (yes/no) [yes]: yes
[ ?? ] Channel name: #django
[ ?? ] Would you like to add another channel? (yes/no) [no]:
[ ** ]
[ ?? ] Would you like to setup another user? (yes/no) [no]:
[ ok ] Writing config [/home/eric/.znc/configs/znc.conf]...
[ ** ]
[ ** ] To connect to this znc you need to connect to it as your irc server
[ ** ] using the port that you supplied. You have to supply your login info
[ ** ] as the irc server password like so... user:pass.
[ ** ]
[ ** ] Try something like this in your IRC client...
[ ** ] /server <znc_server_ip> 6666 eric:<pass>
[ ** ]
[ ?? ] Launch znc now? (yes/no) [yes]:
[ ok ] Opening Config [/home/eric/.znc/configs/znc.conf]...
[ ok ] Binding to port [+6666] using ipv4...
[ ** ] Loading user [eric]
[ ok ] Loading Module [admin]... [/usr/lib/znc/admin.so]
[ ok ] Loading Module [chansaver]... [/usr/lib/znc/chansaver.so]
[ ok ] Loading Module [keepnick]... [/usr/lib/znc/keepnick.so]
[ ok ] Loading Module [simple_away]... [/usr/lib/znc/simple_away.so]
[ ok ] Adding Server [irc.freenode.net 6667]...
[ ok ] Loading Global Module [webadmin]... [/usr/lib/znc/webadmin.so]
[ ok ] Forking into the background... [pid: 15983]
[ ** ] ZNC 0.092+deb3 - http://znc.sourceforge.net
</pre></div>
</div>
</section>
</section>
</section>
I use IRC for work, play, and generic open source questions and
support. I think it’s a pretty integral part of my existence as a
developer. Today I’m going to write about why using an IRC bouncer
makes IRC a ton better and show you how to get one setup.2010-11-05T17:00:00+00:00http://ericholscher.com/blog/2010/nov/8/building-django-app-server-chef/Building a Django App Server with Chef: Part 12010-11-08T17:00:00+00:00<section id="building-a-django-app-server-with-chef-part-1">
<p>Alternate title: Fucking Chef, How does it work?</p>
<p>When I started looking at
<a class="reference external" href="http://wiki.opscode.com/display/chef/Home">Chef</a>, I found it
incredibly complex and lacking in fundamental documentation. This
is going to be my experience understanding Chef while setting up a
single server. This strategy can be used across multiple servers,
with a little tweaking.</p>
<p>I’d like to thank <a class="reference external" href="http://jacobian.org/">Jacob</a> for the ideas and
some of the inspiration behind the code and ideas. The code for
this blog post can be found
<a class="reference external" href="https://github.com/ericholscher/chef-django-example">on github</a>.
It will be expanded as I write updates.</p>
<p>Today’s code will be using tag
<a class="reference external" href="https://github.com/ericholscher/chef-django-example/tree/blog-post-1">blog-post-1</a>
in the repo.</p>
<section id="goal">
<h2>Goal</h2>
<p>I’m hoping to write a blog series that goes from explaining what
Chef is, to having a working Django server, and to release all this
code so that you can tweak and use it with your own servers. I’ll
be doing this to set up a new and configuration managed server for
<a class="reference external" href="http://readthedocs.org/">ReadTheDocs</a>.</p>
<p>There are a
<a class="reference external" href="http://brainspl.at/articles/2009/01/31/cooking-with-chef-101">couple of</a>
other
<a class="reference external" href="http://morethanseven.net/2010/10/30/Chef-hello-world.html">good chef intros</a>
available. However, they only get you to the super basic first step
of setting up the server. Hopefully this series will be more in
depth and useful to actually getting a real server running under
Chef.</p>
</section>
<section id="basic-terminology">
<h2>Basic terminology</h2>
<p>Chef has some basic terminology that you need to understand before
we get into things. I’m going to purposefully leave out a ton of
stuff, because it isn’t really important for me now.</p>
<ul class="simple">
<li><p>Cookbook (cookbooks/*)</p></li>
</ul>
<p>A cookbook, not surprisingly, contains recipes. With my
configuration, we’re only going to use one cookbook, that has
multiple recipes. This is probably wrong and horrible, but it’s
simple, which is the goal for now.</p>
<ul class="simple">
<li><p>Recipe (cookbooks/*/recipes/*.rb)</p></li>
</ul>
<p>A recipe is the basic building block of Chef. It is what does the
meat of the work that you want done.</p>
<ul class="simple">
<li><p>Resources</p></li>
</ul>
<p>A resources is an abstraction that defines a specific piece of your
configuration. It can be a file, a user, or just about anything you
want to talk about on your system.</p>
<ul class="simple">
<li><p>Attributes (node.json)</p></li>
</ul>
<p>Attributes are just a blob of JSON that Chef uses to pass around
data. It will be (slightly) different for each server that you want
to set up. I really like this approach, because I think a
data-driven approach is the correct way to solve this problem.</p>
</section>
<section id="bootstrapping-chef">
<h2>Bootstrapping chef</h2>
<p>We’ll be running what is called
<a class="reference external" href="http://wiki.opscode.com/display/chef/Chef+Solo">chef solo</a>. This
means that chef will be run independently of a server, and just
execute on our one host. This seems to be the easiest way to get
things running.</p>
<p>When you first run <code class="docutils literal notranslate"><span class="pre">chef-solo</span></code>, it will look for a <code class="docutils literal notranslate"><span class="pre">solo.rb</span></code>
file. The
<a class="reference external" href="http://wiki.opscode.com/display/chef/Chef+Solo#ChefSolo-ConfigureChefSolo">documentation about configuring chef-solo</a>
does a decent job of explaining this. By default it looks in
/etc/chef/solo.rb, so lets go ahead and use that. It has 2 required
fields, a <code class="docutils literal notranslate"><span class="pre">cookbook_path</span></code> and <code class="docutils literal notranslate"><span class="pre">json_attribs</span></code>. <code class="docutils literal notranslate"><span class="pre">cookbook_path</span></code>
tells chef where to find the “cookbooks” it uses to run your code,
and <code class="docutils literal notranslate"><span class="pre">json_attribs</span></code> tells it where to load in your data
dictionary.</p>
<p><strong>Note</strong>: The docs say that <code class="docutils literal notranslate"><span class="pre">file_cache_path</span></code> is required, but it
just defaults to <code class="docutils literal notranslate"><span class="pre">/var/chef/cache</span></code>.</p>
<p>For simplicity, we’re going to keep our cookbooks and json data
file in <code class="docutils literal notranslate"><span class="pre">/etc/chef</span></code>. On my local machine I keep everything in a
<code class="docutils literal notranslate"><span class="pre">~/projects/chef</span></code> directory, and then sync that to the remote
box.</p>
<p>In production you’ll probably want to set up your remote server to
pull from a repository somewhere, so that you can have a stable
deployment base, but again, syncing from the local filesystem is
simple and works.</p>
</section>
<section id="bootstrapping-your-new-server">
<h2>Bootstrapping your new server</h2>
<p>I’ll be using a <a class="reference external" href="http://docs.fabfile.org/">fabric</a> script to run
the commands on a remote server, which also allows me to run them
again later on a new machine in an automated fashion. So this is
the first simple script that we’ll use to bootstrap our new server,
<code class="docutils literal notranslate"><span class="pre">fabfile.py</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">fabric.api</span> <span class="kn">import</span> <span class="n">env</span><span class="p">,</span> <span class="n">local</span><span class="p">,</span> <span class="n">sudo</span>
<span class="n">env</span><span class="o">.</span><span class="n">user</span> <span class="o">=</span> <span class="s1">'root'</span>
<span class="n">env</span><span class="o">.</span><span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'204.232.205.196'</span><span class="p">]</span>
<span class="n">env</span><span class="o">.</span><span class="n">chef_executable</span> <span class="o">=</span> <span class="s1">'/var/lib/gems/1.8/bin/chef-solo'</span>
<span class="k">def</span> <span class="nf">install_chef</span><span class="p">():</span>
<span class="n">sudo</span><span class="p">(</span><span class="s1">'apt-get update'</span><span class="p">,</span> <span class="n">pty</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">sudo</span><span class="p">(</span><span class="s1">'apt-get install -y git-core rubygems ruby ruby-dev'</span><span class="p">,</span> <span class="n">pty</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">sudo</span><span class="p">(</span><span class="s1">'gem install chef'</span><span class="p">,</span> <span class="n">pty</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">sync_config</span><span class="p">():</span>
<span class="n">local</span><span class="p">(</span><span class="s1">'rsync -av . </span><span class="si">%s</span><span class="s1">@</span><span class="si">%s</span><span class="s1">:/etc/chef'</span> <span class="o">%</span> <span class="p">(</span><span class="n">env</span><span class="o">.</span><span class="n">user</span><span class="p">,</span> <span class="n">env</span><span class="o">.</span><span class="n">hosts</span><span class="p">[</span><span class="mi">0</span><span class="p">]))</span>
<span class="k">def</span> <span class="nf">update</span><span class="p">():</span>
<span class="n">sync_config</span><span class="p">()</span>
<span class="n">sudo</span><span class="p">(</span><span class="s1">'cd /etc/chef && </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">env</span><span class="o">.</span><span class="n">chef_executable</span><span class="p">,</span> <span class="n">pty</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</pre></div>
</div>
<p><strong>A couple notes</strong>: the chef executable is version-dependent, but
that’s because I don’t know enough ruby to query it dynamically.
You will also need to change the value of the env.hosts to a server
that you actually control.</p>
<p>This will install the ruby dependencies, and get chef up and
running for you. You’ll need to install fabric
(<code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">fabric</span></code>, presumably in a virtualenv), and then run
<code class="docutils literal notranslate"><span class="pre">fab</span> <span class="pre">install_chef</span></code> to get it up and running. Then you can go
ahead and run <code class="docutils literal notranslate"><span class="pre">fab</span> <span class="pre">sync_config</span></code> to get your chef configuration
onto the remote server.</p>
<p>Now we need to go ahead and figure out how to make chef actually do
something. You’ll see in the <code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/default.rb</span></code>
file we have a simple <code class="docutils literal notranslate"><span class="pre">package</span></code> declaration. This simply means to
make sure that the package is installed on the remote system. This
is as simple as it gets, so lets go ahead and run it. With the
<code class="docutils literal notranslate"><span class="pre">fab</span> <span class="pre">update</span></code> command, it will sync your local directory, and run
chef on the remote server.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">-></span> <span class="n">fab</span> <span class="n">update</span>
<span class="p">[</span><span class="n">localhost</span><span class="p">]</span> <span class="n">run</span><span class="p">:</span> <span class="n">rsync</span> <span class="o">-</span><span class="n">av</span> <span class="o">.</span> <span class="n">root</span><span class="o">@</span><span class="mf">204.232.205.196</span><span class="p">:</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">chef</span>
<span class="p">[</span><span class="mf">204.232.205.196</span><span class="p">]</span> <span class="n">sudo</span><span class="p">:</span> <span class="n">cd</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">chef</span> <span class="o">&&</span> <span class="o">/</span><span class="n">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">gems</span><span class="o">/</span><span class="mf">1.8</span><span class="o">/</span><span class="n">gems</span><span class="o">/</span><span class="n">chef</span><span class="o">-</span><span class="mf">0.9.12</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">chef</span><span class="o">-</span><span class="n">solo</span>
<span class="p">[</span><span class="mf">204.232.205.196</span><span class="p">]</span> <span class="n">err</span><span class="p">:</span> <span class="n">stdin</span><span class="p">:</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">a</span> <span class="n">tty</span>
<span class="p">[</span><span class="mf">204.232.205.196</span><span class="p">]</span> <span class="n">out</span><span class="p">:</span> <span class="p">[</span><span class="n">Tue</span><span class="p">,</span> <span class="mi">09</span> <span class="n">Nov</span> <span class="mi">2010</span> <span class="mi">01</span><span class="p">:</span><span class="mi">42</span><span class="p">:</span><span class="mi">01</span> <span class="o">+</span><span class="mi">0000</span><span class="p">]</span> <span class="n">INFO</span><span class="p">:</span> <span class="n">Setting</span> <span class="n">the</span> <span class="n">run_list</span> <span class="n">to</span> <span class="p">[</span><span class="s2">"main::default"</span><span class="p">]</span> <span class="kn">from</span> <span class="nn">JSON</span>
<span class="p">[</span><span class="mf">204.232.205.196</span><span class="p">]</span> <span class="n">out</span><span class="p">:</span> <span class="p">[</span><span class="n">Tue</span><span class="p">,</span> <span class="mi">09</span> <span class="n">Nov</span> <span class="mi">2010</span> <span class="mi">01</span><span class="p">:</span><span class="mi">42</span><span class="p">:</span><span class="mi">01</span> <span class="o">+</span><span class="mi">0000</span><span class="p">]</span> <span class="n">INFO</span><span class="p">:</span> <span class="n">Starting</span> <span class="n">Chef</span> <span class="n">Run</span> <span class="p">(</span><span class="n">Version</span> <span class="mf">0.9.12</span><span class="p">)</span>
<span class="p">[</span><span class="mf">204.232.205.196</span><span class="p">]</span> <span class="n">out</span><span class="p">:</span> <span class="p">[</span><span class="n">Tue</span><span class="p">,</span> <span class="mi">09</span> <span class="n">Nov</span> <span class="mi">2010</span> <span class="mi">01</span><span class="p">:</span><span class="mi">42</span><span class="p">:</span><span class="mi">01</span> <span class="o">+</span><span class="mi">0000</span><span class="p">]</span> <span class="n">INFO</span><span class="p">:</span> <span class="n">Installing</span> <span class="n">package</span><span class="p">[</span><span class="n">curl</span><span class="p">]</span> <span class="n">version</span> <span class="mf">7.21.0</span><span class="o">-</span><span class="mi">1</span><span class="n">ubuntu1</span>
<span class="p">[</span><span class="mf">204.232.205.196</span><span class="p">]</span> <span class="n">out</span><span class="p">:</span> <span class="p">[</span><span class="n">Tue</span><span class="p">,</span> <span class="mi">09</span> <span class="n">Nov</span> <span class="mi">2010</span> <span class="mi">01</span><span class="p">:</span><span class="mi">42</span><span class="p">:</span><span class="mi">04</span> <span class="o">+</span><span class="mi">0000</span><span class="p">]</span> <span class="n">INFO</span><span class="p">:</span> <span class="n">Chef</span> <span class="n">Run</span> <span class="n">complete</span> <span class="ow">in</span> <span class="mf">2.574963</span> <span class="n">seconds</span>
<span class="p">[</span><span class="mf">204.232.205.196</span><span class="p">]</span> <span class="n">out</span><span class="p">:</span> <span class="p">[</span><span class="n">Tue</span><span class="p">,</span> <span class="mi">09</span> <span class="n">Nov</span> <span class="mi">2010</span> <span class="mi">01</span><span class="p">:</span><span class="mi">42</span><span class="p">:</span><span class="mi">04</span> <span class="o">+</span><span class="mi">0000</span><span class="p">]</span> <span class="n">INFO</span><span class="p">:</span> <span class="n">cleaning</span> <span class="n">the</span> <span class="n">checksum</span> <span class="n">cache</span>
<span class="p">[</span><span class="mf">204.232.205.196</span><span class="p">]</span> <span class="n">out</span><span class="p">:</span> <span class="p">[</span><span class="n">Tue</span><span class="p">,</span> <span class="mi">09</span> <span class="n">Nov</span> <span class="mi">2010</span> <span class="mi">01</span><span class="p">:</span><span class="mi">42</span><span class="p">:</span><span class="mi">04</span> <span class="o">+</span><span class="mi">0000</span><span class="p">]</span> <span class="n">INFO</span><span class="p">:</span> <span class="n">Running</span> <span class="n">report</span> <span class="n">handlers</span>
<span class="p">[</span><span class="mf">204.232.205.196</span><span class="p">]</span> <span class="n">out</span><span class="p">:</span> <span class="p">[</span><span class="n">Tue</span><span class="p">,</span> <span class="mi">09</span> <span class="n">Nov</span> <span class="mi">2010</span> <span class="mi">01</span><span class="p">:</span><span class="mi">42</span><span class="p">:</span><span class="mi">04</span> <span class="o">+</span><span class="mi">0000</span><span class="p">]</span> <span class="n">INFO</span><span class="p">:</span> <span class="n">Report</span> <span class="n">handlers</span> <span class="n">complete</span>
</pre></div>
</div>
<p><strong>You now have Chef running on your server</strong>. That was pretty easy,
eh? For tomorrow’s lesson, we’ll be making it actually do
something, like installed nginx and gunicorn, and keeping track of
config files.</p>
</section>
</section>
Alternate title: Fucking Chef, How does it work?2010-11-08T17:00:00+00:00http://ericholscher.com/blog/2010/nov/9/building-django-app-server-chef-part-2/Building a Django App Server with Chef: Part 22010-11-09T17:00:00+00:00<section id="building-a-django-app-server-with-chef-part-2">
<p>Alternate title: Actually doing something useful.</p>
<p><a class="reference external" href="http://ericholscher.com/blog/2010/nov/8/building-django-app-server-chef/">Yesterday</a>
we covered the basics to getting started with Chef. You should have
a remote server configured with chef, and have curl installed! Now
lets go ahead and get some useful bits for your Django
application.</p>
<section id="what-we-ll-need">
<h2>What we’ll need</h2>
<p>So this is going to be based around the way that I set up my
servers, so if this is different than you, I’m sorry. However, I
think it is a pretty solid way of managing them. A lot of the ideas
here are stolen from <a class="reference external" href="http://traviscline.com/">Travis</a> when he
set up the server for Pypants.</p>
<p>So lets assemble a list of things we’re going to want in order to
get a super basic Django configuration running:</p>
<ul class="simple">
<li><p>A user to run our code as and who’s home directory we’ll store
the data.</p></li>
<li><p>A basic global python ecosystem, including setuptools and pip</p></li>
<li><p>A virtualenv to store all the project-specific packages and code
in</p></li>
<li><p>A copy of the project that we’ll be running</p></li>
</ul>
<p>Let’s get started.</p>
<p>The finished code for today is located on github, with the
<a class="reference external" href="https://github.com/ericholscher/chef-django-example/tree/blog-post-2">tag blog-post-2</a>.
It is a copy of the completed steps, so feel free to peek through
that and come back here for clarification (or to ask questions).</p>
</section>
<section id="setting-up-our-user">
<h2>Setting up our user</h2>
<p>For <a class="reference external" href="http://readthedocs.org/">RTD</a>, I run everything under the
user docs. So we’ll go ahead and set up that user so that we can
get our site set up. We’re going to go ahead and replace our
“default” recipe, because right now it isn’t doing anything much
useful. The relevant part is below:</p>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/default.rb</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>node[:base_packages].each do |pkg|
package pkg do
:upgrade
end
end
node[:users].each_pair do |username, info|
group username do
gid info[:id]
end
user username do
comment info[:full_name]
uid info[:id]
gid info[:id]
shell info[:disabled] ? "/sbin/nologin" : "/bin/bash"
supports :manage_home => true
home "/home/#{username}"
end
directory "/home/#{username}/.ssh" do
owner username
group username
mode 0700
end
file "/home/#{username}/.ssh/authorized_keys" do
owner username
group username
mode 0600
content info[:key]
end
end
node[:groups].each_pair do |name, info|
group name do
gid info[:gid]
members info[:members]
end
end
</pre></div>
</div>
<p>There’s a lot of stuff going on here, so lets go over it. First you
might notice that there’s this node variable, the node data
structure is the JSON that you have in your node.json file. It is
looping over the keys and values with ruby’s each_pair and pair
functions.</p>
<p>The base_packages bit is a cool example of the power of the chef
configuration. We have a list of packages that we want to install
in our Attributes, and we’re looping over them and setting using
the package Resource.</p>
<p>I realize I skipped over the run_list part yesterday, but it
basically is just a list of recipes to run. Each of the resources
in the default.rb file should be pretty self explanatory. The
<a class="reference external" href="http://wiki.opscode.com/display/chef/Resources">Chef Resource Documentation</a>
is really comprehensive, and will probably be the most referenced
document that you use. The main resource’s that we used were
<strong>group, user, file, directory</strong>, let’s take a look at the
<a class="reference external" href="http://wiki.opscode.com/display/chef/Resources#Resources-User">User</a>
declaration in particular.</p>
<p>Everything there should be pretty obvious, as it’s the information
that goes into /etc/passwd for the user. However, the <code class="docutils literal notranslate"><span class="pre">supports</span></code>
keyword isn’t obvious at first. This is part of the
<a class="reference external" href="http://wiki.opscode.com/display/chef/Resources#Resources-CommonAttributes">Common Attributes</a>
that can be set on all Resources. It’s a way of passing along
configuration options to the Resource. manage_home actually just
makes it so that the users home directory is created when the user
is created.</p>
<p>So we’re going to have to go ahead and put some data in there for
it to work with. Our node.json will now look like this:</p>
<p><code class="docutils literal notranslate"><span class="pre">node.json</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">"run_list"</span><span class="p">:</span> <span class="p">[</span> <span class="s2">"main::default"</span><span class="p">,</span> <span class="s2">"main::python"</span><span class="p">,</span> <span class="s2">"main::readthedocs"</span> <span class="p">],</span>
<span class="s2">"base_packages"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"git-core"</span><span class="p">,</span> <span class="s2">"bash-completion"</span><span class="p">],</span>
<span class="s2">"users"</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">"docs"</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">"id"</span><span class="p">:</span> <span class="mi">1001</span><span class="p">,</span>
<span class="s2">"full_name"</span><span class="p">:</span> <span class="s2">"Docs User"</span><span class="p">,</span>
<span class="s2">"key"</span><span class="p">:</span> <span class="s2">"ssh-rsa key-goes-here eric@Bahamut"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="s2">"groups"</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">"docs"</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">"gid"</span><span class="p">:</span> <span class="mi">201</span><span class="p">,</span>
<span class="s2">"members"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"docs"</span><span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
</section>
<section id="adding-a-basic-python-environment">
<h2>Adding a Basic Python Environment</h2>
<p>Now lets go ahead and add a python recipe to build out some basic
python stuff that we’ll be needing.</p>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/python.rb</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">node</span><span class="p">[:</span><span class="n">ubuntu_python_packages</span><span class="p">]</span><span class="o">.</span><span class="n">each</span> <span class="n">do</span> <span class="o">|</span><span class="n">pkg</span><span class="o">|</span>
<span class="n">package</span> <span class="n">pkg</span> <span class="n">do</span>
<span class="p">:</span><span class="n">upgrade</span>
<span class="n">end</span>
<span class="n">end</span>
<span class="c1"># System-wide packages installed by pip.</span>
<span class="c1"># Careful here: most Python stuff should be in a virtualenv.</span>
<span class="n">node</span><span class="p">[:</span><span class="n">pip_python_packages</span><span class="p">]</span><span class="o">.</span><span class="n">each_pair</span> <span class="n">do</span> <span class="o">|</span><span class="n">pkg</span><span class="p">,</span> <span class="n">version</span><span class="o">|</span>
<span class="n">execute</span> <span class="s2">"install-#</span><span class="si">{pkg}</span><span class="s2">"</span> <span class="n">do</span>
<span class="n">command</span> <span class="s2">"pip install #</span><span class="si">{pkg}</span><span class="s2">==#</span><span class="si">{version}</span><span class="s2">"</span>
<span class="n">not_if</span> <span class="s2">"[ `pip freeze | grep #</span><span class="si">{pkg}</span><span class="s2"> | cut -d'=' -f3` = '#</span><span class="si">{version}</span><span class="s2">' ]"</span>
<span class="n">end</span>
<span class="n">end</span>
</pre></div>
</div>
<p>Additions to <code class="docutils literal notranslate"><span class="pre">node.json</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="s2">"ubuntu_python_packages"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"python-setuptools"</span><span class="p">,</span> <span class="s2">"python-pip"</span><span class="p">,</span> <span class="s2">"python-dev"</span><span class="p">,</span> <span class="s2">"libpq-dev"</span><span class="p">],</span>
<span class="s2">"pip_python_packages"</span><span class="p">:</span> <span class="p">{</span><span class="s2">"virtualenv"</span><span class="p">:</span> <span class="s2">"1.5.1"</span><span class="p">,</span> <span class="s2">"mercurial"</span><span class="p">:</span> <span class="s2">"1.7"</span><span class="p">},</span>
</pre></div>
</div>
<p>Here we’re adding some global packages that we need. We’re going to
install setuptools and pip so that we can install further python
packages. python-dev and libpq-dev are so that we have the headers
for libraries that need to compile against postgres and python.
We’ll also be installing virtualenv and mercurial globally so that
we can create our virtualenv and install packages from mercurial.</p>
</section>
<section id="creating-a-virtualenv">
<h2>Creating a virtualenv</h2>
<p>We’re going to introduce the first new Chef concept here, which is
called a
<a class="reference external" href="http://wiki.opscode.com/display/chef/Definitions">Definition</a>.</p>
<ul>
<li><p>Definition (cookbooks/*/definitions/*.rb)</p>
<p>A definition is a custom Resource that you build to abstract a set
of operations. Pretty simple</p>
</li>
</ul>
<p>This is a definition that
<a class="reference external" href="https://gist.github.com/612395">Jacob published</a> and then I
updated to make the permissions correct. It allows you to set up a
virtualenv:</p>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/definitions/virtualenv.rb</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>define :virtualenv, :action => :create, :owner => "root", :group => "root", :mode => 0755, :packages => {} do
path = params[:path] ? params[:path] : params[:name]
if params[:action] == :create
# Manage the directory.
directory path do
owner params[:owner]
group params[:group]
mode params[:mode]
end
execute "create-virtualenv-#{path}" do
user params[:owner]
group params[:group]
command "virtualenv #{path}"
not_if "test -f #{path}/bin/python"
end
params[:packages].each_pair do |package, version|
pip = "#{path}/bin/pip"
execute "install-#{package}-#{path}" do
user params[:owner]
group params[:group]
command "#{pip} install #{package}==#{version}"
not_if "[ `#{pip} freeze | grep #{package} | cut -d'=' -f3` = '#{version}' ]"
end
end
elsif params[:action] == :delete
directory path do
action :delete
recursive true
end
end
end
</pre></div>
</div>
<p>As you can see, it takes a bunch of arguments, then just wraps up a
bunch of Resource definitions in a nice little package. There is a
little bit of magic with the pip freezing things, but it’s
basically just how we’re checking to make sure that a package isn’t
instead before we install it. We are using only using the
<strong>directory and execute</strong> Resources here.</p>
<p>Now we’re going to use this virtualenv Definition, and create the
home virtualenv for our site. I like to keep my virtualenv’s in
<code class="docutils literal notranslate"><span class="pre">~/sites/<domain></span></code>, so this will go into
<code class="docutils literal notranslate"><span class="pre">/home/docs/sites/readthedocs.org/</span></code>. Since this is becoming
specific to the site we’re building, it’s going to go into a
readthedocs recipe:</p>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/readthedocs.rb</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">directory</span> <span class="s2">"/home/docs/sites/"</span> <span class="n">do</span>
<span class="n">owner</span> <span class="s2">"docs"</span>
<span class="n">group</span> <span class="s2">"docs"</span>
<span class="n">mode</span> <span class="mi">0775</span>
<span class="n">end</span>
<span class="n">virtualenv</span> <span class="s2">"/home/docs/sites/readthedocs.org"</span> <span class="n">do</span>
<span class="n">owner</span> <span class="s2">"docs"</span>
<span class="n">group</span> <span class="s2">"docs"</span>
<span class="n">mode</span> <span class="mi">0775</span>
<span class="n">end</span>
</pre></div>
</div>
<p>This will set up a basic virtualenv in our directory.</p>
</section>
<section id="getting-our-site-set-up">
<h2>Getting our site set up</h2>
<p>To get our site set up, we need to pull in the source code, and
make sure our virtualenv has all the requirements. This code is a
little bit hacky, and could probably be abstracted out a bit, but
it will work for now. We’re going to go ahead and add some things
to our readthedocs Recipe.</p>
<p>Additions to <code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/readthedocs.rb</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">directory</span> <span class="s2">"/home/docs/sites/readthedocs.org/run"</span> <span class="n">do</span>
<span class="n">owner</span> <span class="s2">"docs"</span>
<span class="n">group</span> <span class="s2">"docs"</span>
<span class="n">mode</span> <span class="mi">0775</span>
<span class="n">end</span>
<span class="n">git</span> <span class="s2">"/home/docs/sites/readthedocs.org/checkouts/readthedocs.org"</span> <span class="n">do</span>
<span class="n">repository</span> <span class="s2">"git://github.com/rtfd/readthedocs.org.git"</span>
<span class="n">reference</span> <span class="s2">"HEAD"</span>
<span class="n">user</span> <span class="s2">"docs"</span>
<span class="n">group</span> <span class="s2">"docs"</span>
<span class="n">action</span> <span class="p">:</span><span class="n">sync</span>
<span class="n">end</span>
<span class="n">script</span> <span class="s2">"Install Requirements"</span> <span class="n">do</span>
<span class="n">interpreter</span> <span class="s2">"bash"</span>
<span class="n">user</span> <span class="s2">"docs"</span>
<span class="n">group</span> <span class="s2">"docs"</span>
<span class="n">code</span> <span class="o"><<-</span><span class="n">EOH</span>
<span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">docs</span><span class="o">/</span><span class="n">sites</span><span class="o">/</span><span class="n">readthedocs</span><span class="o">.</span><span class="n">org</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">pip</span> <span class="n">install</span> <span class="o">-</span><span class="n">r</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">docs</span><span class="o">/</span><span class="n">sites</span><span class="o">/</span><span class="n">readthedocs</span><span class="o">.</span><span class="n">org</span><span class="o">/</span><span class="n">checkouts</span><span class="o">/</span><span class="n">readthedocs</span><span class="o">.</span><span class="n">o</span>
<span class="n">rg</span><span class="o">/</span><span class="n">deploy_requirements</span><span class="o">.</span><span class="n">txt</span>
<span class="n">EOH</span>
<span class="n">end</span>
</pre></div>
</div>
<p>I like to have my runtime files in the <code class="docutils literal notranslate"><span class="pre">venv/run</span></code> directory, so
we’ll go ahead and create that directory. Then comes the fun part.</p>
<p>We are checking the Readthedocs source out of github with the
<a class="reference external" href="http://wiki.opscode.com/display/chef/Deploy+Resource#DeployResource-Examples">git</a>
Resource. Chef only supports git and svn as far as I can tell, so
luckily I’m using git.</p>
<p>Then we’re going to install from the pip requirements file. This is
using the
<a class="reference external" href="http://wiki.opscode.com/display/chef/Resources#Resources-Script">script Resource</a>,
which allows you to inline a bash, ruby, python, or more script
inside your Recipe. This is using a hard coded bash script to
install the requirements, which sucks, but will work for now.</p>
<p><strong>Note</strong>: Chef appears to buffer output and not show itself as
doing anything when running the script Resource here, so it will
look like your build will hang while it installs your pip
requirements file for the first time.</p>
</section>
<section id="done-for-now">
<h2>Done for now</h2>
<p>Alright, this post has gotten long enough, so we’re done for today.
But we’re in a pretty awesome spot, I think. We now have our app
server set up with a runnable version of our code. You can go ssh
in and play around, you should be able to run simple manage.py
commands inside the virtualenv and whatnot (after a syncdb).</p>
<p>Tomorrow we’ll talk about deploying our code with Nginx and
Gunicorn. I’ve been having trouble with Upstart, so we might switch
our deployment to Supervisord, but we’ll see how it goes.</p>
<p>Don’t forget to check out the finished code
<a class="reference external" href="https://github.com/ericholscher/chef-django-example/tree/blog-post-2">on Github</a>
to see the actual running examples.</p>
</section>
</section>
Alternate title: Actually doing something useful.2010-11-09T17:00:00+00:00http://ericholscher.com/blog/2010/nov/10/building-django-app-server-chef-part-3/Building a Django App Server with Chef: Part 32010-11-10T18:12:00+00:00<section id="building-a-django-app-server-with-chef-part-3">
<p>Alternate title: Show the world what you’ve got.</p>
<p>This is Part 3 of my Chef tutorial. Today we’re talking about
deployment. You can check out the first 2 parts of the series:</p>
<ul class="simple">
<li><p><a class="reference external" href="http://ericholscher.com/blog/2010/nov/8/building-django-app-server-chef/">Part 1: Chef Beginnings</a></p></li>
<li><p><a class="reference external" href="http://ericholscher.com/blog/2010/nov/9/building-django-app-server-chef-part-2/">Part 2: Python environment buildout</a></p></li>
</ul>
<p>Today’s code will be in the git repo under the tag
<a class="reference external" href="https://github.com/ericholscher/chef-django-example/tree/blog-post-3">blog-post-3</a>.</p>
<section id="what-we-ll-need">
<h2>What we’ll need</h2>
<p>We’ll be taking the Django application that we have on the server
and actually deploying it. Let’s make a list of what we’ll need:</p>
<ul class="simple">
<li><p>A web server to sit in front and proxy requests</p></li>
<li><p>A WSGI server</p></li>
<li><p>A way to keep both of these things running</p></li>
<li><p>A caching layer</p></li>
</ul>
<p>We’ll be using Nginx, Memcached, Upstart, and Gunicorn. This is my
preferred deployment stack as of late, mainly because of the simple
setup.</p>
<p>Let’s get started</p>
</section>
<section id="a-web-server">
<h2>A web server</h2>
<p>Getting Nginx up and running should be old hat by this point. We’re
going to need the package and service Resources, which will tell
Chef to install and run it.</p>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/nginx.rb</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">package</span> <span class="s2">"nginx"</span> <span class="n">do</span>
<span class="p">:</span><span class="n">upgrade</span>
<span class="n">end</span>
<span class="n">service</span> <span class="s2">"nginx"</span> <span class="n">do</span>
<span class="n">enabled</span> <span class="n">true</span>
<span class="n">running</span> <span class="n">true</span>
<span class="n">supports</span> <span class="p">:</span><span class="n">status</span> <span class="o">=></span> <span class="n">true</span><span class="p">,</span> <span class="p">:</span><span class="n">restart</span> <span class="o">=></span> <span class="n">true</span><span class="p">,</span> <span class="p">:</span><span class="n">reload</span> <span class="o">=></span> <span class="n">true</span>
<span class="n">action</span> <span class="p">[:</span><span class="n">start</span><span class="p">,</span> <span class="p">:</span><span class="n">enable</span><span class="p">]</span>
<span class="n">end</span>
<span class="n">cookbook_file</span> <span class="s2">"/etc/nginx/sites-enabled/readthedocs"</span> <span class="n">do</span>
<span class="n">source</span> <span class="s2">"nginx/readthedocs"</span>
<span class="n">mode</span> <span class="mi">0640</span>
<span class="n">owner</span> <span class="s2">"root"</span>
<span class="n">group</span> <span class="s2">"root"</span>
<span class="n">notifies</span> <span class="p">:</span><span class="n">restart</span><span class="p">,</span> <span class="n">resources</span><span class="p">(:</span><span class="n">service</span> <span class="o">=></span> <span class="s2">"nginx"</span><span class="p">)</span>
<span class="n">end</span>
<span class="n">cookbook_file</span> <span class="s2">"/etc/nginx/nginx.conf"</span> <span class="n">do</span>
<span class="n">source</span> <span class="s2">"nginx/nginx.conf"</span>
<span class="n">mode</span> <span class="mi">0640</span>
<span class="n">owner</span> <span class="s2">"root"</span>
<span class="n">group</span> <span class="s2">"root"</span>
<span class="n">notifies</span> <span class="p">:</span><span class="n">restart</span><span class="p">,</span> <span class="n">resources</span><span class="p">(:</span><span class="n">service</span> <span class="o">=></span> <span class="s2">"nginx"</span><span class="p">)</span>
<span class="n">end</span>
</pre></div>
</div>
<p>As you can see, we’re providing our own nginx.conf and a
readthedocs site configuration. I’m not going to paste these in, as
they are pretty application specific, but you can look at them
<a class="reference external" href="https://github.com/ericholscher/chef-django-example/tree/blog-post-3/cookbooks/main/files/default/nginx/">on Github</a>
if you’re curious. I also wrote about it
<a class="reference external" href="http://ericholscher.com/blog/2010/aug/28/new-feautures-read-docs/">a while back</a>.</p>
<p>The only new part here is the <code class="docutils literal notranslate"><span class="pre">notifies</span></code> command, which is pretty
nifty. It basically means that whenever you change the nginx.conf
file, it should restart Nginx, which is a really nice feature.</p>
</section>
<section id="a-wsgi-server">
<h2>A WSGI Server</h2>
<p>Yesterday, when we installed the deploy_requirements.txt with pip,
it pulled in <a class="reference external" href="http://gunicorn.org/">gunicorn</a>. So we actually
have Gunicorn already installed in our virtualenv, waiting for us
to use it. The only difference is I actually committed a change to
the ReadTheDocs source so that it will pull Gunicorn from the git
master, which I’ll explain below.</p>
</section>
<section id="upstart">
<h2>Upstart</h2>
<p><strong>Note</strong>: I use upstart because it ships with Ubuntu, so you don’t
need to install a separate package. However, it has pretty horrible
documentation, with the
<a class="reference external" href="http://upstart.ubuntu.com/wiki/Stanzas">Stanzas</a> doc probably
the best clue as to what it supports.</p>
<p>Here is where things get interesting. I spent a bunch of time
trying to get gunicorn and upstart to play nicely yesterday night,
but it wasn’t working. I went on the #gunicorn IRC channel on
Freenode today, and talked with benoitc. He was awesome and
<a class="reference external" href="https://github.com/benoitc/gunicorn/commit/f29c61091691135dcfae029a7eadf1663a06a73e">patched gunicorn</a>
to work with Upstart for me.</p>
<p>Here is the upstart script that we’re using to keep gunicorn
running:</p>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/files/default/gunicorn.conf</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>description "Gunicorn for ReadTheDocs"
start on runlevel [2345]
stop on runlevel [!2345]
#Send KILL after 5 seconds
kill timeout 5
respawn
env VENV="/home/docs/sites/readthedocs.org"
#Serve Gunicorn on the internal rackspace IP.
script
exec sudo -u docs $VENV/bin/gunicorn_django --preload -w 2 --log-level debug --log-file $VENV/run/gunicorn.log -p $VENV/run/gunicorn.pid -b 10.177.69.207:8888 $VENV/checkouts/readthedocs.org/settings/postgres.py
end script
</pre></div>
</div>
<p>As you can see, an Upstart script is a pretty clean way to do this.
If you’ve ever tried to write an old SysV-style init script, this
will look beautiful. You’ll notice that we aren’t passing the
–daemon parameter to gunicorn, this is because upstart will
background the process for us, and keep track of everything, so we
don’t need gunicorn’s daemonizing behavior.</p>
<p>It should be pointed out how awesome it is that we can run a
production ready WSGI server with a single line of bash. If you’ve
ever set up a mod_wsgi install, needing to fuddle with your
apache.conf and a WSGI file and everything makes it a chore. This
is quite simply the easiest way to deploy a WSGI application.</p>
<p>Then we need some additions to
<code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/readthedocs.rb</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cookbook_file</span> <span class="s2">"/etc/init/readthedocs-gunicorn.conf"</span> <span class="n">do</span>
<span class="n">source</span> <span class="s2">"gunicorn.conf"</span>
<span class="n">owner</span> <span class="s2">"root"</span>
<span class="n">group</span> <span class="s2">"root"</span>
<span class="n">mode</span> <span class="mi">0644</span>
<span class="n">end</span>
<span class="n">service</span> <span class="s2">"readthedocs-gunicorn"</span> <span class="n">do</span>
<span class="n">provider</span> <span class="n">Chef</span><span class="p">::</span><span class="n">Provider</span><span class="p">::</span><span class="n">Service</span><span class="p">::</span><span class="n">Upstart</span>
<span class="n">enabled</span> <span class="n">true</span>
<span class="n">running</span> <span class="n">true</span>
<span class="n">supports</span> <span class="p">:</span><span class="n">restart</span> <span class="o">=></span> <span class="n">true</span><span class="p">,</span> <span class="p">:</span><span class="n">reload</span> <span class="o">=></span> <span class="n">true</span><span class="p">,</span> <span class="p">:</span><span class="n">status</span> <span class="o">=></span> <span class="n">true</span>
<span class="n">action</span> <span class="p">[:</span><span class="n">enable</span><span class="p">,</span> <span class="p">:</span><span class="n">start</span><span class="p">]</span>
<span class="n">end</span>
</pre></div>
</div>
<p>Here you can see we’re doing a similar thing to the other service
declarations. We however need to tell Chef to use Upstart for this
service, instead of defaulting to init.d. Other than that,
everything here should look similar to the other files and services
we’ve set up.</p>
</section>
<section id="memcached">
<h2>Memcached</h2>
<p>As you would expect, installing memcached is just like nginx:</p>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/memcached.rb</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">package</span> <span class="s2">"memcached"</span> <span class="n">do</span>
<span class="p">:</span><span class="n">upgrade</span>
<span class="n">end</span>
<span class="n">service</span> <span class="s2">"memcached"</span> <span class="n">do</span>
<span class="n">enabled</span> <span class="n">true</span>
<span class="n">running</span> <span class="n">true</span>
<span class="n">supports</span> <span class="p">:</span><span class="n">status</span> <span class="o">=></span> <span class="n">true</span><span class="p">,</span> <span class="p">:</span><span class="n">restart</span> <span class="o">=></span> <span class="n">true</span>
<span class="n">action</span> <span class="p">[:</span><span class="n">enable</span><span class="p">,</span> <span class="p">:</span><span class="n">start</span><span class="p">]</span>
<span class="n">end</span>
<span class="n">cookbook_file</span> <span class="s2">"/etc/memcached.conf"</span> <span class="n">do</span>
<span class="n">source</span> <span class="s2">"memcached.conf"</span>
<span class="n">mode</span> <span class="mi">0640</span>
<span class="n">owner</span> <span class="s2">"root"</span>
<span class="n">group</span> <span class="s2">"root"</span>
<span class="n">notifies</span> <span class="p">:</span><span class="n">restart</span><span class="p">,</span> <span class="n">resources</span><span class="p">(:</span><span class="n">service</span> <span class="o">=></span> <span class="s2">"memcached"</span><span class="p">)</span>
<span class="n">end</span>
</pre></div>
</div>
<p>The memcached.conf is so short, I might as well include it here:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">-</span><span class="n">d</span>
<span class="n">logfile</span> <span class="o">/</span><span class="n">var</span><span class="o">/</span><span class="n">log</span><span class="o">/</span><span class="n">memcached</span><span class="o">.</span><span class="n">log</span>
<span class="o">-</span><span class="n">m</span> <span class="mi">64</span>
<span class="o">-</span><span class="n">p</span> <span class="mi">11211</span>
<span class="o">-</span><span class="n">u</span> <span class="n">nobody</span>
<span class="o">-</span><span class="n">l</span> <span class="mf">127.0.0.1</span>
</pre></div>
</div>
<p>Memcache’s config file is pretty neat, because it’s basically just
a list of arguments to pass to the daemon when it’s started. A
little bit like a pip requirements file is just commands to pass to
pip install when it’s run.</p>
</section>
<section id="wrapping-up">
<h2>Wrapping up</h2>
<p>Now that you have these awesome new recipes, and additions to old
ones, we need to make sure they’re actually being run. Your
run_list in your node.json file should now look something like
this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="s2">"run_list"</span><span class="p">:</span> <span class="p">[</span> <span class="s2">"main::default"</span><span class="p">,</span> <span class="s2">"main::python"</span><span class="p">,</span> <span class="s2">"main::readthedocs"</span><span class="p">,</span> <span class="s2">"main::memcached"</span><span class="p">,</span> <span class="s2">"main::nginx"</span><span class="p">],</span>
</pre></div>
</div>
<p>At this point, it’s pretty neat. I can run a
<code class="docutils literal notranslate"><span class="pre">fab</span> <span class="pre">install_chef</span> <span class="pre">update</span></code>, wait about 5 minutes, and go from a
freshly paved server to a fully functioning app server.</p>
<p>Tomorrow we’ll be adding some monitoring and auxiliary niceties.
This includes setting up Munin, Celery, generating the /etc/hosts
file, and throwing in a little .bashrc magic to make the user
experience nicer.</p>
<p>There were a couple of questions yesterday about databases and
other things. My current problem is running an application server,
which is what I’ve accomplished. However, with my new-found love
affair for chef, I will definitely be making my Database/Utility
box into a chef configuration really soon. I might not write it up
in so much detail, but hopefully you’ve learned enough from this
series that I can just publish the code.</p>
</section>
</section>
Alternate title: Show the world what you’ve got.2010-11-10T18:12:00+00:00http://ericholscher.com/blog/2010/nov/11/building-django-app-server-chef-part-4/Building a Django App Server with Chef: Part 42010-11-11T15:10:00+00:00<section id="building-a-django-app-server-with-chef-part-4">
<p>Alternate title: There’s no place like home!</p>
<p>This is Part 4, the final part, of my Chef tutorial. Today we’re
talking about the odds and ends left over to make the server nice
to use. You can check out the first 3 parts of the series:</p>
<ul class="simple">
<li><p><a class="reference external" href="http://ericholscher.com/blog/2010/nov/8/building-django-app-server-chef/">Part 1: Chef Beginnings</a></p></li>
<li><p><a class="reference external" href="http://ericholscher.com/blog/2010/nov/9/building-django-app-server-chef-part-2/">Part 2: Python environment buildout</a></p></li>
<li><p><a class="reference external" href="http://ericholscher.com/blog/2010/nov/10/building-django-app-server-chef-part-3/">Part 3: Deployment</a></p></li>
</ul>
<p>Today’s code will be in the git repo under the tag
<a class="reference external" href="https://github.com/ericholscher/chef-django-example/tree/blog-post-4">blog-post-4</a>.</p>
<section id="what-we-ll-need">
<h2>What we’ll need</h2>
<p>So we have our app server up and running, and ready for traffic.
Now we just need to add some other bits around the outside for it
to be fully functioning and nice to use.</p>
<ul class="simple">
<li><p>Monitoring with <a class="reference external" href="http://munin-monitoring.org/">Munin</a></p></li>
<li><p>Background tasks with <a class="reference external" href="http://celeryproject.org/">Celery</a></p></li>
<li><p>A firewall for security</p></li>
<li><p>A /etc/hosts file for talking with other nodes</p></li>
<li><p>A .bash_profile file so that when you shell in you’ll have a
nice environment</p></li>
</ul>
<p>Let’s get started.</p>
</section>
<section id="monitoring-with-munin">
<h2>Monitoring with Munin</h2>
<p>For doing monitoring with munin, we’re going to need to learn our
final Chef concept, which is Templates. You should be pretty
familiar with them already, except they use Erb, which is a
template language that lets you embed Ruby.</p>
<p>We’re only going to be configuring the Munin node here. This
assumes that you have a munin server running on another machine
that you want to give access to monitor your new app server. These
configs depend on you putting an entry like this in your node.json,
which points at the IP of the master server:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="s2">"munin_servers"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"10.177.243.34"</span><span class="p">],</span>
</pre></div>
</div>
<p>Then here is how you would write the Recipe.</p>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/munin.rb</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>package "munin-node" do
:upgrade
end
service "munin-node" do
enabled true
running true
supports :status => true, :restart => true, :reload => true
action [:enable, :start]
end
if node.attribute?("munin_servers")
template "/etc/munin/munin-node.conf" do
source "munin-node.conf"
mode 0640
owner "root"
group "root"
variables :munin_servers => node[:munin_servers] || []
notifies :restart, resources(:service => "munin-node")
end
end
</pre></div>
</div>
<p>The template Resource here is the interesting part. We’re
surrounding it with a conditional, that makes sure that we’re
defined a ‘munin_servers’ key in our node.json. Then we’re saying
that we’re going to render the munin-node.conf file with the source
template ‘munin-node.conf’. This template will be given the extra
varibale ‘munin_servers’, which is passed in using the variables
attribute.</p>
<p>Template are placed inside the cookbook in a similar place to
files.</p>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/templates/default/munin-node.conf</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><% @munin_servers.each do |server| -%>
allow ^<%= server.to_s.gsub(/\./, '\.') %>$
<% end -%>
allow ^127\.0\.0\.1$
host *
port 4949
log_level 4
log_file /var/log/munin/munin-node.log
pid_file /var/run/munin/munin-node.pid
background 1
setsid 1
user root
group root
ignore_file ~$
ignore_file DEADJOE$
ignore_file \.bak$
ignore_file %$
ignore_file \.dpkg-(tmp|new|old|dist)$
ignore_file \.rpm(save|new)$
ignore_file \.pod$
</pre></div>
</div>
<p>The interesting part here is the iteration over the munin_servers
list. It’s just doing a simple ruby loop, and then outputting the
IP address that it contains into the format that munin’s
configuration file expects.</p>
<p><strong>Note</strong>: This data-driven template rendering is a really powerful
idiom, and one of my favorite parts about Chef. This allows you to
add a new server to your pool, and have all of your configuration
files updated automatically across all your server. This is hugely
powerful, and one of the primary wins of Configuration Management.
This will be shown to better effect in the /etc/hosts file later.</p>
</section>
<section id="installing-celery">
<h2>Installing Celery</h2>
<p>Installing celery is much akin to Gunicorn that was discussed
yesterady. The dependencies were installed from our pip
requirements file, and we just need to make it run in upstart.
We’ll be doing that with the following setup.</p>
<p>Additions to <code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/readthedocs.rb</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cookbook_file</span> <span class="s2">"/etc/init/readthedocs-celery.conf"</span> <span class="n">do</span>
<span class="n">source</span> <span class="s2">"celery.conf"</span>
<span class="n">owner</span> <span class="s2">"root"</span>
<span class="n">group</span> <span class="s2">"root"</span>
<span class="n">mode</span> <span class="mi">0644</span>
<span class="n">notifies</span> <span class="p">:</span><span class="n">restart</span><span class="p">,</span> <span class="n">resources</span><span class="p">(:</span><span class="n">service</span> <span class="o">=></span> <span class="s2">"readthedocs-celery"</span><span class="p">)</span>
<span class="n">end</span>
<span class="n">service</span> <span class="s2">"readthedocs-celery"</span> <span class="n">do</span>
<span class="n">provider</span> <span class="n">Chef</span><span class="p">::</span><span class="n">Provider</span><span class="p">::</span><span class="n">Service</span><span class="p">::</span><span class="n">Upstart</span>
<span class="n">enabled</span> <span class="n">true</span>
<span class="n">running</span> <span class="n">true</span>
<span class="n">supports</span> <span class="p">:</span><span class="n">restart</span> <span class="o">=></span> <span class="n">true</span><span class="p">,</span> <span class="p">:</span><span class="n">reload</span> <span class="o">=></span> <span class="n">true</span><span class="p">,</span> <span class="p">:</span><span class="n">status</span> <span class="o">=></span> <span class="n">true</span>
<span class="n">action</span> <span class="p">[:</span><span class="n">enable</span><span class="p">,</span> <span class="p">:</span><span class="n">start</span><span class="p">]</span>
<span class="n">end</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/files/celery.conf</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>description "Celery for ReadTheDocs"
start on runlevel [2345]
stop on runlevel [!2345]
#Send KILL after 20 seconds
kill timeout 20
script
exec sudo -i -u docs django-admin.py celeryd -f /home/docs/sites/readthedocs.org/run/celery.log -c 2 -E -B
end script
respawn
</pre></div>
</div>
<p>There isn’t anything new or interesting here. Just more of the same
as before, to get another piece of infrastructure up and running.</p>
</section>
<section id="a-ghetto-firewall-install">
<h2>A ghetto firewall install</h2>
<p>I’m a big fan of not enabling services that aren’t running as a
fundamental security practice, but having a basic firewall to make
sure that those are the only ports open isn’t a bad idea either.
I’m not a great expert, so this is probably the weakest part of my
knowledge in this series, so take it with a grain of salt.</p>
<p>My favorite firewall utility is ufw. It makes managing your
firewall really simple. Here is my super basic way to configure my
firewall, it pretty much sucks :)</p>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/security.rb</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">package</span> <span class="s2">"ufw"</span> <span class="n">do</span>
<span class="p">:</span><span class="n">upgrade</span>
<span class="n">service</span> <span class="s2">"ufw"</span> <span class="n">do</span>
<span class="n">enabled</span> <span class="n">true</span>
<span class="n">running</span> <span class="n">true</span>
<span class="n">supports</span> <span class="p">:</span><span class="n">status</span> <span class="o">=></span> <span class="n">true</span><span class="p">,</span> <span class="p">:</span><span class="n">restart</span> <span class="o">=></span> <span class="n">true</span><span class="p">,</span> <span class="p">:</span><span class="n">reload</span> <span class="o">=></span> <span class="n">true</span>
<span class="n">action</span> <span class="p">[:</span><span class="n">enable</span><span class="p">,</span> <span class="p">:</span><span class="n">start</span><span class="p">]</span>
<span class="n">end</span>
<span class="n">bash</span> <span class="s2">"Enable UFW"</span> <span class="n">do</span>
<span class="n">user</span> <span class="s2">"root"</span>
<span class="n">code</span> <span class="o"><<-</span><span class="n">EOH</span>
<span class="n">ufw</span> <span class="n">allow</span> <span class="mi">22</span> <span class="c1">#SSH</span>
<span class="n">ufw</span> <span class="n">allow</span> <span class="mi">80</span> <span class="c1">#Nginx</span>
<span class="n">ufw</span> <span class="n">allow</span> <span class="mi">4949</span> <span class="c1">#Munin</span>
<span class="n">EOH</span>
<span class="n">end</span>
</pre></div>
</div>
<p>As you can see, we’re just enabling SSH, Nginx, and Munin. If we
need to install any more packages, we’ll need to expicitly open a
port, which is usually good to remind me that I’m doing it.</p>
</section>
<section id="etc-hosts">
<h2>/etc/hosts</h2>
<p>Whenever I’m in the cloud, I find keeping track of my other servers
to be a pain. You generally want to use the internal backplane to
communicate between your servers, so I use the /etc/hosts file to
make that simple.</p>
<p>We’re going to depend on an entry in your node.json that looks
something like this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="s2">"all_servers"</span><span class="p">:</span> <span class="p">{</span><span class="s2">"Golem"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"10.177.234.234"</span><span class="p">,</span> <span class="s2">"173.203.234.234"</span><span class="p">],</span>
<span class="s2">"Chimera"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"10.177.234.234"</span><span class="p">,</span> <span class="s2">"204.232.234.234"</span><span class="p">],</span>
<span class="s2">"Hydra"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"10.177.234.234"</span><span class="p">,</span> <span class="s2">"173.203.234.234"</span><span class="p">]</span> <span class="p">}</span>
</pre></div>
</div>
<p>Which is a mapping of all your servers, with their internal and
external IPs. This will be useful to have for lots of different
recipes, and it would be nice to autogenerate this, but when you
only have a few servers it isn’t so bad.</p>
<p>The rest of out hosts configuration looks like this:</p>
<p>Addition to <code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/default.rb</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>if node.attribute?("all_servers")
template "/etc/hosts" do
source "hosts"
mode 644
variables :all_servers => node[:all_servers] || {}
end
end
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/templates/default/hosts</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mf">127.0.0.1</span> <span class="n">localhost</span> <span class="n">localhost</span><span class="o">.</span><span class="n">localdomain</span>
<span class="o"><%</span> <span class="nd">@all_servers</span><span class="o">.</span><span class="n">each_pair</span> <span class="n">do</span> <span class="o">|</span><span class="n">name</span><span class="p">,</span> <span class="n">ips</span><span class="o">|</span> <span class="o">-%></span>
<span class="o"><%=</span> <span class="n">ips</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">%></span> <span class="o"><%=</span> <span class="n">ips</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">%></span> <span class="o"><%=</span> <span class="n">name</span> <span class="o">%></span>
<span class="o"><%</span> <span class="n">end</span> <span class="o">-%></span>
</pre></div>
</div>
<p>As you can see, when we add a server to the all_servers hash, it
will propogate out to the /etc/hosts file of our app server. This
makes me really happy, and showcases some of the more advanced use
cases of Chef.</p>
</section>
<section id="customizing-your-shell">
<h2>Customizing your shell</h2>
<p>Now that we have the server all set up, it won’t be much good if it
isn’t nice to use when we shell in. So here is how I go ahead and
add in some nicities to bash for when you log in.</p>
<p>Addition to <code class="docutils literal notranslate"><span class="pre">cookbooks/main/recipes/readthedocs.rb</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cookbook_file</span> <span class="s2">"/home/docs/.bash_profile"</span> <span class="n">do</span>
<span class="n">source</span> <span class="s2">"bash_profile"</span>
<span class="n">owner</span> <span class="s2">"docs"</span>
<span class="n">group</span> <span class="s2">"docs"</span>
<span class="n">mode</span> <span class="mi">0755</span>
<span class="n">end</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">cookbooks/main/files/default/bash_profile</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>. .bashrc
export PIP_DOWNLOAD_CACHE=/tmp/pip
export DJANGO_SETTINGS_MODULE=settings
export PYTHONPATH=$PYTHONPATH:~/sites/readthedocs.org/checkouts/readthedocs.org
export EDITOR=vim
. sites/readthedocs.org/bin/activate
cd ~/sites/readthedocs.org/
alias chk='cd /home/docs/sites/readthedocs.org/checkouts/readthedocs.org'
alias run='cd /home/docs/sites/readthedocs.org/run'
</pre></div>
</div>
<p>First off, we’re sourcing the .bashrc file, so that we get all the
nice things it provides, like a colored PS1. Then we’re setting
some environment variables so that django-admin.py and pip work
nicely. Then we activate our virtualenv and switch into it’s base
directory, so we’re always starting where we want to be on login.
Then we just have a couple of aliases for easy navigation around.</p>
<p>I like how this makes the user experience of shelling into the
server a lot nicer, and makes the manual workflow that you’ll
eventually have to fiddle with really nice.</p>
</section>
<section id="conclusion">
<h2>Conclusion</h2>
<p>So that’s the end of this tutorial. I hope that it was instructive
in learning Chef, as well as providing some insights into the
deployment of a Django application. Tomorrow (or if I’m too tired,
next week), I’ll be providing some thoughts on how I think chef
treated me, and how I feel about the build out.</p>
</section>
</section>
Alternate title: There’s no place like home!2010-11-11T15:10:00+00:00http://ericholscher.com/blog/2010/nov/12/site-upgrades/Site upgrades2010-11-12T19:48:34+00:00<section id="site-upgrades">
<p>Today I went ahead and flipped the switch on a couple of server
migrations I’ve had queued up. One of these updates is moving
<a class="reference external" href="http://readthedocs.org">ReadTheDocs</a> over to its own dedicated
server, that I built up over the week in my
<a class="reference external" href="http://ericholscher.com/tag/chef-series/">Chef Tutorials</a>.</p>
<p>Over at RTD, you won’t notice too many changes, other than it
should be FASTER! I had a bunch of sites running on an underpowered
server, and now I have it set up nicely, and running on it’s own
machine, it’s chugging along great.</p>
<p>The other change is that I migrated my blog (what you’re reading!)
over to <a class="reference external" href="https://github.com/montylounge/django-mingus">Mingus</a>. I
was running an oold copy of django-basic-blog, which is what Mingus
is based off, so the migration was easy. I moved it over from my
legacy Slicehost account onto my new server infrastructure that
I’ve been building. There is also a slight refresh of the theme of
the sight, mainly the Mingus defaults poking through on top of my
old theme.</p>
<p>The other bits you might notice is that my code snippets should now
be syntax highlighted. It seems pygments gets confused sometimes,
but it’s better than it was.</p>
<p>Please report any bugs that you see on either of the sites above to
me on Twitter, or here in the comments. Thanks!</p>
</section>
Today I went ahead and flipped the switch on a couple of server
migrations I’ve had queued up. One of these updates is moving
ReadTheDocs over to its own dedicated
server, that I built up over the week in my
Chef Tutorials.2010-11-12T19:48:34+00:00http://ericholscher.com/blog/2010/nov/15/correct-commands-check-out-and-update-vcs-repos/Correct commands to check out and update VCS repos2010-11-15T18:28:14+00:00<section id="correct-commands-to-check-out-and-update-vcs-repos">
<p>In my work in <a class="reference external" href="http://readthedocs.org/">ReadTheDocs</a>, we now
support all of the major VCS repositories: svn, bzr, hg, and git.
At this point in time we’re only checking out the repos to their
default branches, and then trying to trying to update them again to
another revision. While writing this code I have had at least 3
different bugs that caused the repos not to be updated correctly.
So I’m going to detail here the exact code that allows me to do
this for each of these types of repos, hopefully so that when you
or I need to do this in the future, we can at least start from
here.</p>
<p>Let me know if any of these are wrong, because they probably are.</p>
<section id="git">
<h2>Git</h2>
<p>Checking out a repo:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">git</span> <span class="n">clone</span> <span class="o">--</span><span class="n">depth</span><span class="o">=</span><span class="mi">1</span> <span class="o"><</span><span class="n">remote_url</span><span class="o">></span> <span class="o"><</span><span class="n">local_filepath</span><span class="o">></span>
</pre></div>
</div>
<p>Updating a repo:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">git</span> <span class="o">--</span><span class="n">git</span><span class="o">-</span><span class="nb">dir</span><span class="o">=.</span><span class="n">git</span> <span class="n">fetch</span>
<span class="n">git</span> <span class="o">--</span><span class="n">git</span><span class="o">-</span><span class="nb">dir</span><span class="o">=.</span><span class="n">git</span> <span class="n">reset</span> <span class="o">--</span><span class="n">hard</span> <span class="n">origin</span><span class="o">/</span><span class="n">master</span>
</pre></div>
</div>
<p>I’m specifying the –git-dir here because my master repository is
git as well, and I don’t want to risk the git commands cascading up
and applying to the outer repository. I’m also specifying –depth=1
so that I don’t clone the entire repository, but only the latest
commit. I don’t need the history, so I’m doing this. As you can
tell, I’m more familiar with git than the other VCS systems here.</p>
</section>
<section id="svn">
<h2>Svn</h2>
<p>Checking out a repo:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">svn</span> <span class="n">checkout</span> <span class="o"><</span><span class="n">remote_url</span><span class="o">></span> <span class="o"><</span><span class="n">local_filepath</span><span class="o">></span>
</pre></div>
</div>
<p>Updating a repo:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">svn</span> <span class="n">revert</span> <span class="o">--</span><span class="n">recursive</span> <span class="o">.</span>
<span class="n">svn</span> <span class="n">up</span> <span class="o">--</span><span class="n">accept</span> <span class="n">theirs</span><span class="o">-</span><span class="n">full</span>
</pre></div>
</div>
<p>I ran into problems here where I was calling revert without
recursive and it wasn’t doing anything! You need to do this from
the top-level of the repo, and it will make sure all the state
lower in the repo is reverted.</p>
</section>
<section id="bzr">
<h2>Bzr</h2>
<p>Checking out a repo:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">bzr</span> <span class="n">checkout</span> <span class="o"><</span><span class="n">remote_url</span><span class="o">></span> <span class="o"><</span><span class="n">local_filepath</span><span class="o">></span>
</pre></div>
</div>
<p>Updating a repo:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">bzr</span> <span class="n">revert</span>
<span class="n">bzr</span> <span class="n">up</span>
</pre></div>
</div>
<p>This one is nice and easy.</p>
</section>
<section id="hg">
<h2>Hg</h2>
<p>Checking out a repo:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">hg</span> <span class="n">clone</span> <span class="o"><</span><span class="n">remote_url</span><span class="o">></span> <span class="o"><</span><span class="n">local_filepath</span><span class="o">></span>
</pre></div>
</div>
<p>Updating a repo:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">hg</span> <span class="n">pull</span>
<span class="n">hg</span> <span class="n">update</span> <span class="o">-</span><span class="n">C</span> <span class="o">.</span>
</pre></div>
</div>
<p>Again, a slightly different syntax to make sure that you’re
deleting all the files, and with the update command.</p>
<p>I hope this helps people in the future at least get to the point
where they can pull down code and update it. My next task is
figuring out how to support branching in all of the different
repositories which is going to be fun, because they have different
filesystem structures.</p>
</section>
</section>
In my work in ReadTheDocs, we now
support all of the major VCS repositories: svn, bzr, hg, and git.
At this point in time we’re only checking out the repos to their
default branches, and then trying to trying to update them again to
another revision. While writing this code I have had at least 3
different bugs that caused the repos not to be updated correctly.
So I’m going to detail here the exact code that allows me to do
this for each of these types of repos, hopefully so that when you
or I need to do this in the future, we can at least start from
here.2010-11-15T18:28:14+00:00http://ericholscher.com/blog/2010/nov/16/using-haystack-index-non-database-content/Using Haystack to index non-database content2010-11-16T21:01:00+00:00<section id="using-haystack-to-index-non-database-content">
<p>Over on ReadTheDocs, I wanted to build
<a class="reference external" href="http://readthedocs.org/search/?q=crawler">search</a> around the
documentation that we’re hosting. I chose
<a class="reference external" href="http://haystacksearch.org/">Haystack</a> and
<a class="reference external" href="http://lucene.apache.org/solr/">Solr</a> for this, because it’s the
best way to do search in Django these days. However, I’ve only ever
used Haystack to index content that is in the database. I thought
about trying to add all the rendered HTML from the documentation
into the database, but that was a non-starter.</p>
<p>I ended up adding a ImportedFile model to the database, which would
contain the metadata for the HTML file:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="ch">#!python</span>
<span class="k">class</span> <span class="nc">ImportedFile</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">project</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">ForeignKey</span><span class="p">(</span><span class="n">Project</span><span class="p">,</span> <span class="n">related_name</span><span class="o">=</span><span class="s1">'imported_files'</span><span class="p">)</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">255</span><span class="p">)</span>
<span class="n">slug</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">SlugField</span><span class="p">()</span>
<span class="n">path</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">255</span><span class="p">)</span>
<span class="n">md5</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">255</span><span class="p">)</span>
</pre></div>
</div>
<p>This allows me to link the SearchIndex in haystack to a model. Then
the interesting part is in the Haystack SearchIndex, where I
override the prepare_text method, allowing me to read the data in
from the filesystem instead of from the database.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="ch">#!python</span>
<span class="k">class</span> <span class="nc">ImportedFileIndex</span><span class="p">(</span><span class="n">SearchIndex</span><span class="p">):</span>
<span class="n">text</span> <span class="o">=</span> <span class="n">CharField</span><span class="p">(</span><span class="n">document</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">author</span> <span class="o">=</span> <span class="n">CharField</span><span class="p">(</span><span class="n">model_attr</span><span class="o">=</span><span class="s1">'project__user'</span><span class="p">)</span>
<span class="n">project</span> <span class="o">=</span> <span class="n">CharField</span><span class="p">(</span><span class="n">model_attr</span><span class="o">=</span><span class="s1">'project__name'</span><span class="p">)</span>
<span class="n">title</span> <span class="o">=</span> <span class="n">CharField</span><span class="p">(</span><span class="n">model_attr</span><span class="o">=</span><span class="s1">'name'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">prepare_text</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">):</span>
<span class="n">full_path</span> <span class="o">=</span> <span class="n">obj</span><span class="o">.</span><span class="n">project</span><span class="o">.</span><span class="n">full_html_path</span>
<span class="n">to_read</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">full_path</span><span class="p">,</span> <span class="n">obj</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">lstrip</span><span class="p">(</span><span class="s1">'/'</span><span class="p">))</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">codecs</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">to_read</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">"utf-8"</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'r'</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="k">return</span> <span class="n">content</span>
<span class="k">except</span> <span class="ne">IOError</span><span class="p">:</span>
<span class="nb">print</span> <span class="s2">"</span><span class="si">%s</span><span class="s2"> not found"</span> <span class="o">%</span> <span class="n">full_path</span>
<span class="n">site</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">ImportedFile</span><span class="p">,</span> <span class="n">ImportedFileIndex</span><span class="p">)</span>
</pre></div>
</div>
<p>This means that I don’t have to bloat my database with all my
rendered HTML, but have the full HTML stored in Solr which works
for querying.</p>
</section>
Over on ReadTheDocs, I wanted to build
search around the
documentation that we’re hosting. I chose
Haystack and
Solr for this, because it’s the
best way to do search in Django these days. However, I’ve only ever
used Haystack to index content that is in the database. I thought
about trying to add all the rendered HTML from the documentation
into the database, but that was a non-starter.2010-11-16T21:01:00+00:00http://ericholscher.com/blog/2010/nov/17/required-reading/Required Reading2010-11-17T20:38:47+00:00<section id="required-reading">
<p>At <a class="reference external" href="http://jobs.github.com/companies/Mediaphormedia">work</a>, we
have a wiki page that’s called Required Reading. Named after that
oh-so-lovely tradition in high school or college of having books
that you needed to read over the summer before you started class.
The idea being that they are relics of the culture of the company,
and if you read everything, you will understand a lot about how
work (and play) is done.</p>
<p>I think this is something useful that most companies should have.
It was basically split into two parts: Silly and Serious. The silly
parts are so that you can understand all of the inside jokes and
references that people are bound to make. The serious parts are
more philosophy and thoughts behind how we write code and do
things.</p>
<section id="silly">
<h2>Silly</h2>
<p>Being a python shop, this
<a class="reference external" href="http://www.youtube.com/view_play_list?p=CDFEA6D52E5CC0EC">Youtube Monty Python playlist</a>
is a must watch. It’s got most of the famous Python gags, and being
knowledgeable about the languages namesake makes you a more rounded
human being. Speaking of namesakes, Django Reinhardt is a great
jazz player, so I recommend you
<a class="reference external" href="http://djangopedia.com/wiki/index.php?title=Main_Page">learn about</a>
him as well. He was quite the bad ass.</p>
<p>After that, there are just some of the classic online videos that
everyone should watch:</p>
<ul class="simple">
<li><p><a class="reference external" href="http://www.youtube.com/watch?v=PLUS00QrYWw">DJ Ango</a></p></li>
<li><p>Bill O’Reily
<a class="reference external" href="http://youtube.com/watch?v=5j2YDq6FkVE&NR=1">DO IT LIVE (remix)</a>
for when you’re doing it live.</p></li>
<li><p>Nerf guns <a class="reference external" href="www.vimeo.com/2830418">gone wild</a>.
<a class="reference external" href="http://flickr.com/photos/webology/3023204926/">Again</a>.</p></li>
<li><p><a class="reference external" href="http://youtube.com/watch?v=izibSMAQhEY">SCHNAPPI</a>. Cutest
Crocodile Ever.</p></li>
</ul>
<p>and such.</p>
</section>
<section id="serious">
<h2>Serious</h2>
<p>The serious posts are a little more relevant, because they are more
about the philosophy of how to code. I have been looking through a
lot of really great posts on this subject lately, and these are
some of my favorites:</p>
<ul class="simple">
<li><p><a class="reference external" href="http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html">Code like a Pythonista</a>
(which links to pep8 and pep20)</p></li>
<li><p><a class="reference external" href="http://docs.pylonshq.com/community/testing.html">Pylons Unit Testing Guidelines</a></p></li>
<li><p><a class="reference external" href="http://www.faqs.org/docs/artu/">The Art of Unix Programming</a></p></li>
</ul>
<p>Then there are some of the more topical guides that give a good
knowledge and understanding of some of the fundamental things that
you do at work. The unit testing guide above is a good example, but
there are a few as well:</p>
<ul class="simple">
<li><p><a class="reference external" href="http://martinfowler.com/articles/richardsonMaturityModel.html">Richardson Maturity Model (for REST)</a></p></li>
<li><p><a class="reference external" href="http://www.mnot.net/cache_docs/">HTTP Caching</a></p></li>
<li><p><a class="reference external" href="http://blip.tv/file/2232410">Understanding the GIL</a> - Talk by
David Beazley</p></li>
<li><p><a class="reference external" href="http://prodjango.com/">Pro Django</a> ($$)</p></li>
</ul>
<p>I think that having a guide to the culture of the company is really
useful for people that are getting started. Plus I think it’s a
good way to remind people of what the values are of your team.
Hopefully silliness is valued as much as good work, and you have a
place to go when you want to see silly bits of the past.</p>
<p>I’d love to hear if other people have other ways of introducing
culture to their companies. I’d also love to see some other really
good topical guides on things that you may love, as I’m sure there
are tons more out there.</p>
</section>
</section>
At work, we
have a wiki page that’s called Required Reading. Named after that
oh-so-lovely tradition in high school or college of having books
that you needed to read over the summer before you started class.
The idea being that they are relics of the culture of the company,
and if you read everything, you will understand a lot about how
work (and play) is done.2010-11-17T20:38:47+00:00