Cloud and AutomationCurrent Focus: Chef and OpenStack for your Private Cloudhttp://ehaselwanter.com2015-04-19T00:00:00ZEdmund HaselwanterChef Fundamentals Training Nürnberghttp://ehaselwanter.com/en/blog/2015/04/19/chef-fundamentals-training-nurenberg/2015-04-19T00:00:00Z2015-10-04T13:07:50+02:00Edmund Haselwanter<p>Chef Fundamentals back in Nürnberg.</p>
<p>A lot of interesting discussions and deep dives.</p>
<h2>Feedback Results:</h2>
<table><thead>
<tr>
<th>Avg</th>
<th>Min</th>
<th>Max</th>
<th></th>
</tr>
</thead><tbody>
<tr>
<td>4,55</td>
<td>4</td>
<td>5</td>
<td>„course adressing the topics announced“</td>
</tr>
<tr>
<td>4,60</td>
<td>4</td>
<td>5</td>
<td>„pre-requisites appropriate“</td>
</tr>
<tr>
<td>4,36</td>
<td>3</td>
<td>5</td>
<td>„do you believe you will benefit“</td>
</tr>
<tr>
<td>4,45</td>
<td>3</td>
<td>5</td>
<td>„how did you like the course overall“</td>
</tr>
<tr>
<td>5,00</td>
<td>5</td>
<td>5</td>
<td>„instructor competent and knowledgable“</td>
</tr>
<tr>
<td>5,00</td>
<td>5</td>
<td>5</td>
<td>„Instructor friendly/open to questions</td>
</tr>
<tr>
<td>4,45</td>
<td>4</td>
<td>5</td>
<td>„instructor taught understandable“</td>
</tr>
<tr>
<td>4,27</td>
<td>4</td>
<td>5</td>
<td>„was this a good training event“</td>
</tr>
<tr>
<td>4,55</td>
<td>4</td>
<td>5</td>
<td>„book another course?“</td>
</tr>
<tr>
<td>4,82</td>
<td>4</td>
<td>5</td>
<td>„recommend training“</td>
</tr>
</tbody></table>
<p><img class="center" data-thumbnails="original:article-images/Chef-Fundamentals-Nurenberg-2015-1.png|small:article-images/Chef-Fundamentals-Nurenberg-2015-1-small-200x.png|half:article-images/Chef-Fundamentals-Nurenberg-2015-1-half-390x.png|blog:article-images/Chef-Fundamentals-Nurenberg-2015-1-blog-780x.png" src="/images/article-images/Chef-Fundamentals-Nurenberg-2015-1-half-390x.png" /></p>
<p>If you ever want a training class organized in the DACH region (or beyond ;), don’t hesitate to reach <a href="/en/chef-trainings">out to me :-)</a></p>
Chef Fundamentals Training Berlinhttp://ehaselwanter.com/en/blog/2015/02/22/chef-fundamentals-training-berlin/2015-02-22T00:00:00Z2015-10-04T13:07:50+02:00Edmund Haselwanter<p>Chef Fundamentals back in Berlin.</p>
<p>A lot of interesting discussions and deep dives.</p>
<h2>Feedback Results:</h2>
<table><thead>
<tr>
<th>Avg</th>
<th>Min</th>
<th>Max</th>
<th></th>
</tr>
</thead><tbody>
<tr>
<td>4,78</td>
<td>4</td>
<td>5</td>
<td>„course adressing the topics announced“</td>
</tr>
<tr>
<td>4,67</td>
<td>4</td>
<td>5</td>
<td>„pre-requisites appropriate“</td>
</tr>
<tr>
<td>4,78</td>
<td>4</td>
<td>5</td>
<td>„do you believe you will benefit“</td>
</tr>
<tr>
<td>4,67</td>
<td>4</td>
<td>5</td>
<td>„how did you like the course overall“</td>
</tr>
<tr>
<td>4,89</td>
<td>4</td>
<td>5</td>
<td>„instructor competent and knowledgable“</td>
</tr>
<tr>
<td>5,00</td>
<td>5</td>
<td>5</td>
<td>„Instructor friendly/open to questions</td>
</tr>
<tr>
<td>4,67</td>
<td>3</td>
<td>5</td>
<td>„instructor taught understandable“</td>
</tr>
<tr>
<td>4,56</td>
<td>4</td>
<td>5</td>
<td>„was this a good training event“</td>
</tr>
<tr>
<td>4,56</td>
<td>4</td>
<td>5</td>
<td>„book another course?“</td>
</tr>
<tr>
<td>4,89</td>
<td>4</td>
<td>5</td>
<td>„recommend training“</td>
</tr>
</tbody></table>
<p><img class="center" data-thumbnails="original:article-images/Chef-Fundamentals-Innsbruck-2015-1.png|small:article-images/Chef-Fundamentals-Innsbruck-2015-1-small-200x.png|half:article-images/Chef-Fundamentals-Innsbruck-2015-1-half-390x.png|blog:article-images/Chef-Fundamentals-Innsbruck-2015-1-blog-780x.png" src="/images/article-images/Chef-Fundamentals-Innsbruck-2015-1-half-390x.png" /></p>
<p>If you ever want a training class organized in the DACH region (or beyond ;), don’t hesitate to reach <a href="/en/chef-trainings">out to me :-)</a></p>
Chef Fundamentals Training Innsbruckhttp://ehaselwanter.com/en/blog/2015/02/12/chef-fundamentals-training-innsbruck/2015-02-12T00:00:00Z2015-10-04T13:07:50+02:00Edmund Haselwanter<p>Chef Fundamentals on the road again, this time clode to my roots in Innsbruck in Tyrol.</p>
<p>Quite a new situation with a lot of questions on how to run chef-client as a non-privileged user.</p>
<h2>Feedback Results:</h2>
<table><thead>
<tr>
<th>Avg</th>
<th>Min</th>
<th>Max</th>
<th></th>
</tr>
</thead><tbody>
<tr>
<td>4,78</td>
<td>4</td>
<td>5</td>
<td>„course adressing the topics announced“</td>
</tr>
<tr>
<td>4,33</td>
<td>2</td>
<td>5</td>
<td>„pre-requisites appropriate“</td>
</tr>
<tr>
<td>4,67</td>
<td>4</td>
<td>5</td>
<td>„do you believe you will benefit“</td>
</tr>
<tr>
<td>4,44</td>
<td>4</td>
<td>5</td>
<td>„how did you like the course overall“</td>
</tr>
<tr>
<td>4,89</td>
<td>4</td>
<td>5</td>
<td>„instructor competent and knowledgable“</td>
</tr>
<tr>
<td>5,00</td>
<td>5</td>
<td>5</td>
<td>„Instructor friendly/open to questions</td>
</tr>
<tr>
<td>4,67</td>
<td>4</td>
<td>5</td>
<td>„instructor taught understandable“</td>
</tr>
<tr>
<td>4,56</td>
<td>4</td>
<td>5</td>
<td>„was this a good training event“</td>
</tr>
<tr>
<td>4,56</td>
<td>4</td>
<td>5</td>
<td>„book another course?“</td>
</tr>
<tr>
<td>4,67</td>
<td>4</td>
<td>5</td>
<td>„recommend training“</td>
</tr>
</tbody></table>
<p><img class="center" data-thumbnails="original:article-images/Chef-Fundamentals-Berlin-2015-1.png|small:article-images/Chef-Fundamentals-Berlin-2015-1-small-200x.png|half:article-images/Chef-Fundamentals-Berlin-2015-1-half-390x.png|blog:article-images/Chef-Fundamentals-Berlin-2015-1-blog-780x.png" src="/images/article-images/Chef-Fundamentals-Berlin-2015-1-half-390x.png" /></p>
<p>If you ever want a training class organized in the DACH region (or beyond ;), don’t hesitate to reach <a href="/en/chef-trainings">out to me :-)</a></p>
MTU Issue?. Nope, It is LRO with Bridge and Bondhttp://ehaselwanter.com/en/blog/2014/11/02/mtu-issue--nope-it-is-lro-with-bridge-and-bond/2014-11-02T00:00:00Z2014-11-06T12:06:06+01:00Edmund Haselwanter<p>This one bugged me for a while. Cause it was so miss-leading to debug. Most of the time when faced with connection loss on larger packages one immediately thinks: Damn it, bitten again by PMTU, have to fix the MTU all the way in and out. Recently I got the exact same behaviour on a OpenStack cluster with VLAN provider networking on 10GbE bonds (port channels/lacp/bond4).</p>
<h2>Debug Symptom:</h2>
<p>On a node accessible from the target create two files:</p>
<pre class="highlight shell">head -c 1600 /dev/urandom > 1600.txt
head -c 500 /dev/urandom > 500.txt
</pre>
<p>1600 bytes is big enough to cause problems with fragmentation. (default MTU is 1500 almost everywhere), increase the file size if you have jumbo frames enabled on your path.</p>
<p>Then e.g. scp the packet.</p>
<p>you will see, that copying the small packet will succeed</p>
<pre class="highlight shell">scp admin@10.10.91.1:500.txt .
admin@10.10.91.1<span class="s1">'s password:
500.txt 100% 500 0.5KB/s 00:00
</span></pre>
<p>but copying the large file will stall</p>
<pre class="highlight plaintext">scp admin@10.10.91.1:1600.txt .
admin@10.10.91.1's password:
1600.txt 0% 0 0.0KB/s - stalled -^C
</pre>
<p>This is how a MTU problem reveals itself.</p>
<p>Checked everything, played with various fixes that helped in the past.</p>
<p>Things like:</p>
<ul>
<li>http://lartc.org/howto/lartc.cookbook.mtu-mss.html</li>
</ul>
<pre class="highlight shell">iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
</pre>
<p>does not help</p>
<ul>
<li><a href="http://openstack.sys-con.com/node/2938779">http://openstack.sys-con.com/node/2938779</a></li>
<li><a href="http://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/13725-56.html">http://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/13725-56.html</a></li>
<li><a href="http://packetlife.net/blog/2008/aug/18/path-mtu-discovery/">http://packetlife.net/blog/2008/aug/18/path-mtu-discovery/</a></li>
<li><a href="http://mccltd.net/blog/?p=1577">http://mccltd.net/blog/?p=1577</a></li>
<li><a href="http://roie9876.wordpress.com/2014/04/29/nsx-minimum-mtu/">http://roie9876.wordpress.com/2014/04/29/nsx-minimum-mtu/</a></li>
</ul>
<h2>A Hint from a Friend - LRO Findings</h2>
<p>Then I got a hint from a friend (<a href="https://www.linkedin.com/pub/thore-bahr/72/b99/7a7">Thore Bahr</a>) to have a look into <code>rx-vlan-offload</code>. I did so, but that did not help. </p>
<p><strong>But</strong>, this triggered further investigation and more deep digging into the root cause. </p>
<p>Finally it revealed itself derived from some other observations:</p>
<ul>
<li><a href="https://bugzilla.redhat.com/show_bug.cgi?id=772317">https://bugzilla.redhat.com/show_bug.cgi?id=772317</a></li>
<li><a href="http://www.novell.com/support/kb/doc.php?id=7013089">http://www.novell.com/support/kb/doc.php?id=7013089</a></li>
</ul>
<p>=> So there is an issue with the lro setting and bonding with Intel igbxe adapters. We have to turn off lro</p>
<p>From the [Base Driver for the Intel® Ethernet 10 Gigabit PCI Express Family of Adapters README](http://downloadmirror.intel.com/22919/eng/README.txt:</p>
<blockquote>
<h1>IMPORTANT NOTE</h1>
<p>WARNING: The ixgbe driver compiles by default with the LRO (Large Receive
Offload) feature enabled. This option offers the lowest CPU utilization for
receives, but is completely incompatible with <em>routing/ip forwarding</em> and
<em>bridging</em>. If enabling ip forwarding or bridging is a requirement, it is
necessary to disable LRO using compile time options as noted in the LRO
section later in this document. The result of not disabling LRO when combined
with ip forwarding or bridging can be low throughput or even a kernel panic.</p>
</blockquote>
<h2>Change Offload Settings with <code>ethtool</code>:</h2>
<p>First try to get a connection to proof it is not working</p>
<pre class="highlight shell"><span class="gp">$ </span>ip netns <span class="nb">exec </span>qdhcp-9d444bee-0395-47d9-ae7e-ae315c25e088 ssh 50.0.0.9
</pre>
<p>Change the settings with <code>ethtool -K <adapter> lro off</code></p>
<pre class="highlight plaintext">$ ethtool -K p3p1 lro off
$ ethtool -K p3p2 lro off
$ ethtool -k p3p1
Offload parameters for eth6:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
</pre>
<p>This was NOT working, now it works:</p>
<pre class="highlight shell"><span class="gp">$ </span>ip netns <span class="nb">exec </span>qdhcp-9d444bee-0395-47d9-ae7e-ae315c25e088 ssh 50.0.0.9 <span class="s1">'uptime'</span>
Warning: Permanently added <span class="s1">'50.0.0.9'</span> <span class="o">(</span>ECDSA<span class="o">)</span> to the list of known hosts.
17:34pm up 10 days 3:20, 0 users, load average: 0.00, 0.01, 0.05
</pre>
<h3>Make it Permanent!</h3>
<p><a href="http://www.novell.com/support/kb/doc.php?id=7013089">http://www.novell.com/support/kb/doc.php?id=7013089</a> suggests adding the following options to the network config: </p>
<pre class="highlight plaintext">ETHTOOL_OPTIONS='-K iface lro off'
</pre>
<p>We have to add this to the automation for all interfaces in a/the bond.</p>
<p>Finally since we use Chef we can push down this setting to all compute nodes:</p>
<pre class="highlight shell">knife ssh roles:<span class="k">*</span>comp<span class="k">*</span> -- ethtool -K p3p1 lro off
knife ssh roles:<span class="k">*</span>comp<span class="k">*</span> -- ethtool -k p3p1 |grep large
df0-xx-xx-xx-aa-aa.test-openstack.org large-receive-offload: off
df0-xx-xx-xx-aa-aa.test-openstack.org large-receive-offload: off
<span class="o">[</span> .. snipped .. <span class="o">]</span>
df0-xx-xx-xx-aa-aa.test-openstack.org large-receive-offload: off
</pre>Deploying Openstack with Stackforge Chef-Zero Stylehttp://ehaselwanter.com/en/blog/2014/10/15/deploying-openstack-with-stackforge-chef-zero-style/2014-10-15T00:00:00Z2015-10-04T13:07:50+02:00Edmund Haselwanter<h2>OpenStack Lifecycle Management Tools</h2>
<p>In a joint effort <a href="https://twitter.com/jannis_r">Jannis Rake-Revelant</a>, <a href="https://twitter.com/jaybrueder">Jürgen Brüder</a>, and myself <a href="https://twitter.com/ehaselwanter">Edmund Haselwanter</a> had a look at several what we call “Openstack Lifecycle Management tools”.</p>
<p>This time <a href="https://twitter.com/jaybrueder">Jürgen Brüder</a> did most of the work, so thanks for sharing your findings :-)</p>
<h1>Deploying Openstack with Stackforge</h1>
<p>Stackforge provides an open-source repository to setup Openstack with Chef.
You can deploy it with a Chef-Server or simply by using Chef-Zero.</p>
<p>It currently includes all OpenStack core projects: Compute, Dashboard, Identity, Image, Network, Object Storage, Block Storage, Telemetry and Orchestration.</p>
<p>Stackforge also comes with a couple of Vagrantfiles that can be used to create a multi-node test deployment from scratch.
This way, you also won’t need a Chef Server for deployment.</p>
<h2>Documentation and Tutorials</h2>
<p>The following Git repository contains good documentation about using Stackforge:
<a href="https://github.com/stackforge/openstack-chef-repo/tree/stable/icehouse">https://github.com/stackforge/openstack-chef-repo/tree/stable/icehouse</a></p>
<p>It also contains example configuration files for environments and roles.</p>
<h2>Using Vagrant for Test setup</h2>
<p>Vagrant can be installed on nearly all operating systems.
We will be using Mac OS X for this example.</p>
<p>If you are on Mac OS X, you’ll need to install Xcode Command Line Tools from
<a href="https://developer.apple.com/downloads/">https://developer.apple.com/downloads/</a></p>
<h3>Installing ChefDK</h3>
<p>To ensure a proper working deployment, we recommend using the ChefDK for installing all needed Gem dependencies.
This will also install Berkshelf etc.</p>
<p>Just follow this link a download the version that fits your OS. Then install it:
<a href="http://downloads.getchef.com/chef-dk/">http://downloads.getchef.com/chef-dk/</a></p>
<h3>Installing VirtualBox and Vagrant</h3>
<p>Install the latest VirtualBox for your operating system:
<a href="https://www.virtualbox.org/wiki/Downloads">https://www.virtualbox.org/wiki/Downloads</a></p>
<p>Then install the latest version of Vagrant:
<a href="https://www.vagrantup.com/downloads">https://www.vagrantup.com/downloads</a></p>
<p>To make the Vagrantfiles work, we need three additonal plugins for Vagrant.
Open a terminal window and run these commands:</p>
<p>Now install the Omnibus, Chef-Zero and Berkshelf Vagrant plugin like this (order!)</p>
<pre class="highlight shell">vagrant plugin install vagrant-berkshelf
vagrant plugin install vagrant-chef-zero
vagrant plugin install vagrant-omnibus
</pre>
<p>Please make sure to stick to the installation-order as listed above.
Some plugins can have issues if installed in the wrong order.</p>
<p>Check the three plugins are really installed</p>
<pre class="highlight shell">vagrant plugin list
</pre>
<p>Here is what we got back:</p>
<pre class="highlight shell">vagrant-berkshelf <span class="o">(</span>3.0.1<span class="o">)</span>
vagrant-chef-zero <span class="o">(</span>0.7.1<span class="o">)</span>
vagrant-login <span class="o">(</span>1.0.1, system<span class="o">)</span>
vagrant-omnibus <span class="o">(</span>1.4.1<span class="o">)</span>
vagrant-share <span class="o">(</span>1.1.2, system<span class="o">)</span>
</pre>
<h2>Deploying the Stackforge Cookbook</h2>
<p>Clone the Stackforge openstack-chef-repo into your home directory:</p>
<pre class="highlight shell">git clone -b stable/icehouse https://github.com/stackforge/openstack-chef-repo
</pre>
<p>Navigate into the created directory and rename the <strong>Vagrantfile-multi-neutron</strong> to <strong>Vagrantfile</strong></p>
<pre class="highlight shell">mv Vagrantfile-multi-neutron Vagrantfile
</pre>
<p>Now install all gems and cookbooks that are needed</p>
<pre class="highlight shell">bundle install
berks install
</pre>
<p>Now you can simply run this command to let Vagrant provision two VMs for you:</p>
<pre class="highlight shell">vagrant up /ubuntu1204/
</pre>
<p>One will be an all-in-one node, the second one will be an additional compute node.</p>
<h3>Vagrant Troubleshooting</h3>
<p>If you run into any problems with Vagrant try setting:</p>
<pre class="highlight shell"><span class="nb">export </span><span class="nv">VAGRANT_LOG</span><span class="o">=</span>debug
</pre>
<p>Also make sure that the Vagrant Plugins are compatible to each other. They are developed on independent release schedules and a new version of a plugin might not work with an older version of the other plugins.</p>
<h2>Testing the Openstack installation</h2>
<p>You can now login at the URL of the first VM. Use the username <strong>admin</strong> and the password <strong>admin</strong> for this.
If you navigate to <strong>Admin -> System Panel -> Host Aggregates</strong> you should see both nodes listed.</p>
<p><img class="center" data-thumbnails="original:article-images/host_aggregates.png|small:article-images/host_aggregates-small-200x.png|half:article-images/host_aggregates-half-390x.png|blog:article-images/host_aggregates-blog-780x.png" src="/images/article-images/host_aggregates-blog-780x.png" /> </p>
<h2>Analysing Vagrant Setup</h2>
<p>To be able to create a production ready bare-metal deployment without Vagrant, we need to understand what Vagrant is doing to deploy Openstack on VMs.</p>
<h3>Machine and Network setup</h3>
<p>Vagrant uses two VMs with each having 2 CPUs and 2048MB memory. It also adds two promiscuous interfaces to each VM and allows all frames through.</p>
<p>Each machine will need access to two private networks. Vagrant is configuring this for each VM.
The additional Compute node will have only these two networks configured:</p>
<pre class="highlight shell"><span class="c"># Vagrantfile excerpt</span>
ubuntu1204comp1.vm.network <span class="s2">"private_network"</span>, ip: <span class="s2">"192.168.3.61"</span>
ubuntu1204comp1.vm.network <span class="s2">"private_network"</span>, ip: <span class="s2">"172.16.10.61"</span>
</pre>
<p>Additionally, the Controller/Compute node will have three ports forwarded. Here the configuration of the Controller/Compute node:</p>
<pre class="highlight shell"><span class="c"># Vagrantfile excerpt</span>
ubuntu1204cont.vm.network <span class="s2">"forwarded_port"</span>, guest: 443, host: 8443 <span class="c"># dashboard-ssl</span>
ubuntu1204cont.vm.network <span class="s2">"forwarded_port"</span>, guest: 8773, host: 8773 <span class="c"># compute-ec2-api</span>
ubuntu1204cont.vm.network <span class="s2">"forwarded_port"</span>, guest: 8774, host: 8774 <span class="c"># compute-api</span>
ubuntu1204cont.vm.network <span class="s2">"private_network"</span>, ip: <span class="s2">"192.168.3.60"</span>
ubuntu1204cont.vm.network <span class="s2">"private_network"</span>, ip: <span class="s2">"172.16.10.60"</span>
</pre>
<h3>Chef-Zero and prerequisites</h3>
<p>Vagrant is using Chef-Zero to setup the deployment. This means, that there is no actual Chef-Server necessary. This is a good approach for small deployments. It will also install all needed Chef dependencies (e.g. Berkshelf) on the VM with an omnibus installer.</p>
<p>During its run it will upload all needed cookbooks, all Openstack projects (Nova, Swift etc.) are available as a Chef cookbook, to Chef-Zero. On a bare machine you would run:</p>
<pre class="highlight shell"><span class="c"># Install Chef-Omnibus</span>
curl -L https://www.opscode.com/chef/install.sh | bash
<span class="c"># Checkout Stackforge repo</span>
git clone -b stable/icehouse https://github.com/stackforge/openstack-chef-repo
<span class="nb">cd </span>openstack-chef-repo
<span class="c"># Installing Berkshelf gem and cookbooks</span>
/opt/chef/embedded/bin/gem install berkshelf --no-ri --no-rdoc
/opt/chef/embedded/bin/berks vendor ./cookbooks
</pre>
<h3>Create a Chef environment</h3>
<p>The environment provides an overall configuration for our deployment. It can tell each VM where to look for specific services, which network interface to use for what etc. Stackforge comes with a number of predefined environments. For a production deployment we recommend to write your own.</p>
<p>We will modify one of the provided environments to fit our scenario better. Here you can see the environment that we will be using:</p>
<pre class="highlight shell"><span class="o">{</span>
<span class="s2">"name"</span>: <span class="s2">"vagrant-multi-neutron"</span>,
<span class="s2">"description"</span>: <span class="s2">"Environment used in testing the upstream cookbooks and reference Chef repository with vagrant. To be used with the Vagrantfile-multi-neutron vagrantfile. Defines the necessary attributes for a working mutltinode (1 controller/n computes) openstack deployment, using neutron (with gre tunnels between hosts) for the networking component."</span>,
<span class="s2">"cookbook_versions"</span>: <span class="o">{}</span>,
<span class="s2">"json_class"</span>: <span class="s2">"Chef::Environment"</span>,
<span class="s2">"chef_type"</span>: <span class="s2">"environment"</span>,
<span class="s2">"default_attributes"</span>: <span class="o">{}</span>,
<span class="s2">"override_attributes"</span>: <span class="o">{</span>
<span class="s2">"mysql"</span>: <span class="o">{</span>
<span class="s2">"allow_remote_root"</span>: <span class="nb">true</span>,
<span class="s2">"root_network_acl"</span>: <span class="o">[</span><span class="s2">"%"</span><span class="o">]</span>
<span class="o">}</span>,
<span class="s2">"openstack"</span>: <span class="o">{</span>
<span class="s2">"developer_mode"</span>: <span class="nb">true</span>,
<span class="s2">"identity"</span>: <span class="o">{</span>
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"endpoints"</span>: <span class="o">{</span>
<span class="s2">"host"</span>: <span class="s2">"192.168.3.60"</span>,
<span class="s2">"mq"</span>: <span class="o">{</span>
<span class="s2">"host"</span>: <span class="s2">"192.168.3.60"</span>,
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"db"</span>: <span class="o">{</span>
<span class="s2">"host"</span>: <span class="s2">"192.168.3.60"</span>,
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"network"</span>: <span class="o">{</span>
<span class="s2">"debug"</span>: <span class="s2">"True"</span>,
<span class="s2">"dhcp"</span>: <span class="o">{</span>
<span class="s2">"enable_isolated_metadata"</span>: <span class="s2">"True"</span>
<span class="o">}</span>,
<span class="s2">"metadata"</span>: <span class="o">{</span>
<span class="s2">"nova_metadata_ip"</span>: <span class="s2">"192.168.3.60"</span>
<span class="o">}</span>,
<span class="s2">"openvswitch"</span>: <span class="o">{</span>
<span class="s2">"tunnel_id_ranges"</span>: <span class="s2">"1:1000"</span>,
<span class="s2">"enable_tunneling"</span>: <span class="s2">"True"</span>,
<span class="s2">"tenant_network_type"</span>: <span class="s2">"gre"</span>,
<span class="s2">"local_ip_interface"</span>: <span class="s2">"eth2"</span>
<span class="o">}</span>,
<span class="s2">"api"</span>: <span class="o">{</span>
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>
<span class="o">}</span>,
<span class="s2">"image"</span>: <span class="o">{</span>
<span class="s2">"api"</span>: <span class="o">{</span>
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"registry"</span>: <span class="o">{</span>
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"image_upload"</span>: <span class="nb">true</span>,
<span class="s2">"upload_images"</span>: <span class="o">[</span>
<span class="s2">"cirros"</span>,
<span class="s2">"ubuntu"</span>
<span class="o">]</span>,
<span class="s2">"upload_image"</span>: <span class="o">{</span>
<span class="s2">"ubuntu"</span>: <span class="s2">"http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img"</span>,
<span class="s2">"cirros"</span>: <span class="s2">"https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img"</span>
<span class="o">}</span>
<span class="o">}</span>,
<span class="s2">"compute"</span>: <span class="o">{</span>
<span class="s2">"xvpvnc_proxy"</span>: <span class="o">{</span>
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"novnc_proxy"</span>: <span class="o">{</span>
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"libvirt"</span>: <span class="o">{</span>
<span class="s2">"virt_type"</span>: <span class="s2">"qemu"</span>
<span class="o">}</span>,
<span class="s2">"network"</span>: <span class="o">{</span>
<span class="s2">"public_interface"</span>: <span class="s2">"eth1"</span>,
<span class="s2">"service_type"</span>: <span class="s2">"neutron"</span>
<span class="o">}</span>,
<span class="s2">"config"</span>: <span class="o">{</span>
<span class="s2">"ram_allocation_ratio"</span>: 5
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</pre>
<p>You can create your the environment file under <strong>/openstack-chef-repo/environments/</strong>. The filename will needs to be <strong>vagrant-multi-neutron.json</strong>.</p>
<h3>Define a run_list</h3>
<p>Each machine gets it’s own run<em>list. The run</em>list defines which <strong>roles</strong> or <strong>recipes</strong> are being used on that machine.</p>
<p>Here a listing of the run_list of each machine:</p>
<pre class="highlight shell"><span class="c"># Controller/Compute node</span>
role[os-compute-single-controller-no-network]
recipe[openstack-network::identity_registration]
role[os-network-openvswitch]
role[os-network-dhcp-agent]
role[os-network-metadata-agent]
role[os-network-server]
<span class="c"># Compute node</span>
role[os-compute-worker]
recipe[apt::cacher-client]
</pre>
<p>If you would want to add a role to a machine manually, you can do this with this command:</p>
<pre class="highlight shell">knife node run_list add NODE_NAME -z <span class="s1">'role[NAME_OF_ROLE]'</span>
</pre>
<h3>Run the Chef-Client</h3>
<p>After everything is configured, Vagrant runs the Chef-Client on each VM. This will install everything that is needed and will create running deployment.
This would be an equivalent command to run this manually:</p>
<pre class="highlight shell"><span class="c"># Assuming you have a my-deployment.json inside the environments directory</span>
chef-client -z -E my-deployment
</pre>
<h3>A quick rundown</h3>
<ol>
<li>Setup networking for your machines</li>
<li>Install the Chef-Omnibus-Installer on your machines</li>
<li>Install all needed cookbooks with Berkshelf</li>
<li>Create a Chef environment</li>
<li>Define the run_list for each machine</li>
<li>Run Chef-Client on each machine</li>
</ol>
Deploying Openstack with Stackforge Chef-Server Stylehttp://ehaselwanter.com/en/blog/2014/10/15/deploying-openstack-with-stackforge-chef-server-style/2014-10-15T00:00:00Z2015-10-04T13:07:50+02:00Edmund Haselwanter<h2>OpenStack Lifecycle Management Tools</h2>
<p>In a joint effort <a href="https://twitter.com/jannis_r">Jannis Rake-Revelant</a>, <a href="https://twitter.com/jaybrueder">Jürgen Brüder</a>, and myself <a href="https://twitter.com/ehaselwanter">Edmund Haselwanter</a> had a look at several what we call “Openstack Lifecycle Management tools”.</p>
<p>This time <a href="https://twitter.com/jaybrueder">Jürgen Brüder</a> did most of the work, so thanks for sharing your findings :-)</p>
<h1>Deploying Openstack with Stackforge and Chef Server</h1>
<p>Stackforge provides an open-source repository to setup Openstack with Chef.
You can deploy it with a Chef-Server or simply by using Chef-Zero.</p>
<p>It currently includes all OpenStack core projects: Compute, Dashboard, Identity, Image, Network, Object Storage, Block Storage, Telemetry and Orchestration.</p>
<p>In this documents we will be using Chef Server to setup our deployment. We will be using Vagrant to create three virtual machines for us.</p>
<h2>Using Vagrant for Test setup</h2>
<p>Vagrant can be installed on nearly all operating systems.
We will be using Mac OS X for this example.</p>
<p>If you are on Mac OS X, you’ll need to install Xcode Command Line Tools from
<a href="https://developer.apple.com/downloads/">https://developer.apple.com/downloads/</a></p>
<h3>Installing ChefDK</h3>
<p>To ensure a proper working deployment, we recommend using the ChefDK for installing all needed Gem dependencies.
This will also install Berkshelf etc.</p>
<p>Just follow this link a download the version that fits your OS. Then install it:
<a href="http://downloads.getchef.com/chef-dk/">http://downloads.getchef.com/chef-dk/</a></p>
<h3>Installing VirtualBox and Vagrant</h3>
<p>Install the latest VirtualBox for your operating system:
<a href="https://www.virtualbox.org/wiki/Downloads">https://www.virtualbox.org/wiki/Downloads</a></p>
<p>Then install the latest version of Vagrant:
<a href="https://www.vagrantup.com/downloads">https://www.vagrantup.com/downloads</a></p>
<h2>Create the VMs</h2>
<p>Create a directory called <strong>stackforge-chef-server</strong> in your home directory. Inside of it, create a file called <strong>Vagrantfile</strong>.</p>
<p>Add this content to the Vagrantfile:</p>
<pre class="highlight plaintext">Vagrant.require_version ">= 1.1"
Vagrant.configure("2") do |config|
# get local ip so that we can force chef zero onto a different port per
# machine, allowing for multiple simultaneous vagrant up runs
local_ip = Socket.ip_address_list.detect{|intf| intf.ipv4_private?}.ip_address
# virtualbox provider settings
config.vm.provider "virtualbox" do |vb|
vb.customize ["modifyvm", :id, "--cpus", 2]
vb.customize ["modifyvm", :id, "--memory", 2048]
vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
vb.customize ["modifyvm", :id, "--nicpromisc3", "allow-all"]
end
#################################
# Ubuntu 12.04 controller #
#################################
config.vm.define :ubuntu1204cont do |ubuntu1204cont|
ubuntu1204cont.vm.hostname = "ubuntu1204cont"
ubuntu1204cont.vm.box = "opscode-ubuntu-12.04"
ubuntu1204cont.vm.box_url = "http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-12.04_chef-provisionerless.box"
ubuntu1204cont.vm.network "forwarded_port", guest: 443, host: 8443 # dashboard-ssl
ubuntu1204cont.vm.network "forwarded_port", guest: 8773, host: 8773 # compute-ec2-api
ubuntu1204cont.vm.network "forwarded_port", guest: 8774, host: 8774 # compute-api
ubuntu1204cont.vm.network "private_network", ip: "192.168.3.60"
ubuntu1204cont.vm.network "private_network", ip: "172.16.10.60"
end
#################################
# Ubuntu 12.04 compute1 #
#################################
config.vm.define :ubuntu1204comp1 do |ubuntu1204comp1|
ubuntu1204comp1.vm.hostname = "ubuntu1204comp1"
ubuntu1204comp1.vm.box = "opscode-ubuntu-12.04"
ubuntu1204comp1.vm.box_url = "http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-12.04_chef-provisionerless.box"
ubuntu1204comp1.vm.network "private_network", ip: "192.168.3.61"
ubuntu1204comp1.vm.network "private_network", ip: "172.16.10.61"
end
#################################
# Ubuntu 12.04 chefsererv #
#################################
config.vm.define :ubuntu1204chef do |ubuntu1204comp1|
ubuntu1204comp1.vm.hostname = "ubuntu1204chef"
ubuntu1204comp1.vm.box = "opscode-ubuntu-12.04"
ubuntu1204comp1.vm.box_url = "http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-12.04_chef-provisionerless.box"
ubuntu1204comp1.vm.network "private_network", ip: "192.168.3.62"
end
end
</pre>
<p>What this <strong>Vagrantfile</strong> does it to setup three VMs for us. This way, we can give the VMs all necessary network configurations that we need directly through the file.</p>
<p>Simply run this command in the directory with the <strong>Vagrantfile</strong>:</p>
<pre class="highlight shell">vagrant up
</pre>
<p>To SSH into the Machines, simply run from the <strong>stackforge-chef-server</strong> directory:</p>
<pre class="highlight shell">vagrant ssh ubuntu1204chef
vagrant ssh ubuntu1204cont
vagrant ssh ubuntu1204comp1
</pre>
<h2>Setup Chef Server</h2>
<p>On the <strong>ubuntu1204chef</strong> machine, run the following commands to setup a Chef Server:</p>
<pre class="highlight shell">wget https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef-server_11.1.4-1_amd64.deb
sudo dpkg -i chef-server_11.1.4-1_amd64.deb
sudo chef-server-ctl reconfigure
</pre>
<p>Point your browser to the Chef VM and login using <strong>admin</strong> and <strong>p@ssw0rd1</strong>. Change the password after the login, regenerate the key and copy the key somewhere safe.</p>
<h2>Setup Workstation</h2>
<p>After installing the <strong>ChefDK</strong> earlier on your workstation we have installed everything we need to connect to your Chef Server.</p>
<p>Inside your terminal run:</p>
<pre class="highlight shell"><span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="s2">"/opt/chefdk/embedded/bin:</span><span class="k">${</span><span class="nv">HOME</span><span class="k">}</span><span class="s2">/.chefdk/gem/ruby/2.1.0/bin:</span><span class="nv">$PATH</span><span class="s2">"</span>
chef
</pre>
<p>If both works, everything is setup just fine.</p>
<p>Clone the Chef-Repo Inside your <strong>home</strong> directory like so:</p>
<pre class="highlight shell">git clone git://github.com/opscode/chef-repo.git
Cloning into <span class="s1">'chef-repo'</span>...
remote: Counting objects: 199, <span class="k">done</span>.
remote: Compressing objects: 100% <span class="o">(</span>119/119<span class="o">)</span>, <span class="k">done</span>.
remote: Total 199 <span class="o">(</span>delta 71<span class="o">)</span>, reused 160 <span class="o">(</span>delta 47<span class="o">)</span>
Receiving objects: 100% <span class="o">(</span>199/199<span class="o">)</span>, 30.45 KiB, <span class="k">done</span>.
Resolving deltas: 100% <span class="o">(</span>71/71<span class="o">)</span>, <span class="k">done</span>.
</pre>
<p>Inside of the <strong>chef-repo</strong> directory create a new directory called <strong>.chef</strong>:</p>
<pre class="highlight shell">sudo mkdir -p .chef
</pre>
<p>Now we only need to configure the <strong>knife plugin</strong>:</p>
<pre class="highlight shell">knife configure --initial
</pre>
<p>Answer the questions of the setup. If you have configured everything correctly, you can run the following command to see if everything worked:</p>
<pre class="highlight shell">knife client list
</pre>
<h2>Bootstrap Nodes</h2>
<p>Navigate into your <strong>chef-repo</strong> and run:</p>
<pre class="highlight shell">knife bootstrap localhost --sudo -x vagrant -P vagrant --ssh-port 2222 -N ubuntu1204cont --bootstrap-version 11.14.6-1
knife bootstrap localhost --sudo -x vagrant -P vagrant --ssh-port 2200 -N ubuntu1204comp1 --bootstrap-version 11.14.6-1
</pre>
<p>Please make sure, that the ports numbers are correct. While running the <strong>vagrant up</strong> command earlier, Vagrant will have shown you the correct port numbers.</p>
<h2>Checkout Repo and bundle upload</h2>
<p>Navigate into <strong>chef-repo/cookbooks</strong> and clone the Stackforge cookbook there:</p>
<pre class="highlight shell">git clone -b stable/icehouse https://github.com/stackforge/openstack-chef-repo.git
</pre>
<p>To setup all cookbooks with your Chef server do</p>
<pre class="highlight shell"><span class="nb">cd </span>openstack-chef-repo
berks install
berks upload
</pre>
<p>If you encounter problems regarding SSL, you can create a <strong>config.json</strong> file in your <strong>~/.berkshelf</strong> directory:</p>
<pre class="highlight json"><span class="p">{</span><span class="w">
</span><span class="s2">"ssl"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"verify"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></pre>
<h2>Create environment</h2>
<p>Now we create the environment our deployment will be using:</p>
<pre class="highlight shell"><span class="nb">export </span><span class="nv">EDITOR</span><span class="o">=</span><span class="k">$(</span>which vi<span class="k">)</span>
knife environment create multi-node
<span class="o">{</span>
<span class="s2">"name"</span>: <span class="s2">"multi-node"</span>,
<span class="s2">"description"</span>: <span class="s2">"Environment used in testing the upstream cookbooks and reference Chef repository with vagrant. To be used with the Vagrantfile-multi-neutron vagrantfile. Defines the necessary attributes for a working mutltinode (1 controller/n computes) openstack deployment, using neutron (with gre tunnels between hosts) for the networking component."</span>,
<span class="s2">"cookbook_versions"</span>: <span class="o">{}</span>,
<span class="s2">"json_class"</span>: <span class="s2">"Chef::Environment"</span>,
<span class="s2">"chef_type"</span>: <span class="s2">"environment"</span>,
<span class="s2">"default_attributes"</span>: <span class="o">{}</span>,
<span class="s2">"override_attributes"</span>: <span class="o">{</span>
<span class="s2">"mysql"</span>: <span class="o">{</span>
<span class="s2">"allow_remote_root"</span>: <span class="nb">true</span>,
<span class="s2">"root_network_acl"</span>: <span class="o">[</span><span class="s2">"%"</span><span class="o">]</span>
<span class="o">}</span>,
<span class="s2">"openstack"</span>: <span class="o">{</span>
<span class="s2">"developer_mode"</span>: <span class="nb">true</span>,
<span class="s2">"identity"</span>: <span class="o">{</span>
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"endpoints"</span>: <span class="o">{</span>
<span class="s2">"host"</span>: <span class="s2">"192.168.3.60"</span>,
<span class="s2">"mq"</span>: <span class="o">{</span>
<span class="s2">"host"</span>: <span class="s2">"192.168.3.60"</span>,
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"db"</span>: <span class="o">{</span>
<span class="s2">"host"</span>: <span class="s2">"192.168.3.60"</span>,
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"network"</span>: <span class="o">{</span>
<span class="s2">"debug"</span>: <span class="s2">"True"</span>,
<span class="s2">"dhcp"</span>: <span class="o">{</span>
<span class="s2">"enable_isolated_metadata"</span>: <span class="s2">"True"</span>
<span class="o">}</span>,
<span class="s2">"metadata"</span>: <span class="o">{</span>
<span class="s2">"nova_metadata_ip"</span>: <span class="s2">"192.168.3.60"</span>
<span class="o">}</span>,
<span class="s2">"openvswitch"</span>: <span class="o">{</span>
<span class="s2">"tunnel_id_ranges"</span>: <span class="s2">"1:1000"</span>,
<span class="s2">"enable_tunneling"</span>: <span class="s2">"True"</span>,
<span class="s2">"tenant_network_type"</span>: <span class="s2">"gre"</span>,
<span class="s2">"local_ip_interface"</span>: <span class="s2">"eth2"</span>
<span class="o">}</span>,
<span class="s2">"api"</span>: <span class="o">{</span>
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>
<span class="o">}</span>,
<span class="s2">"image"</span>: <span class="o">{</span>
<span class="s2">"api"</span>: <span class="o">{</span>
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"registry"</span>: <span class="o">{</span>
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"image_upload"</span>: <span class="nb">true</span>,
<span class="s2">"upload_images"</span>: <span class="o">[</span>
<span class="s2">"cirros"</span>,
<span class="s2">"ubuntu"</span>
<span class="o">]</span>,
<span class="s2">"upload_image"</span>: <span class="o">{</span>
<span class="s2">"ubuntu"</span>: <span class="s2">"http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img"</span>,
<span class="s2">"cirros"</span>: <span class="s2">"https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img"</span>
<span class="o">}</span>
<span class="o">}</span>,
<span class="s2">"compute"</span>: <span class="o">{</span>
<span class="s2">"xvpvnc_proxy"</span>: <span class="o">{</span>
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"novnc_proxy"</span>: <span class="o">{</span>
<span class="s2">"bind_interface"</span>: <span class="s2">"eth1"</span>
<span class="o">}</span>,
<span class="s2">"libvirt"</span>: <span class="o">{</span>
<span class="s2">"virt_type"</span>: <span class="s2">"qemu"</span>
<span class="o">}</span>,
<span class="s2">"network"</span>: <span class="o">{</span>
<span class="s2">"public_interface"</span>: <span class="s2">"eth1"</span>,
<span class="s2">"service_type"</span>: <span class="s2">"neutron"</span>
<span class="o">}</span>,
<span class="s2">"config"</span>: <span class="o">{</span>
<span class="s2">"ram_allocation_ratio"</span>: 5
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</pre>
<p>Now we add the <strong>multi-node</strong> environment to our nodes:</p>
<pre class="highlight shell">knife node environment_set ubuntu1204cont multi-node
knife node environment_set ubuntu1204comp1 multi-node
</pre>
<h2>Add Roles and recipes to nodes</h2>
<p>Next, we will be adding the necessary <strong>roles</strong> to our Chef Server.</p>
<p>Once again inside <strong>chef-repo/cookbooks/openstack-chef-repo</strong> run</p>
<pre class="highlight shell">knife role from file roles/<span class="k">*</span>.json
</pre>
<p>Now that the roles are on the Chef Server, we can add them to the nodes:</p>
<pre class="highlight shell">knife node run_list add ubuntu1204cont <span class="s2">"role[os-compute-single-controller-no-network],recipe[openstack-network::identity_registration]"</span>,<span class="s2">"role[os-network-openvswitch]"</span>,<span class="s2">"role[os-network-dhcp-agent]"</span>,<span class="s2">"role[os-network-metadata-agent]"</span>,<span class="s2">"role[os-network-server]"</span>
knife node run_list add ubuntu1204comp1 <span class="s2">"role[os-compute-worker]"</span>
</pre>
<p>As you can see, we configure one node to to be the Controller/Compute node and one to be just an additional Compute node.</p>
<h2>Chef Client run</h2>
<p>SSH on your nodes (ubuntu1204cont, ubuntu1204comp1) and run:</p>
<pre class="highlight shell">sudo chef-client
</pre>
<h2>Test the deployment</h2>
<p>You can now login at the URL of the first VM. Use the username <strong>admin</strong> and the password <strong>admin</strong> for this.
If you navigate to <strong>Admin -> System Panel -> Host Aggregates</strong> you should see both nodes listed.</p>
<p><img class="center" data-thumbnails="original:article-images/host_aggregates.png|small:article-images/host_aggregates-small-200x.png|half:article-images/host_aggregates-half-390x.png|blog:article-images/host_aggregates-blog-780x.png" src="/images/article-images/host_aggregates-blog-780x.png" /> </p>