tags/localchNico Schotteliushttps://www.nico.schottelius.org//tags/localch/Nico Schotteliusikiwiki2016-02-25T13:34:32Z(Virtual) Network loop powered by Qemu, Bonding, Bridging and Aristahttps://www.nico.schottelius.org//blog/comic-qemu-tap-bridge-bond-lacp-arista-network-loop/2016-02-25T13:34:32Z2015-02-03T14:47:26Z
<p><a href="https://www.nico.schottelius.org//blog/comic-qemu-tap-bridge-bond-lacp-arista-network-loop/comic.png"><img src="https://www.nico.schottelius.org//blog/comic-qemu-tap-bridge-bond-lacp-arista-network-loop/comic.png" width="800" height="631" alt="Comic" title="A real loop created by a virtual switch" class="img" /></a></p>
How to control (shutdown) Virtual machines from Qemu/KVM via commandlinehttps://www.nico.schottelius.org//blog/control-and-shutdown-qemu-kvm-vm-via-unix-socket/2016-02-25T13:34:32Z2015-02-03T14:47:26Z
<h2>Introduction</h2>
<p>When you have read
<a href="https://www.nico.schottelius.org//blog/tunneling-qemu-kvm-unix-socket-via-ssh/">the article about tunneling a vnc socket over ssh</a>,
you already know that I am a fan of using simple technologies like Unix
sockets and ssh to access virtual machine information.</p>
<p>I am currently in a project to create a new virtual machine infrastructure
based on Qemu/KVM. The problem we were facing is "How to shutdown a virtual
machine, when the host is being shut down".</p>
<h2>Background</h2>
<p>Qemu/KVM has a so called
<a href="https://en.wikibooks.org/wiki/QEMU/Monitor">monitor</a> to control
the VM. Usually this monitor is reachable from the VNC socket
(or whatever you use to view the console) by pressing
Ctrl-Alt-2.</p>
<p>This is inappropriate for automatic shutdown (I'm not in the mood
to script vnc sessions currently), so there must be a better solution.</p>
<h2>The solution</h2>
<p>Qemu/KVM is able to redirect the monitor to a "character device".
There are ways to create a character device with Qemu/KVM and then
attach the monitor to it, but you can have it more easy by connecting
the monitor directly to a UNIX socket:</p>
<pre><code>qemu-kvm ... -monitor unix:/opt/local.ch/sys/kvm/vm/kvmtest-vm-inx01.intra.local.ch/monitor,server,nowait
</code></pre>
<p>This way we can connect (and control!) Qemu/KVM using
[socat](http://www.dest-unreach.org/socat/]:</p>
<pre><code>socat - UNIX-CONNECT:/opt/local.ch/sys/kvm/vm/kvmtest-vm-inx01.intra.local.ch/monitor
</code></pre>
<p>And when we can connect to it, we can also shutdown a virtual machine:</p>
<pre><code>echo system_powerdown | socat - UNIX-CONNECT:/opt/local.ch/sys/kvm/vm/kvmtest-vm-inx01.intra.local.ch/monitor
</code></pre>
<p>Or we could reset it:</p>
<pre><code>echo system_reset | socat - UNIX-CONNECT:/opt/local.ch/sys/kvm/vm/kvmtest-vm-inx01.intra.local.ch/monitor
</code></pre>
<h2>The full implementation</h2>
<p>The full command line for running a VM as we do it in this project looks like this:</p>
<pre><code>[root@kvm-hw-inx01 kvmtest-vm-inx01.intra.local.ch]# cat start
#!/bin/sh
/usr/libexec/qemu-kvm -m 65536 \
-hda /opt/local.ch/sys/kvm/vm/kvmtest-vm-inx01.intra.local.ch/system-disk \
-vnc unix:/opt/local.ch/sys/kvm/vm/kvmtest-vm-inx01.intra.local.ch/vnc \
-boot order=nc \
-net nic,macaddr=00:16:3e:00:00:2d,vlan=200 \
-net tap,script=/opt/local.ch/sys/kvm/bin/ifup-pz,downscript=/opt/local.ch/sys/kvm/bin/ifdown,vlan=200 \
-net nic,macaddr=00:16:3e:00:00:2e,vlan=300 \
-net tap,script=/opt/local.ch/sys/kvm/bin/ifup-fz,downscript=/opt/local.ch/sys/kvm/bin/ifdown,vlan=300 \
-smp 16 \
-name kvmtest-vm-inx01.intra.local.ch \
-enable-kvm \
-monitor unix:/opt/local.ch/sys/kvm/vm/kvmtest-vm-inx01.intra.local.ch/monitor,server,nowait
</code></pre>
<p>As the VMs are currently not performing as well as they should, we will do
some investigations into performance tuning of Qemu/KVM. So stay tuned, if
you are interested in this topic.</p>
Find e-mail adresses of people in git log outputhttps://www.nico.schottelius.org//blog/find-emails-in-git-log-for-notification/2016-02-25T13:34:32Z2015-02-03T14:47:26Z
<h2>Motivation</h2>
<p>Some days ago I've replaced <strong>cronwrapper</strong>, a script to monitor output of
cron scripts with the replacement <strong>cwrap</strong> in local.ch's puppet configuration.</p>
<p>If the script prints on stdout, <strong>cwrap</strong> does not raise an error by default,
which <strong>cronwrapper</strong> did.</p>
<p>To notify every user of the change, I want to send an email to every
ex-<strong>cronwrapper</strong> user.</p>
<h2>Solution</h2>
<p>The configuration is stored in a subversion repo, which I locally sync using
<strong>git svn</strong>. Thus I can use <strong>git log -p</strong> to see all changes.</p>
<p>A typical line of interest looks like this:</p>
<pre><code>- command => '/usr/local/bin/cronwrapper.sh EMAIL@EXAMPLE.COM "[mob][low][dev03-sth][front] description" /usr/bin/php /some/script',
</code></pre>
<p>Thanks to git, grep, sed, awk, there is a pretty simple solution
(not the most beautiful) to this problem. First of all, get all patches:</p>
<pre><code>git log -p
</code></pre>
<p>Then find all removal entries of cronwrapper:</p>
<pre><code>grep ^- | grep cronwrapper
</code></pre>
<p>But only those containing an e-mail address:</p>
<pre><code>grep '@'
</code></pre>
<p>And filter out the e-mail address:</p>
<pre><code>sed 's/.* \(.*@.*\)/\1/' | awk '{ print $1 }'
</code></pre>
<p>Replace all quotes and backslash quotes:</p>
<pre><code>sed -e 's/\\"//g' -e 's/"//g' -e "s/'//g"
</code></pre>
<p>The problem now is that some e-mail adresses are indeed multiple e-mail adresses
(abc@example.com;def@example.com) and some e-mail adresses are lower, some upper case.</p>
<p>Breaking up the concatenated addresses can be done use awk easily:</p>
<pre><code>awk '{ gsub(";", "\n"); print $0 }'
</code></pre>
<p>Transforming all addresses to lower case can be done using the fine utility <strong>tr</strong>:</p>
<pre><code>tr '[A-Z]' '[a-z]'
</code></pre>
<p>Filter out all duplicates:</p>
<pre><code>sort | uniq
</code></pre>
<p>The result is a list of e-mail addresses. Making them usable for copy & paste
into webmail of exchange needs another filter to convert <strong>\n</strong> to <strong>;</strong>, but
add one <strong>\n</strong> at the end:</p>
<pre><code>awk 'ORS=";" { print $0 } END { ORS="\n"; print "" }'
</code></pre>
<p>So in the end, the complete chanin looks like this:</p>
<pre><code>git log -p | grep ^- | grep cronwrapper | \
grep '@' | sed 's/.* \(.*@.*\)/\1/' | awk '{ print $1 }' | \
sed -e 's/\\"//g' -e 's/"//g' -e "s/'//g" | \
tr '[A-Z]' '[a-z]' | \
awk '{ gsub(";", "\n"); print $0 }' | \
sort | uniq | \
awk 'ORS=";" { print $0 } END { ORS="\n"; print "" }'
</code></pre>
<p>For me, this is a nice demonstration of the power of shell, unix tools and filtering via pipes.</p>
Great Rails Hosting: A symlink for an apphttps://www.nico.schottelius.org//blog/great-rails-hosting-a-symlink-for-an-app/2016-02-25T13:34:32Z2015-02-03T14:47:26Z
<h2>Introduction</h2>
<p>As <a href="http://www.ungleich.ch">ungleich</a> focusses on educated customers,
we meet pretty cool infrastructures from time to time.
In some sense I count <a href="http://www.local.ch">local.ch</a>
as a customer: they supported me with one day off
per week so I was able to found <a href="http://www.ungleich.ch">ungleich</a>
and acquire first customers.
This article is dedicated to <a href="http://www.local.ch">local.ch</a> and describes
a very elegant solution for Ruby on Rails hosting.</p>
<h2>Overview</h2>
<p>The setup consists of the following services, glued
together in an elegant way:</p>
<ul>
<li><a href="http://nginx.org/">nginx</a></li>
<li><a href="http://unicorn.bogomips.org/">unicorn</a></li>
<li><a href="https://www.isc.org/downloads/bind/">ISC Bind</a></li>
<li><a href="https://github.com/capistrano/capistrano/wiki">Capistrano</a></li>
<li>Symlinks</li>
</ul>
<h2>Nginx</h2>
<p>The great trick of the setup is that nginx is used to forward requests
to a unix socket that depends on the <strong>hostname</strong>, which is
exposed as <strong>$host</strong> by nginx. The following
configuration snippet contains the important parts:</p>
<pre><code>server {
listen 80;
location @error_page {
root /var/nginx/$host/current/public;
internal;
[...]
location ~ "^/assets/.*-[a-z0-9]{32}.\w+" {
root /var/nginx/$host/current/public;
[...]
location ~ ^/assets/ {
root /var/nginx/$host/current/public;
[...]
root /var/nginx/$host/current/public;
location @unicorn {
proxy_pass http://unix:/var/nginx/$host/unicorn.sock;
# Forward original host name to be seen in unicorn
proxy_set_header Host $host;
# Server name and address like being available in PHP
proxy_set_header SERVER_NAME $server_name;
proxy_set_header SERVER_ADDR $server_addr;
# The real client IP address - header has ben setup by Zeus
proxy_set_header X-Real-IP $http_x_cluster_client_ip;
# Needed second header for rails - See SYS-1587
proxy_set_header X_FORWARDED_FOR $http_x_cluster_client_ip;
</code></pre>
<p>As you can see, all paths are dependent on the actual hostname
as setup by nginx.</p>
<h2>Application Deployment</h2>
<p>Applications are deployed under their project name below
<strong>/var/nginx</strong>
(like <strong>ws-locomotive.dev-deploy</strong> or <strong>ws-locomotive.master</strong>).
As you can see from the naming, developers can deploy one application
from different branches easily (dev-deploy and master branches is this
case).
Developers can use <a href="https://github.com/capistrano/capistrano/wiki">Capistrano</a> to deploy their applications
and don't need to interact (reload/restart) with nginx, as it is
already configured to accept any hostname.</p>
<h2>Name Server Configuration</h2>
<p>As you can imagine, it would be quite cumbersome for developers to
reach a host named <strong>ws-locomotive.dev-deploy</strong>.
That is why a wildcard domain is configured to point
to the host running nginx:</p>
<pre><code>*.play.intra.local.ch. CNAME rails-dev-vm-snr01.intra.local.ch.
</code></pre>
<h2>Give the application a name</h2>
<p>A new hostname can be assigned to an application simply by symlinking
it to the application:</p>
<pre><code>% cd /var/nginx
% ln -s ws-locomotive.dev-deploy my-fancy-name.play.intra.local.ch
</code></pre>
<p>This way, developers can use <strong>any name</strong> below
play.intra.local.ch for their application. Some applications
actually behave differently depending on the name they are accessed
with:</p>
<pre><code>info.ws-locomotive.master.play.intra.local.ch -> ws-locomotive.master
hp.ws-locomotive.master.play.intra.local.ch -> ws-locomotive.master
</code></pre>
<h2>Conclusions</h2>
<p>The setup is pretty elegant, because it allows developers to
create new development environments without interacting with any
sysadmin to configure nginx, bind or whatsoever.
There is a security drawback though:
An attacker could try to use hostnames like
<strong>../../../../etc/</strong> and request the file <strong>passwd</strong>.
That is the reasion why this service is not exposed
to the outside world directly, but all external requests
are filtered (whitelisting) by a load balancer in front
of the rails hosts.</p>
A guide for IT bosseshttps://www.nico.schottelius.org//blog/guide-for-it-bosses/2016-02-25T13:34:32Z2015-02-03T14:47:26Z
<h2>Introduction</h2>
<p>You are an IT boss. Your job is to manage the
<a href="https://en.wikipedia.org/wiki/The_IT_Crowd">IT crowd</a>.
Depending on your skills and knowledge you may find this
job more easy or hard.</p>
<p>This guide is created by those who you try to manage: The IT crowd.</p>
<h2>Background</h2>
<p>I am a System Engineer currently working 80%
for <a href="http://www.local.ch">local.ch (Swiss Phonebook)</a> and 20% for
<a href="http://www.ungleich.ch">ungleich (Unix/Linux infrastructure company)</a>.</p>
<p>On a daily basis I see how employees and bosses are acting and I spent time
on analysing the behaviour of both parties (for fun - not profit).
As I often see common mistakes and behaviour patterns, which make
good or bad bosses, the idea was born to create a guide for IT bosses.</p>
<h2>Guidelines</h2>
<h3>Be honest</h3>
<p>Not a special requirement in regards of IT, but if you want your
employees to respect you, you definitely need to be honest.</p>
<p>Don't even think about playing tricks on them, they will find out and
everybody will lose the respect for you. Guaranteed.</p>
<h3>Be available</h3>
<p>Your job involves a lot of meetings and coordination.
Your employees understand that and may even be very thankful you took that job.
Still, as you are the boss, communicate straightly when you are available, so
people can bring their questions and problems to you.</p>
<p>If you see there is too less time to be available for your team, it's probably
good time to split up the team or to move on to another position
and promote somebody else for being the head of IT crowd.</p>
<h3>Give freedom</h3>
<p>More important than in probably most other areas is the amount of freedom you
give: IT professionals are usually bright people who understand their job very
well. The learn on the job (which includes getting side tracked from time to time),
they are keen to touch the latest and newest technologies and have a high motivation.</p>
<p>Adding artifical borders to the way the work makes them less productive, less motivated and
in the worst case leave your workplace.</p>
<p>Pay even more attention on this topic, if you have some technical background.
You may know (or think you know!) what the best solution or technical choice is,
but you hired those people to do a good job, not just to execute your thoughts, did you?</p>
<h3>Don't assume</h3>
<p>Don't try to enhance the working situation
of your employees with stuff you assume could be good for them.</p>
<p>You will most likely be wrong.</p>
<p>Instead listen to your employees or ask them about your idea.
Spending an hour or day discussing is probably more worth than throwing away your
shiny new invention.</p>
<h3>Give tools</h3>
<p>Have you ever seen a good craftsman working with broken tools? Probably not. Take the same
approach for your IT professionals: If they request specific tools
(software, notebook, mobile phone, screen, etc.), they probably have a good reason for it.</p>
<p>Don't hesitate to question the request ("Why do you need this / how does it make you more
efficient?"), but also don't hesitate to let them buy the right tools afterwards.</p>
<p>Denying to give good tools makes your employees less motivated, less productive and
indicate you don't value their work.</p>
<p>Regarding value: Did you consider that the 3000 USD notebook,
even if it is not better than the employees current computer,
is worth the motivation you gain from it?</p>
<h3>Plan, assist and communicate objectives</h3>
<p>Your key competence as an IT boss is probably planning and communication.
Use this power to <strong>assist (!)</strong> your IT crowd: Aid them in planning their work,
show them how to plan and communicate what you expect from them.</p>
<p>Don't try to squeeze them into a specific way of working. Better: Let your employees know
what the objectives are (expected results, date of delivery).
They probably
figure better out how to reach it than you. Always remember:
IT guys are different,
some of them love to work in the night,
some of them cannot concentrate in open plan offices
and some of them want to work under high pressure (do all the work in one night).</p>
<h3>Consider the difference</h3>
<p>Compared to many other professions, IT people are behaving a <strong>bit</strong> differently
(that's why sysadmins have their own
<a href="http://www.amazon.com/Management-System-Administrators-Thomas-Limoncelli/dp/0596007833/">time management</a>
book, for example). This may require special treatments from your side: For instance
the usual motivation factors may not work as expected. If you listen carefully, you
may hear "weird" requests like "I'd like to start working at 14:00 until the night".
If possible, try to honour these requests: They don't cost a lot of ressources, but
they require an open minded leader.</p>
<h3>Set values in relation</h3>
<p>IT projects are quite often expensive and there are various reasons for it.
One of them being a future orientated market requires using latest high
technology equipment. IT guys are used to carrying around computers worth
a car or a bigger house. Thus IT guys are aware of the money that is being
spent on IT equipment.</p>
<p>As stated before, your IT guys may have special requirements, not only in terms
of working time, but also their choice of tools may be non standard.</p>
<p>Instead of denying to buy simple tools for your IT guys, include those costs into
the project budget. Also consider reading
<a href="https://en.wikipedia.org/wiki/Parkinson%27s_Law_of_Triviality">Parkinson's law of triviality</a>.</p>
<h2>See also</h2>
<p>The following resources may be of interest for you as well:</p>
<pre><code>* [The Hacker FAQ](http://www.seebs.net/faqs/hacker.html)
* [Managing Humans: Biting and Humorous Tales of a Software Engineering Manager](http://www.amazon.com/Managing-Humans-Humorous-Software-Engineering/dp/1430243147/ref=sr_1_1?ie=UTF8&qid=1366379157&sr=8-1)
* [How to Work with Software Engineers](https://www.kennethnorton.com/essays/how-to-work-with-software-engineers.html)
</code></pre>
<h2>More to come</h2>
<p>This article is work in progress and is being enhanced by input
from other IT professionals (thanks for all the great comments!).</p>
<p>If you want to contribute,
you can add a comment on <a href="https://news.ycombinator.com/item?id=5575419">Hackernews</a>,<br />
<a href="http://www.reddit.com/r/sysadmin/comments/1co3y5/a_guide_for_it_bosses/">reddit</a>
or <a href="https://www.nico.schottelius.org//about/">contact me directly</a>.</p>
Anacron and cronie: How cron.hourly, cron.daily and cron.weekly workhttps://www.nico.schottelius.org//blog/how-cronie-anacron-cron-hourly-daily-weekly-work/2016-02-25T13:34:32Z2015-02-03T14:47:26Z
<h2>Situation</h2>
<p>We noticed that the <a href="http://nginx.org/">nginx</a> logfile is not being rotated
on some freshly setup Centos 6.2 servers, which should have been triggered by
logrotate running from cron. The used cron implementation is cronie together
with anacron.</p>
<h2>Background</h2>
<p>I first suspected the setup being broken due permissions issues
with cronie, which requires special permissions as found in the
<strong>CAVEATS</strong> section of cron(8):</p>
<pre><code>The crontab files have to be regular files or symlinks to regular files,
they must not be executable or writable by anyone else than the owner.
This requirement can be overridden by using the -p option on the crond command line.
</code></pre>
<p>We had this bug before, but this time it is different:</p>
<ul>
<li>The Logrotate cronjob is located at <strong>/etc/cron.daily/logrotate</strong></li>
<li>The cron.{daily, weekly, monthly} jobs are defined in <strong>/etc/anacrontab</strong></li>
<li>The <strong>anacron</strong> command interprets jobs in <strong>/etc/anacrontab</strong></li>
<li>Anacron is called from <strong>/etc/cron.hourly/0anacron</strong></li>
<li><strong>/etc/cron.d/0hourly</strong> contains <strong>01 * * * * root run-parts /etc/cron.hourly</strong></li>
</ul>
<h2>Solution</h2>
<p>In our situation <strong>/etc/cron.d/0hourly</strong> was missing, because we removed all
files from <strong>/etc/cron.d/</strong> and put only our own files in there. The simple
fix is to ensure the contents of this directory are not removed anymore and to
reinstall the <strong>cronie</strong> package to recreate the <strong>/etc/cron.d/0hourly</strong> file.</p>
How to find and execute stuff on all hosts?https://www.nico.schottelius.org//blog/how-to-find-and-execute-stuff-on-all-hosts/2016-02-25T13:34:32Z2015-02-03T14:47:26Z
<h2>Motivation</h2>
<p>Assume that you are managing a pretty large infrastructure of hosts,
sometimes there is a need to execute a command on all of them.</p>
<p>The big question is, where to find out, which hosts exist.</p>
<h2>Solution</h2>
<p>The usual approach is to invent some kind of centralised daemon that collects
or searches for available hosts. There is a way simpler solution available in
my situation, which may help you as well:
We do have a monitoring infrastructure, to which all hosts transmit their
configuration. The configuration is stored containing the full hostname
(like <strong>foo.bar.local.ch</strong>) plus the .cfg suffix.</p>
<p>Thus a script that can be used to execute something on all hosts (sequentially though)
can look like this:</p>
<pre><code>for host in $(ssh monitoring01 "cd /opt/icinga/etc/hosts.d; ls"); do
host=${host%.cfg}
ssh "root@$host" "$@"
done
</code></pre>
KVM Virtual Machines managed with cdist and sexy @ local.chhttps://www.nico.schottelius.org//blog/kvm-vms-with-cdist-at-local.ch/2016-02-25T13:34:32Z2015-02-03T14:47:26Z
<h2>Introduction</h2>
<p>This article describes the KVM setup of <a href="http://www.local.ch">local.ch</a>, which is
managed by <a href="https://www.nico.schottelius.org//software/sexy/">sexy</a> and configured by <span class="createlink">cdist</span>.</p>
<p>If you haven't so far, you may want to have a look at the
<a href="https://www.nico.schottelius.org//blog/sexy-and-cdist-at-local.ch/">Sexy and cdist @ local.ch</a>
article before continuing to read this one.</p>
<h2>KVM Host configuration</h2>
<p>The KVM hosts are Dell R815 with CentOS 6.x installed. Why Dell? Because they
offered a good price/value combination. Why CentOS? Historical
reasons. The hosts got a minimal set of BIOS tuning to support the VM performance:</p>
<ul>
<li>Enable the usual virtualisation flags (don't forget to enable the IOMMU!)</li>
<li>Change the power profile to <strong>Maximum Perforamnce</strong></li>
</ul>
<p>Furthermore, as the CentOS kernel is pretty old (2.6.32-279) and
conservatively configured, the kernel needs the following
command line option to enable the IOMMU:</p>
<pre><code>amd_iommu=on
</code></pre>
<p>Not enabling this option degrades the performance.
In our case, enabling it reduced the latency of the
application running in the VM by a factor of 10.</p>
<p>One big design consideration of the the KVM setup at local.ch is to make the
KVM hosts as independent as possible and sensibly fault tolerant. That said,
VMs are stored on local storage and hosts are always redundantly connected
to two switches use <a href="https://en.wikipedia.org/wiki/Link_aggregation">LACP</a>.</p>
<h2>KVM Host Network Configuration</h2>
<p><a href="https://www.nico.schottelius.org//blog/kvm-vms-with-cdist-at-local.ch/kvm-setup-local.ch-overview.png"><img src="https://www.nico.schottelius.org//blog/kvm-vms-with-cdist-at-local.ch/kvm-setup-local.ch-overview.png" width="770" height="620" alt="Overview of KVM setup at local.ch" class="img" /></a></p>
<p>As can be seen in the picture above, every KVM host is connected to two
<strong>10G Arista switches (7050T-52-R)</strong> using LACP. Besides being capable
of running 10G, the Arista switches are actually pretty neat for the Unix geek,
because they are Linux based with a
<a href="https://en.wikipedia.org/wiki/Field-programmable_gate_array">FPGA</a>
attached. Furthermore you can easily
gain access to a shell by typing <strong>enable</strong> followed by <strong>bash</strong>.</p>
<p>The Arista switches are connected together with 2x 10G links, over which LACP+MLAG
is configured. This gives us the ability to connect every KVM host with LACP to two
<strong>different</strong> switches: They use MLAG to synchronise their LACP states.</p>
<p>On the KVM host, the network is configured as follows:</p>
<p>The dual Port 10G card (Intel Corporation 82599EB) is bonded together into bond0.</p>
<pre><code>[root@kvm-hw-inx01 network-scripts]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
Aggregator ID: 3
Number of ports: 2
Actor Key: 33
Partner Key: 30
Partner Mac Address: 02:1c:73:1b:f5:b2
Slave Interface: eth4
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 68:05:ca:0b:5b:6a
Aggregator ID: 3
Slave queue ID: 0
Slave Interface: eth5
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 68:05:ca:0b:5b:6b
Aggregator ID: 3
Slave queue ID: 0
</code></pre>
<p>The following configuration is used to create the bond0 device:</p>
<pre><code>[root@kvm-hw-inx01 network-scripts]# cat ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
BONDING_OPTS="mode=802.3ad"
ONBOOT=yes
MTU=9000
[root@kvm-hw-inx01 sysconfig]# cat network-scripts/ifcfg-eth4
DEVICE="eth4"
NM_CONTROLLED="yes"
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
[root@kvm-hw-inx01 sysconfig]# cat network-scripts/ifcfg-eth5
DEVICE="eth5"
NM_CONTROLLED="yes"
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
</code></pre>
<p>The MTU of the 10G cards has been set to 9000, as the Arista switches support
<a href="https://en.wikipedia.org/wiki/Jumbo_frame">Jumbo Frames</a>.</p>
<p>Every VM is attached to two different networks:</p>
<ul>
<li>PZ: presentation zone (for general traffic) (10.18x.0.0/22 network)</li>
<li>FZ: filer zone (for NFS and database traffic) (10.18x.64.0/22 network)</li>
</ul>
<p>Both networks are seperated using the VLAN tags 2 (pz) and 3 (fz), which result
in <strong>bond0.2</strong> and <strong>bond0.3</strong>:</p>
<pre><code>[root@kvm-hw-inx01 network-scripts]# ip l | grep bond
6: eth4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000
7: eth5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP
139: bond0.2@bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP
140: bond0.3@bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP
</code></pre>
<p>To keep things simple, the two vlan tagged (bonded) interfaces are added to a bridge each,
to which the VMs are attached later on. The configuration looks like this:</p>
<pre><code>[root@kvm-hw-inx01 network-scripts]# cat ifcfg-bond0.2
DEVICE="bond0.2"
ONBOOT=yes
VLAN=yes
BRIDGE=brpz
[root@kvm-hw-inx01 network-scripts]# cat ifcfg-brpz
DEVICE=brpz
TYPE=Bridge
ONBOOT=yes
DELAY=0
NM_CONTROLLED=no
MTU=9000
</code></pre>
<p>This is how a bridge looks like in production (with about 70 lines stripped):</p>
<pre><code>[root@kvm-hw-inx01 network-scripts]# brctl show
bridge name bridge id STP enabled interfaces
brfz 8000.024db29ca91f no bond0.3
tap13
tap73
[...]
brpz 8000.02f6742800b2 no bond0.2
tap0
tap1
[...]
</code></pre>
<p>Summarised, the network configuration of a KVM host looks like this:</p>
<pre><code>arista1 arista2
| |
[eth4 + eth5] -> bond0
|
|
/ \
bond0.2 bond0.3
/ \
brpz brfz
\ /
tap1 tap2
\ /
VM
</code></pre>
<h2>VM configuration</h2>
<p>The VM configuration can be found below <strong>/opt/local.ch/sys/kvm</strong>
on every KVM host. Every VM is stored below
<strong>/opt/local.ch/sys/kvm/vm/</strong> and contains the following
files:</p>
<pre><code>[root@kvm-hw-inx03 jira-vm-inx01.intra.local.ch]# ls
monitor pid start start-on-boot system-disk vnc
</code></pre>
<ul>
<li>monitor: socket to the monitor from KVM</li>
<li>pid: the pid of the VM</li>
<li>start: the script to start the VM (see below for an example)</li>
<li>start-on-boot: if this file exists, the VM will be started on boot</li>
<li>system-disk: the qcow2 image of the system disk</li>
<li>vnc: socket to the screen of the VM</li>
</ul>
<p>With the exception of monitor, pid and vnc are all files generated by cdist.
The start script of a VM looks like this:</p>
<pre><code>[root@kvm-hw-inx03 jira-vm-inx01.intra.local.ch]# cat start
#!/bin/sh
# Generated shell script - do not modify
#
/usr/libexec/qemu-kvm \
-name jira-vm-inx01.intra.local.ch \
-enable-kvm \
-m 8192 \
-drive file=/opt/local.ch/sys/kvm/vm/jira-vm-inx01.intra.local.ch/system-disk,if=virtio \
-vnc unix:/opt/local.ch/sys/kvm/vm/jira-vm-inx01.intra.local.ch/vnc \
-cpu host \
-boot order=nc \
-pidfile "/opt/local.ch/sys/kvm/vm/jira-vm-inx01.intra.local.ch/pid" \
-monitor "unix:/opt/local.ch/sys/kvm/vm/jira-vm-inx01.intra.local.ch/monitor,server,nowait" \
-net nic,macaddr=00:16:3e:02:00:ab,model=virtio,vlan=200 \
-net tap,script=/opt/local.ch/sys/kvm/bin/ifup-pz,downscript=/opt/local.ch/sys/kvm/bin/ifdown,vlan=200 \
-net nic,macaddr=00:16:3e:02:00:ac,model=virtio,vlan=300 \
-net tap,script=/opt/local.ch/sys/kvm/bin/ifup-fz,downscript=/opt/local.ch/sys/kvm/bin/ifdown,vlan=300 \
-smp 4
</code></pre>
<p>Most parameter values depend on output of sexy,
which uses the cdist type <strong>__localch_kvm_vm</strong>,
which in turn assembles this start script.
The above script may be useful for one or more of my readers,
as it includes a lot of tuning we have done to KVM.</p>
<h2>Automatic startup of VMs</h2>
<p>The virtual machines are brought up by an init script located at
<strong><em>/etc/init.d/kvm-vms</em></strong>. As every VM contains its own startup script
and is marked whether it should be started at boot, the init script
is pretty simple:</p>
<pre><code>basedir=/opt/local.ch/sys/kvm/vm
broken_lock_file_for_centos=/var/lock/subsys/kvm-vms
case "$1" in
start)
cd "$basedir"
# Specific VM given
if [ "$2" ]; then
vm_list=$2
else
vm_list=$(ls)
fi
for vm in $vm_list; do
vm_base_dir="$basedir/$vm"
start_script="$vm_base_dir/start"
# Skip start of machines which should not start
if [ ! -f "$vm/start-on-boot" ]; then
continue
fi
echo "Starting VM $vm ..."
logger -t kvm-vms "Starting VM $vm ..."
screen -d -m -S "$vm" "$start_script"
done
touch "$broken_lock_file_for_centos"
;;
</code></pre>
<p>As you can see, every VM is started in its own
<a href="http://www.gnu.org/software/screen/">screen</a> - so if screen decides to
hang up, only one VM is affected.
Furthermore screen supports only a limited number of windows it can server.
The process listing for a running virtual machine looks like this:</p>
<pre><code>root 64611 0.0 0.0 118840 852 ? Ss Mar11 0:00 SCREEN -d -m -S binarypool-vm-inx02.intra.local.ch /opt/local.ch/sys/kvm/vm/binarypool-vm-inx02.intra.local.ch/start
root 64613 0.0 0.0 106092 1180 pts/22 Ss+ Mar11 0:00 /bin/sh /opt/local.ch/sys/kvm/vm/binarypool-vm-inx02.intra.local.ch/start
root 64614 2.9 2.2 9106828 5819748 pts/22 Sl+ Mar11 5221:41 /usr/libexec/qemu-kvm -name binarypool-vm-inx02.intra.local.ch -enable-kvm -m 8192 -drive file=/opt/local.ch/sys/kvm/vm/binarypool-vm-inx02.intra.local.ch/system-disk,if=virtio -vnc unix:/opt/local.ch/sys/kvm/vm/binarypool-vm-inx02.intra.local.ch/vnc -cpu host -boot order=nc -pidfile /opt/local.ch/sys/kvm/vm/binarypool-vm-inx02.intra.local.ch/pid -monitor unix:/opt/local.ch/sys/kvm/vm/binarypool-vm-inx02.intra.local.ch/monitor,server,nowait -net nic,macaddr=00:16:3e:02:00:7f,model=virtio,vlan=200 -net tap,script=/opt/local.ch/sys/kvm/bin/ifup-pz,downscript=/opt/local.ch/sys/kvm/bin/ifdown,vlan=200 -net nic,macaddr=00:16:3e:02:00:80,model=virtio,vlan=300 -net tap,script=/opt/local.ch/sys/kvm/bin/ifup-fz,downscript=/opt/local.ch/sys/kvm/bin/ifdown,vlan=300 -smp 4
</code></pre>
<h2>Common Tasks</h2>
<p>The following sections show you how to do regular maintenance
tasks on the KVM infrastructure.</p>
<h3>Create a VM</h3>
<p>VMs can easily be created using the script <strong>vm/create-vm</strong> from the sysadmin-logs repository
(local.ch internally), which looks like this:</p>
<pre><code>sexy host add --type vm $fqdn
sexy host vm-host-set --vm-host $vmhost $fqdn
sexy host disk-add --size $disksize $fqdn
sexy host memory-set --memory $memory $fqdn
sexy host cores-set --cores $cores $fqdn
mac_pz=$(sexy mac generate)
mac_fz=$(sexy mac generate)
sexy host nic-add $fqdn -m $mac_pz -n pz
sexy host nic-add $fqdn -m $mac_fz -n fz
sexy net-ipv4 host-add "$net_pz" -m "$mac_pz" -f "$fqdn"
sexy net-ipv4 host-add "$net_fz" -m "$mac_fz" -f "$fz_fqdn"
echo "Updating git / github ..."
cd ~/.sexy
git add db
git commit -m "Added host $fqdn"
git pull
git push
# Apply changes: first network, so dhcp & dns are ok, then create VM
cat << eof
Todo for apply:
sexy net-ipv4 apply --all
sexy host apply --all
Start VM on $vmhost: ssh $vmhost /opt/local.ch/sys/kvm/vm/$fqdn/start
eof
</code></pre>
<h3>Delete a VM</h3>
<p>Run the script <strong>remove-host</strong>, which essentially does the following:</p>
<ul>
<li>Remove various monitoring / backup configurations</li>
<li>Detect if it is a VM, if so</li>
<li>Stop it</li>
<li>Remove it from the host</li>
<li>Add mac address to the list of free mac addresses</li>
<li>Delete host from the networks</li>
<li>Delete host from sexy database</li>
</ul>
<h3>Move VM to another server</h3>
<p>To move one VM to another host, the following steps are necessary:</p>
<ul>
<li>sexy host vm-host-set ... # to new host</li>
<li>stop vm</li>
<li>scp/rsync directory from old host to new host</li>
<li>sexy host apply --all # record db change</li>
<li>start vm on new host</li>
</ul>
Puppet bugs that motivated me to migrate away from puppet and write cdisthttps://www.nico.schottelius.org//blog/puppet-bugs-motivation-for-migration-and-cdist/2016-02-25T13:34:32Z2015-02-03T14:47:26Z
<h2>Introduction</h2>
<p>For a long time I had a "secret list" of bugs that made me going crazy when
using puppet. But as I get more often asked <strong><em>Why have you writen
<a href="https://www.nico.schottelius.org//software/cdist/">cdist</a> and migrate away from puppet?</em></strong>, I'm publishing the
list here, to give it a real home.</p>
<p>Early clearification, before rumors come up and bad blood created:
This is <strong>not intented</strong> to be a <strong>bash puppet page</strong>, but a
<strong>I don't need to re-explain why I moved away from puppet and
have written cdist page</strong>.</p>
<h2>Bugs</h2>
<p>This is the initial short list, explanations may follow.</p>
<ul>
<li><p><a href="http://projects.puppetlabs.com/issues/86">Puppet cannot create directories and their parents, 2006, rejected</a></p></li>
<li><p><a href="http://projects.puppetlabs.com/issues/1565">Puppet parser order dependant, 2008, still open in 2012</a></p></li>
<li><a href="http://projects.puppetlabs.com/issues/2538">Cannot get return code of command, 2009</a></li>
<li><a href="http://projects.puppetlabs.com/issues/3767">Local puppet != remote, 2010, >= 2 years idle</a></li>
<li><a href="http://projects.puppetlabs.com/issues/3936">Service stopping broken in debian, 2010, >= 10 months idle</a></li>
<li><a href="http://projects.puppetlabs.com/issues/3987">${var} does not work everywhere, 2010, >= 9 months idle</a></li>
<li><a href="http://projects.puppetlabs.com/issues/3997">"magic var" only in some scopes, 2010, rejected</a></li>
<li><a href="http://projects.puppetlabs.com/issues/3998">Mount/autorequire, 2010, >= 1 year idle</a></li>
<li><a href="http://projects.puppetlabs.com/issues/3998">Old bug marked as duplicate of a new one (see above)</a></li>
<li><a href="http://projects.puppetlabs.com/issues/4220">Parser bug</a></li>
<li><a href="http://projects.puppetlabs.com/issues/4680">SSL cert not submitted to a new puppetmaster, 2010, >= 1 year idle</a></li>
<li><a href="http://projects.puppetlabs.com/issues/4715">Reusing defines not possible, 2010, rejected</a></li>
<li><a href="http://projects.puppetlabs.com/issues/4780">Param a=$undef broken, 2010, fixed</a></li>
<li><a href="http://projects.puppetlabs.com/issues/4805">Templating broken, 2010, fixed</a></li>
<li><a href="http://projects.puppetlabs.com/issues/4922">Puppetd creates empty files, if it gets a 404, 2010, >= 11 months idle</a></li>
<li><a href="http://projects.puppetlabs.com/issues/4922">Fixed symptom, not source, rescheduling of import problem, see above</a></li>
<li><a href="http://projects.puppetlabs.com/issues/5048">"" (empty string) is not a valid resource reference, 2010, fixed</a></li>
<li><a href="http://projects.puppetlabs.com/issues/6209">Puppet changes the errror message on 2nd run, 2011, >= 1 year idle</a></li>
<li><a href="http://projects.puppetlabs.com/issues/6210">Fix error messages to be meaningful, 2011, >= 1 year idle</a></li>
<li><a href="https://www.nico.schottelius.org//blog/puppet-name-is-not-as-expected-but-classname/">In puppet, $name is not always what you expect, 2012</a></li>
<li><a href="http://projects.puppetlabs.com/issues/8229">Error "regexp buffer overflow" when backing up binary data, 2011</a></li>
<li><a href="http://projects.puppetlabs.com/issues/14577">Could not intern from pson: expected (with pseudo random values afterwards), 2012</a></li>
<li><a href="http://projects.puppetlabs.com/issues/16946">Regular expressions take precendence over direct node specifications, 2012</a></li>
</ul>
<h2>Contact</h2>
<p>If you think there's something wrong here and want to discuss the listing, do not hesitate
to discuss it on one of the
<a href="https://www.nico.schottelius.org//software/cdist/">cdist communication channels (irc, mailing list, mail)</a>.</p>
In puppet, $name is not always what you expecthttps://www.nico.schottelius.org//blog/puppet-name-is-not-as-expected-but-classname/2016-02-25T13:34:32Z2015-02-03T14:47:26Z
<h2>Situation</h2>
<p>I've tried to create a smart file definition for two files that belong into one
directory using this code snippet:</p>
<pre><code>file { ["check-disk-shell-net-snmp", "check_icinga_config.sh"]:
ensure => present,
path => "${check_base}/${name}",
source => "puppet:///modules/icinga2/${name}",
owner => icinga,
group => icinga,
mode => 775,
require => File["${check_base}"];
}
</code></pre>
<p>As described in the
<a href="http://docs.puppetlabs.com/references/2.7.0/type.html">puppet documentation</a>,
the path is usually constructed by using <strong>namevar</strong>, which I interpret as
"the variable named <strong>name</strong>".</p>
<h2>The problem</h2>
<p>What happens is actually something totally different (puppet --version: 2.7.5):</p>
<pre><code>err: Failed to apply catalog: Cannot alias File[check-disk-shell-net-snmp] to
["/opt/local.ch/sys/icinga/checks/icinga2::serverchecks"] at
/etc/puppet/modules/icinga2/manifests/serverchecks.pp:25; resource
["File", "/opt/local.ch/sys/icinga/checks/icinga2::serverchecks"] already defined at
/etc/puppet/modules/icinga2/manifests/serverchecks.pp:25
</code></pre>
<p>The internal alias message is a bit confusing
(I did not intentionally create an alias), but that puppet is using the classname
instead of the name supplied to file is surprising.</p>
<p><strong>Update:</strong> I've found the correct documentation part in the
<a href="http://docs.puppetlabs.com/guides/language_guide.html">puppet language guide</a>
that describes the feature I was trying to use:</p>
<pre><code>Most resources have an attribute (often called simply name) whose value
will default to the title if you don’t specify it. (Internally, this is
called the “namevar.”) For the file type, the path will default to the
title. A resource’s namevar value almost always has to be unique.
(The exec and notify types are the exceptions.)
</code></pre>
<h2>The solution</h2>
<p>Well, there are two solutions:</p>
<ul>
<li>rewrite to two file entries (simple, code redundancy, ugly)</li>
<li>switch over to using <a href="https://www.nico.schottelius.org//software/cdist/">cdist</a> (more initial effort, biased author)</li>
</ul>
<p>It is very good from time to time being remembered, which motivations I had
when starting the cdist project. In this case, it had been:</p>
<ul>
<li>Supply understandable, good error messages to the user</li>
<li>Do what the user expects</li>
<li>Consistent behaviour</li>
</ul>
<p>\
If you are interested, there is
<a href="http://firma.schottelius.org/english/infrastructure/">commercial support available</a> for
puppet to cdist migrations.
\</p>
Sexy and cdist interaction: Sexy chooses hosts, cdist configureshttps://www.nico.schottelius.org//blog/sexy-cdist-interaction-sexy-chooses-cdist-configures/2016-02-25T13:34:32Z2015-02-03T14:47:26Z
<h2>Introduction</h2>
<p>Version 2 of <a href="https://www.nico.schottelius.org//software/sexy/">sexy</a>,
the Swiss Army Knife for inventory management, is already
<strong>using</strong> and <strong>usable</strong> from <span class="createlink">cdist</span>.</p>
<p>This is the first blog post of a series showing examples of
using sexy and cdist.</p>
<h2>Example</h2>
<p>Cdist is executed with a list of hosts to operate on:</p>
<pre><code>% cdist config
usage: cdist config [-h] [-d] [-v] [-c CDIST_HOME] [-i MANIFEST] [-p] [-s]
[--remote-copy REMOTE_COPY] [--remote-exec REMOTE_EXEC]
host [host ...]
</code></pre>
<p>Sexy in turn is able to manage hosts, mac addresses and networks:</p>
<pre><code>% sexy
usage: sexy [-h] [-d] [-v] [-V] {net-ipv4,host,mac} ...
sexy: error: too few arguments
</code></pre>
<p>Sexy knows about a command to list hosts, named <strong>host list</strong>.
So I can use sexy to tell cdist which hosts to configure. For instance
all dhcp servers:</p>
<pre><code>% sexy host list | grep dhcp
dhcp-vm-inx01.intra.local.ch
dhcp-vm-inx02.intra.local.ch
dhcp-vm-snr01.intra.local.ch
dhcp-vm-snr02.intra.local.ch
% ./bin/cdist config -vp $(sexy host list | grep dhcp)
INFO: dhcp-vm-inx01.intra.local.ch: Running global explorers
INFO: dhcp-vm-snr01.intra.local.ch: Running global explorers
INFO: dhcp-vm-snr02.intra.local.ch: Running global explorers
INFO: dhcp-vm-inx02.intra.local.ch: Running global explorers
...
</code></pre>
<p>Sexy, isn't it?</p>
Why CentOS does not stop your init scripthttps://www.nico.schottelius.org//blog/why-centos-does-not-stop-your-init-script/2016-02-25T13:34:32Z2015-02-03T14:47:26Z
<h2>Introduction</h2>
<p>If you created a simple init script below <strong>/etc/init.d</strong>, which
gets started, but not stopped on reboot or system halt, this
article is for you.</p>
<h2>Background</h2>
<p>I assume you ensured that the <strong>chkconfig</strong> information in the
script is correct and that you ran chkconfig $name on. The output
of chkconfig should look like this:</p>
<pre><code>[root@kvm-hw-snr01 ~]# chkconfig --list | grep kvm-vms
kvm-vms 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@kvm-hw-snr01 ~]#
</code></pre>
<p>Although this looks correct, there is a small block in <strong>/etc/rc.d/rc</strong>
that prevents your init script from being called on stop:</p>
<pre><code># First, run the KILL scripts.
for i in /etc/rc$runlevel.d/K* ; do
# Check if the subsystem is already up.
subsys=${i#/etc/rc$runlevel.d/K??}
[ -f /var/lock/subsys/$subsys -o -f /var/lock/subsys/$subsys.init ] || continue
check_runlevel "$i" || continue
# Bring the subsystem down.
[ -n "$UPSTART" ] && initctl emit --quiet stopping JOB=$subsys
$i stop
[ -n "$UPSTART" ] && initctl emit --quiet stopped JOB=$subsys
done
</code></pre>
<p>So only if your script creates
/var/lock/subsys/<strong>yourscriptname</strong> or /var/lock/subsys/<strong>yourscriptname</strong>.init,
it will be called on stop.</p>
<h2>Solution</h2>
<p>You can include the following three lines into your script to get
your script stopped:</p>
<pre><code>broken_lock_file_for_centos=/var/lock/subsys/kvm-vms
# In the start block
touch "$broken_lock_file_for_centos"
# In the stop block
rm -f "$broken_lock_file_for_centos"
</code></pre>