Posts Tagged ‘work’

How-to: Change your server’s IP address

Sunday, October 25th, 2009

The Problem

It happens to the best of us. I administer an Internet-facing server for a client, who has native IPv4 connectivity, and IPv6 connectivity via 6to4. All was going well, until one day, the owner needed more IPv4 addresses. So we contacted their ISP (Comcast) and upgraded our /30 block (one IPv4 address) to a /29 block. That’s all and well, except:

  1. The new /29 subnet does not contain the /30 subnet.
  2. Comcast could not give a transition period. As soon as they pushed the config file to the gateway, it would reboot, instantly deactivating the old /30 and activating the new /29.

The Ideal Solution

Needless to say, this is a bit of a problem. In a perfect world, Comcast would activate the /29 alongside the /30, keep both up for a few days, and then shut down the old /30. That would give us enough seamless transition time, so our clients and customers would see no interruption of service.

The Accepted Solution

This solution is partly drawn from experience, and partly from a “Gee, in retrospect, I should have done that” position. For the rest of this post, I will use the following conventions:

192.0.2.1/30
The old IPv4 address and subnet, in CIDR notation.
192.0.2.100/29
The new IPv4 subnet.
2001:db8:0:1::1
The old IPv6 address. (Yes, I know this isn’t in the 6to4 subnet, 2002::/16. I’m opting to use the documentation prefix instead.)
2001:db8:0:1::100
The new IPv6 address.
example.com
The server’s DNS name.

The following post also assumes that you are running your DNS on BIND 9. If you don’t, then the specific DNS changes neededare left as an exercise for the reader; fortunately, this isn’t anything complicated.

1. Notify your users.

Firstly, if your site has any sort of announcement system, like an RSS feed, or a login page, put up a notice warning users of downtime. At the very least, inform them that there will be downtime, and have them plan to not use your site immediately after the IP address changeover. This may seem like a no-brainer, but even the most hardcore of nerds working for a company shouldn’t forget about customer service.

Because our site is tech-oriented, I also included directions on how to reach our site if it refuses to load (that is, having the user change their hosts file). A bit overkill, but I like to cover all bases.

2. Decrease your zone’s TTL.

Secondly, what you want to do is update your DNS records. Two things need to be addressed. The first is to change your zone’s TTL down to a smaller value. Change this to the maximum amount of downtime that is acceptable. They keep their DNS TTL at 86,400 seconds (one day). For this, I’d recommend changing it to 10,800 seconds (3 hours). Change your SOA record accordingly:

1
2
3
4
5
6
7
8
$TTL 10800  ;change this to 10,800 seconds (three hours)
www.example.com. IN SOA authoritative.dnsserver.com. youremail.example.com. (
        2009102501      ;Serial — increment this, as always
        3h              ;Refresh — change this to "3h" (10,800 seconds)
        1h              ;Retry
        4w              ;Expire
        3h              ;Negative caching TTL
 )

Lower TTL values will cause your authoritative nameserver to get a great deal more traffic. If you outsource your DNS hosting like this client does, that may anger your DNS host.

3. Add your new IP addresses to your zone

The second thing to do is update your zone files to include both the old and new addresses. If you have both IPv4 and IPv6 connectivity, then this means that your zone file will have four IP addresses. Bear in mind the following facts about DNS when a host has more than one IP address:

  • Generally, DNS servers will shuffle the records, and return the IP addresses in a different order every time.
  • Generally, a client will pick a random IP address out of the list; A and AAAA records don’t have a priority value like MX and SRV records do.
  • Almost always, a client will try another IP address if the first fails.

Thus, with two IP addresses in the zone file, your clients will connect quickly about 50% of the time during the DNS transition. This is a very simple change:

1
2
3
4
www.example.com. IN A 192.0.2.1
www.example.com. IN A 192.0.2.100
www.example.com. IN AAAA 2001:db8:0:1::1
www.example.com. IN AAAA 2001:db8:0:1::100

Save your zone file, sign it (if you use DNSSEC), and do an

1
rndc reload

to reload the zone and notify your authoritative nameserver. Remember your old TTL value? Wait at least that much time before proceeding with the IP changeover, especially if you have international users.

4. Edit your server’s IP configuration.

You’re going to have a little downtime, and a period of confused users. But keep thinking ahead. Now, log into your server (preferably over a physical console) and update its IP configuration. On Gentoo Linux, this consisted of:

  • Editing the /etc/conf.d/net file, to include the new IP information:
    1
    2
    3
    4
    5
    6
    7
    config_eth0=(
        "192.0.2.1/30"
        "192.0.2.100/29"
        "2001:db8:0:1::1/64"
        "2001:db8:0:100::1/64"
        # Also include your other IP addresses here.
    )
  • Because this server is also an IPv6 router for the local subnet, radvd is announcing the IPv6 prefix to the private IPv4 subnet. Adding a prefix to the /etc/radvd.conf file:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    prefix 2001:db8:0:1::1/64 {
        AdvValidLifetime 86400;
        AdvPreferredLifetime 86400;
        AdvOnLink on;
        AdvAutonomous on;
        AdvRouterAddr on;
    };
    prefix 2001:db8:0:100::1/64 {
        AdvValidLifetime 86400;
        AdvPreferredLifetime 86400;
        AdvOnLink on;
        AdvAutonomous on;
        AdvRouterAddr on;
    };

    Save and reload radvd.

  • Checking the DHCP server settings. DHCP may not be affected, but DHCPv6 will definitely be affected by the changing subnets.
  • If you have the fixed IP address recorded anywhere else, make sure you add the new IP addresses! (I can’t help you on this one, but they did have the IP address specified in: (1) Postfix’s /etc/postfix/main.cf file, with the smtp_bind_address and smtp_bind_address6 directives, (2) switch configurations, and (3) IP address filters in Apache for their Intranet applications.

Reboot the server if necessary. If you can, reboot it anyway — even you, Linux users.

5. Change your IP address.

The big moment has come. Contact your ISP and have them perform the changeover. This resulted in about a minute of downtime as their cable modem received the new settings and rebooted. As soon as you can afterwards:

  • Remove the old IP addresses from the zone file. Increment the serial, reset the TTL, save, sign, and rndc reload.
  • Remove the old IPv6 subnet from /etc/radvd.conf.
  • Reboot some workstations and check basic connectivity to and from the server and the IPv4 and IPv6 Internet. Then go offsite and try to reach the server with the DNS name and new IP addresses.
  • Be prepared to answer phone calls and emails from clients who can’t reach the site.

I hope that this helps anyone else in this predicament. I still don’t know why Comcast couldn’t offer a transition period, especially to a business customer. Regardless, the changeover for this client went very smoothly, and there were no issues reported. Everything is now running just fine, and they’re making use of their new — and limited — IPv4 address space.

Tags: , , , , ,
Posted in Computers, Networking | 4 Comments »

dnssec-signzone and “File not found” error

Tuesday, August 11th, 2009

DNSSEC is a very good thing, in my opinion. For almost twenty years, there has been at least a proof of concept about attacks against DNS. Kaminsky’s DNS flaw last year only strengthened the push to implement cryptographic extensions, and now, ICANN says that com. and . (the root zone) will be signed by year’s end, which means that DNSSEC can be properly implemented.

Generating the keys and signing a bunch of zones, however, quickly becomes arduous. For one domain, it’s not bad. When you administer two external domains, four reverse IPv4 and IPv6 zones, and an internal zone or two, it quickly becomes a hassle. Like any Linux user who spends some time at the command-line, I wrote a rudimentary shell script to automate this process. The latest version is at the end of this post, but read on. I ran into a problem, and not finding anything on Google, I decided to document it here.

Now, I like a nice clean directory listing. When you use SSH as your primary means of getting things done on the server, it’s a requisite. I wrote my script to rename the generated keys, and place them all in the keys folder before signing the zone ($1 being the zone name):

1
2
3
4
/bin/mv $ZSKNAME.key keys/$1.zsk.key
/bin/mv $ZSKNAME.private keys/$1.zsk.key.private
/usr/sbin/dnssec-signzone -t -N increment -H 10 -k $1.ksk.key \
    -e +7776000 -o $1. $1.zone /var/bind/pri/$1.zsk.key

All seemed well until I ran it:

# ./signzone.sh example.com
Will sign example.com.zone
Generating new key-signing keypair.
Generating zone-signing keypair.
Signing zone.
dnssec-signzone: cannot load dnskey keys/example.com.zsk.key: file not found
Done.

It complained about the file not existing, but it definitely did:

# stat example.com.zsk.key
  File: `example.com.zsk.key'
  Size: 395          Blocks: 8          IO Block: 4096   regular file
Device: 6802h/26626d   Inode: 17989721    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2009-08-11 11:20:12.000000000 -0400
Modify: 2009-08-11 11:20:12.000000000 -0400
Change: 2009-08-11 11:20:12.000000000 -0400

It took me a few hours of troubleshooting, modifying the script, wondering about NSEC3 records, and even rebuilding BIND 9.6.1-p1 from source. Finally, I installed and used strace on my script, which revealed the problems:

  1. BIND, especially dnssec-signzone, expect the public keys to end in .key and the private keys to end in .private. Changing their extensions from .key.private to .private fixed it.
  2. dnssec-signzone expects, as an argument, the name of the keypair, minus the extension. With the correctly-named keypair of example.com.ksk.key and example.com.ksk.private, dnssec-signzone should be run with the -k example.com.ksk argument. Note the lack of extensions.

After fixing that, dnssec-signzone stopped complaining, and I was able to sign all of my zones without a hitch. Restarted BIND, and it’s sending out notifies as I type this.

Now, for those of you who came here for the finished, bug-free script, here you go.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
#!/bin/sh
# signzone.sh, Copyright © 2009 Colin Cogle.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program.  If not, see <http ://www.gnu.org/licenses/>.
#
# Usage:  signzone.sh <name of zone to sign>
# Example:  signzone.sh example.com
# Example:  signzone.sh 8.b.d.0.1.0.0.2.ip6.arpa

if [ -f $1.zone ];
then
    echo "Will sign $1.zone"
else
    echo "$1.zone does not exist.";
    exit;
fi

#
# Stash our keys in the keys subdirectory, just to keep things clean.
#
if [ -f keys/$1.ksk.key ];
then
    echo "Using pre-generated key-signing keypair."
else
    echo "Generating new key-signing keypair."
    KSKNAME=$(/usr/bin/dnssec-keygen -f KSK -e -a NSEC3RSASHA1 -b 2048 -n ZONE $1)
    /bin/mv $KSKNAME.key keys/$1.ksk.key
    /bin/mv $KSKNAME.private keys/$1.ksk.private
fi

if [ -f keys/$1.zsk.key ];
then
    echo "Using pre-generated zone-signing keypair."
else
    echo "Generating zone-signing keypair."
    ZSKNAME=$(/usr/bin/dnssec-keygen -a NSEC3RSASHA1 -b 2048 -n ZONE $1)
    /bin/mv $ZSKNAME.key keys/$1.zsk.key
    /bin/mv $ZSKNAME.private keys/$1.zsk.private
fi

echo "Signing zone."
/usr/sbin/dnssec-signzone -a -g -t -N increment -H 10 -3 caf3 -d keys -k keys/$1.ksk -e +7776000 -o $1. $1.zone keys/$1.zsk

echo "Done."

Do note that this Bash script is a rough draft, and represents only about ten minutes of coding and referencing man pages. I’ve licensed it under the AGPL if anyone wants to make a copy for themselves.

Tags: , ,
Posted in Computers, Networking, Programming | 2 Comments »

Naming Software

Friday, August 7th, 2009

Naming software has got to be one of the most difficult steps of the software development lifecycle.

This winter, I quickly and single-handedly developed a Web-based help desk software. Clients log in to submit trouble reports, and they can track and edit them as things progress. It also allows them to track their assets. It makes great use of PHP, HTML 5, CSS 3.0, AJAX, geolocation, Google Gears, and more; it’s a balanced mix of tried-and-true and bleeding-edge technology.

The world’s only (known) installation is on my company’s Web server. However, since then, it’s grown by leaps and bounds, but still running on the same spaghetti PHP code; even though it was developed in a rush, if my old college professors could see what I did, they’d revoke my degree.

So, what I’d like to do is clean it up, document it, make it portable, and release it as a free and open-source project. Naturally, the company would sell support contracts, like Red Hat does with Linux — there’s the moneymaking portion of it, enough to please my boss.

However, I just can’t think of a good name. The best name that I came up with is Mercury, eponymous after the messenger of the Greek gods. Trouble is, while “Project Mercury” rolls off the tongue with aplomb, and what I’m calling the development process, it’s not the greatest of names for a final product.

New and returning readers, it’s discussion time. What’s in a name? How do you name software? Have any ideas?

Tags: , ,
Posted in Computers | No Comments »