$Revision: 1051 $
Copyright © 2011 Matthew
Seaman. All rights reserved.
Here, for no particularly good reason, are a number of scripts I wrote and that I think people might find useful. These are all freely available for anyone to download and use according to the terms of the BSD license.
This is a script designed to be run on FreeBSD machines as part of the periodic(8) system. It should work on just about any version of FreeBSD that uses rc.subr(8) style init scripts. This has been submitted as a PR (bin/118325) but it didn't get anywhere much.
What it does is iterate through all of the system and ports rc.d scripts that are enabled and would be run at system boot, and run the script to find the status of each service. Any rc.d script that supports the 'status' command and exits with an error code will generate a notice in the nightly periodic e-mail.
In fact, I tend to use this script as a manual check that all daemons are still running after a session upgrading ports, when many services may be automatically turned off if there is an update. It's not a substitute for proper monitoring, eg with nagios, and it doesn't help with daemons that have been updated, but not restarted, but I think it is still pretty useful even so. For instance, I just updated isc-dhcp42-server:
# /usr/local/etc/periodic/daily/490.status-rc RC Script Status Problems: ==>> /usr/local/etc/rc.d/isc-dhcpd dhcpd is not running.
To use, simply copy the script to /usr/local/etc/periodic/daily. There are two configuration variables:
daily_status_rc_enable="YES" daily_status_rc_exceptions="tmp var netoptions newsyslog bsdstats.sh othermta bgfsck"
These are already set within the script, so it will run automatically without your having to do anything to /etc/periodic.conf or similar.
Note: the rc.d files listed in the default daily_status_rc_exceptions, if enabled, will run their 'start' action unconditionally even if called as scriptname status. Unless you want that to happen every day, leave those in the exceptions list.
This script creates an archive of old mailboxes sorted by month and year. If you're like me and never throw any old e-mails away, doing this is vital to keep mailboxes down to a manageable size. The script assumes an IMAP server — it minimizes the network traffic to the server by using the built-in capabilities of IMAP to get just the relevant parts ie. the dates associated with each message. Moving messages between mailboxes is carried out entirely server-side.
In detail the script logs into the IMAP server. It requests a list of all of the mailboxes, and scans through them all, except for folders named things like 'Junk', 'Trash' or 'Drafts', or any folder matching '_archive', which is the location where it stores the archive it creates. For each mailbox, it requests a list of all the messages sent 30 or more days previously. Based on the Date: header in the message, it sorts these into archive folders prefixed by the year and month. So a message stored in my FreeBSD/Questions folder and delivered on 30th November 2001 would be moved into _archive/2011/11November/FreeBSD/Questions. Archive mailboxes are created on demand when there is content to be moved into them, and are marked as 'unsubscribed.' Internally IMAP treats 'move' as 'copy, then delete the original,' with the usual IMAP semantics that deletion is just marking the file as deleted (meaning it's easily reversible), so we also clean up by expunging all the mailboxes. This has the useful side effect of avoiding wasted space spent storing old deleted messages.
Other than what is supplied with the perl core, you'll need to have the following modules installed (plus their dependencies):
You will need a mail account on an IMAP server. The script assumes that you will be using TLS to access your mail account, because (quite frankly) not protecting yourself from snooping and so forth is at least irresponsible and quite probably could be considered reckless on today's Internet. It will either connect directly to an IMAPS service (ie. port 993), or it will connect to plain IMAP (port 143) and issue a STARTTLS command before attempting to supply login credentials.
I run this script out of crontab(5) once a week, in the small hours of Sunday morning. The script produces quite voluminous output showing exactly what messages have been moved to where, so direct the STDERR output to /dev/null if you'ld prefer not to have this e-mailed to you.
The script takes one optional parameter: the name of a config file with the connection details to login to your IMAP server. If not specified, this defaults to ${HOME}/.archive-mboxrc. Needless to say that since this contains your password in plain text you should set the file permissions to extremely restrictive: chmod 0400 ~/.archive-mboxrc recommended.
% cat ~/.archive-mboxrc host = localhost user = matthew password = ███████████ type = STARTTLS
You need to supply all four of the parameters shown in 'name = value' format. What they are should be pretty self-explanatory, except perhaps for type. Set this to SSL to connect to an IMAPS server on port 993, or to STARTTLS to connect on port 143 and switch to TLS. Only those two options are accepted.
A sample of the output looks like this:
archive-mbox: Prefix = "" Separator = "/" archive-mbox: Will archive messages dated before 04-Nov-2011 archive-mbox: Retrieving list of all mailboxes [...] archive-mbox: Archiving messages from "FreeBSD/Questions" FreeBSD/Questions|84712|Sun, 30 Oct 2011 13:59:58 +0100|_archive/2011/10October/FreeBSD/Questions FreeBSD/Questions|84841|Thu, 03 Nov 2011 08:18:30 +0100|_archive/2011/11November/FreeBSD/Questions FreeBSD/Questions|84866|Thu, 3 Nov 2011 09:12:58 -0700|_archive/2011/11November/FreeBSD/Questions FreeBSD/Questions|84859|Thu, 3 Nov 2011 14:35:04 +0000 (UTC)|_archive/2011/11November/FreeBSD/Questions FreeBSD/Questions|84676|Fri, 28 Oct 2011 16:35:20 -0500 (CDT)|_archive/2011/10October/FreeBSD/Questions FreeBSD/Questions|84870|Thu, 3 Nov 2011 11:21:09 -0500|_archive/2011/11November/FreeBSD/Questions FreeBSD/Questions|84728|Mon, 31 Oct 2011 02:25:14 +0100|_archive/2011/10October/FreeBSD/Questions FreeBSD/Questions|84851|Thu, 3 Nov 2011 09:18:06 -0400|_archive/2011/11November/FreeBSD/Questions FreeBSD/Questions|84762|Mon, 31 Oct 2011 23:21:04 -0500 (CDT)|_archive/2011/10October/FreeBSD/Questions FreeBSD/Questions|84881|Thu, 03 Nov 2011 18:29:06 +0100|_archive/2011/11November/FreeBSD/Questions FreeBSD/Questions|84779|Wed, 02 Nov 2011 00:12:19 +0100|_archive/2011/11November/FreeBSD/Questions [...]
The format is src mailbox|message number|Date: header|dest mailbox
I use Colin Percival's fantastic Tarsnap online backup service, and I have also built my main system around a ZFS mirror filesystem. This script show how I leverage the snapshotting capability of zfs(1m) to get a clean point-in-time backup despite bandwidth limitations meaning the nightly backup might take several hours.
As the name suggests, tarsnap(1) uses a commandline syntax derived from the venerable Unix tar(1) program. Unlike tar(1) it has a configuration file (see tarsnap.conf(5)) which allows standard command line options to be applied automatically to every invocation of tarsnap(1). I assume that tarsnap.conf(5) has been set up to provide all necessary options — particularly the excludes list of filename patterns to avoid backing up. Or you could invert the login and use includes if you prefer that. Thus the only things that need to be suppplied on the tarsnap(1) command line are the parameters that change from call to call. In this case:
The script also manages what backups are kept in on-line storage. It's not very sophisticated. I just keep the last 30 backups — usually that's 30 days worth but missed backups may mean it spans longer — and delete the oldest backups to recover the space. I also go to some lengths to avoid backing up data that is readily available online (like /usr/ports) or that can be recreated easily from public resources.
A outline of how it works is as follows:
The script is designed to be run every night as a root owned cron(8) job. It both produces voluminous output to STDOUT for cron(8) to mail to you, and logs the same output to syslogd(8). Typical output looks like this:
backupsnap-zfs.sh[50997]: Acquired lock on /var/run/backupsnap-zfs.sh.lock backupsnap-zfs.sh[50997]: Created snapshots zroot@20111210-0100 and children backupsnap-zfs.sh[50997]: Backing up / backupsnap-zfs.sh[50997]: Total size Compressed size backupsnap-zfs.sh[50997]: All archives 377355270316 120773806598 backupsnap-zfs.sh[50997]: (unique data) 25585963168 8769208482 backupsnap-zfs.sh[50997]: This archive 813612954 253993981 backupsnap-zfs.sh[50997]: New data 220612 13530 [...]
All the cool kids are implementing IPv6 on their networks. IPv4 is so 20th Century. It's not just a new numbering system; it's a whole different way of thinking about network management.
The fundamental difference is in the number of addresses you'll get in a single allocation. IPv6 addresses are 128 bits long (compare to 32 bits for IPv4). The smallest chunk routable across the Internet is a /64 network (compare to /24 for IPv4) — although this might be further subnetted within a particular site.
Subnetting down below /64 is not going to be something that is done at all frequently. The standard assumption with IPv6 is that the network part of an end-point address is 64 bits long, and the host specific part consists of the other 64 bits. The standard allocation for an end user is a /48, meaning you get 16 bits worth, or 65,536 network prefixes to assign as you will. An ISP or reseller will typically be allocated a /32, giving them 32 bits (4,294,967,296) worth of network prefixes.
Let's think about that for a moment: a single ISP will be able to allocate as many subnets to their customers as there are possible different single addresses on the entire IPv4 Internet. Each of those subnets consists of 64 bits of host space, or 18,446,744,073,709,551,615 different possible addresses (one is reserved for the network address). That number might make more sense to you expressed as 1.8 × 1019 or 18 quintillion. A big number, however you slice it. For instance, that many pollen grains would weigh millions of tonnes. It is a number which is not just slightly bigger than the entire current IPv4 Internet, but around 10 orders of magnitude bigger. And that is the smallest individual allocation that you as an end-user would generally receive.
Now, actually I've glossed over some important implementation details here. Consult RFC 4291 for the full story, but the management summary is that two of the bits in the host part of the address are reserved to flag some particular conditions on how addresses are used. One of the bits indicates if the address is generated from a predefined number built into your hardware — eg. the MAC address of your network adaptor (This is common in SLAAC — StateLess Address AutoConfiguration, see RFC 4862). The other bit flags multicast vs unicast addresses. So, given we won't need to generate our own addresses for SLAAC and all the addresses we are using are unicast, both of those bits should be set to zero. Meaning we actually only have 62 bits to play with, or 4,611,686,018,427,387,903 possible different addresses. Still way more than you could plausibly imagine needing.
So how does having more addresses available than you could ever conceivably use affect the way you manage your IP number space? Simple. You give every different thing on your net a separate number. Not just physical, material things, like network interfaces — although those should obviously get distinguished numbers (for the hardware, using SLAAC to generate an address from the MAC seems a natural fit) but also for each different service on your network, or each different top-level URL served by your webserver.. Need to move your web site to a new machine in a hurry? No need to renumber (nor to wait out pesky DNS TTLs either). No need to disconnect the old server. No need to adjust firewall rules. Just move one IP. Well, and the web content too, obviously.
So, how do you allocate all of these new IP numbers? Well, there are as many different ways of doing that as there are admins who have thought about the problem. There is not necessarily any one-size-fits-all solution, and many different methods will do this job effectively and successfully. However, I'd like to propose simply allocating addresses randomly as a particularly useful solution. It's simple enough not to be prone to errors, parallelizes well (ie. several people can allocate addresses independently, without reference to each other) and it simply doesn't suffer from collisions — by which I mean the same address being incorrectly assigned to two or more different uses.
For someone used to IPv4 thinking, this sounds like madness. Assume a typical /24 network. If I choose random numbers between 1 and 254, chances are even I'll get a collision within about 20 addresses. This is a manifestation of the Birthday paradox, and if you only have small numbers of addresses to play with, it can completely spoil your day. However, being the progressive, IPv6 adopting types that we are, we have 4 billion billion addresses to play with. Plug in those numbers, and you will see that the chances of a collision are tiny until the number of assigned addresses grows to the order of millions. And when I say "tiny" I mean you've more chance of being killed by a meteorite strike. Seriously.
rand-aaaa.pl requires one argument: an IPv6 network prefix, including the prefix bit length:
% ./rand-aaaa.pl 2001:db8::/64 2001:db8::bcc6:ec4f:b722:d640 0.4.6.d.2.2.7.b.f.4.c.e.6.c.c.b.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa
The output is an IPv6 number with random host part but with bits 70 and 71 zeroed (RFC 4291) and formatted according to inet_ntop(3) (RFC 5952). The second line shows the full PTR record for the address suitable for use in DNS zone files.