Sunday, March 29, 2015

FreeBSD Security Report

The Second of the FreeBSD periodic reports to be looked at is the security report. It runs daily but is sent in a separate e-mail from the daily report. This will hopefully help system administrators to realize that some attention really should be paid to this report.

There are fewer settings but it is important to pay attention to all of them and understand what they are trying to communicate.

Sample Security Run Output

Checking setuid files and devices:
Checking negative group permissions: 
Checking for uids of 0:
root 0
toor 0 
Checking for passwordless accounts: 
Checking login.conf permissions: 
Checking for ports with mismatched checksums: 
Hostname login failures:
Mar 25 17:51:41 Hostname sshd[33490]: Invalid user admin from
Mar 25 17:51:41 Hostname sshd[33490]: input_userauth_request: invalid user admin [preauth]
Mar 25 17:51:45 Hostname sshd[33496]: Invalid user db2fenc1 from
Mar 25 17:51:45 Hostname sshd[33496]: input_userauth_request: invalid user db2fenc1 [preauth]
Mar 25 17:51:47 Hostname sshd[33498]: Invalid user oracle from
Mar 25 17:51:47 Hostname sshd[33498]: input_userauth_request: invalid user oracle [preauth]
Mar 25 17:51:50 Hostname sshd[33502]: Invalid user git from
Mar 25 17:51:50 Hostname sshd[33502]: input_userauth_request: invalid user git [preauth]
Mar 25 17:52:04 Hostname sshd[33516]: Invalid user aaron from
Mar 25 17:52:04 Hostname sshd[33516]: input_userauth_request: invalid user aaron [preauth]
Mar 25 17:52:05 Hostname sshd[33519]: Invalid user gt05 from
Mar 25 17:54:42 Hostname sshd[33707]: input_userauth_request: invalid user oracle [preauth]


Mar 25 19:04:00 Hostname sshd[33953]: reverse mapping checking getaddrinfo for [] failed - POSSIBLE BREAK-IN ATTEMPT! [preauth]
Mar 25 19:04:00 Hostname sshd[33953]: reverse mapping checking getaddrinfo for [] failed - POSSIBLE BREAK-IN ATTEMPT!

Hostname refused connections: 
Checking for packages with security vulnerabilities:
Database fetched: Wed Mar 25 03:12:53 EDT 2015

Checking setuid files and devices

Most days this should be a blank section. The very first time this script runs, it will contain a list of all the files on the system that have the setuid bit enabled. Subsequent runs will only show changes between runs. Operating system upgrades and some 3rd party software may change (usually add) setuid files or devices. This means you will only have one notification of the change.

The setuid permission is a special UNIX permission which says that the program (file) should be run as the owning user, not the user that actually ran the application. This is how FreeBSD (and UNIX systems in general) provide for temporary escalated privileges to ordinary users.

Unexpected changes need to be investigated. Abuse of setuid permissions is still one of the most common methods for internal users to gain unauthorized access to the root account and compromise systems.

You will see a sub-section with the heading, Hostname setuid diffs, when there are changes in setuid permissions.

Checking negative group permissions

There should never be negative group permissions on files or directories. This section should always be blank.

Negative group permissions means that 'other' users (users not in same group as the file/directory) have more permissions than 'group' users.

Naive system administrators have been known to use this tactic in an attempt to let one group of users read files that are created by a different group of users. The problem is that when file access permissions are checked, only the user's active group is checked. Any user in more than one group can change their active group and no longer be considered a 'group' user of the file. This makes a very fragile security model that is too easy to break. 

Filesystem ACLs were created to provide the sort of functionality that an administrator might want in situations where negative groups are considered. Use filesystem ACLs instead.

Hostname changes in mounted filesystems

When the mounted filesystems change between security runs, a section that identifies the differences will be included. If no changes are made, this section does not show up.

Administrators do have cause to make changes to mounted filesystems at times but malicious users may try to introduce new filesystems as a step to gaining unauthorized access to a server. Unexpected filesystem changes should be reviewed to determine the cause.

Checking for uids of 0

By default there are two accounts with UID 0 on FreeBSD. One is the well-known "root" account. The other is the "toor" account. Any other accounts that show up in this looks need to be looked at very carefully to understand not only where they came from but why.

UID 0, regardless of the name associated with it, is the super-user account on UNIX and UNIX-like operating systems (including FreeBSD). The name is traditionally "root". The only thing that makes the account a super-user account is that it has UID 0.

It is perfectly fine to have more than one account with the same UID on FreeBSD although it is very rare to do so with anything other than UID 0.

The "toor" account is included with the base system to allow the system administrator to change the user shell for interactive sessions where full super-user authority is required. The root user shell should not be changed because so many automated tasks need to run as root and rely on the user-shell being the bourn shell that is included with the base system.

Some environments choose to remove the "toor" account from FreeBSD because it violates their internal security policy.

Checking for passwordless accounts

This provides a list of all local accounts that have a blank password set. This is distinct from having no password at all; which would prevent password login from working. No account should ever be created with a blank password.

Checking login.conf permissions

This checks to ensure that the permission on the /etc/login.conf file have not changed from the default. Anything in this section indicates corrective action needs to be taken.

The /etc/login.conf file defines system level permissions for users. Changes to this file will change system level access permission for one or more users. Attacks on login.conf are a common way to gain unauthorized administrative privileges or introduce vulnerabilities to a system. Only the root user should have read and write permission to this file (group and other should have read-only).

Checking for ports with mismatched checksums

This provides a list of entries in the ports tree (starting at /usr/ports) that have invalid checksums when compared to the default checksums. Anything showing up here should be considered a problem that needs to be addressed.

The ports subsystem is another alternative to the package subsystem (discussed in a previous post). The main difference is that the ports subsystem provides source code and the system administrator must compile the source before installing the software. The ports subsystem includes all the functionality needed to easily compile and install on the local system.

Hostname kernel log messages

This is an extract of all kernel generated messages in /var/log/messages since the last run. It is usually blank.

When a system is booted (or rebooted) there will be some kernel messages generated. Also when there are errors in the kernel, there will be generated. This can include problems identified by low level hardware drives.

Any kernel messages that are not associated with a system boot/reboot should be reviewed and understood because they could be an early indication of hardware problems.

Some messages from the boot/reboot process could also indicates problems. It is worthwhile becoming familiar with any kernel messages that show up.

Hostname login failures

This is a list of all login failures on the system. On servers it would be empty unless an administrator made a typing error. On systems that support end-users there will usually be a few mistyped passwords every day (depending on the end-user population). On Internet facing servers with login support there are bound to be many entries every day as the result of (hopefully failed) unauthorized login attempts.

Every mistyped password would be included for most system login mechanisms (including console login, PAM, ssh, and x11). The included sample (above) shows a snippet of what is generated when an ssh server is exposed to the Internet. This happened less than an hour after the ssh server was made accessible.

The example was truncated to keep it reasonable. There were actually 58 separate attempts to connect  over a period of 13 minutes. This would have been done by a simple script that just tries a bunch of different passwords.

The bottom section shows a different sort of attempt to break in. It was probably more sophisticated than the first since it gave up immediately.

Hostname refused connections

Provides a list of TCP refused connections. Internally this should be empty. If not, something may be misconfigured in your network. For Internet facing servers, it is yet another indication of the common attacks (such as port scans). Little can be done about the Internet side. It is a good reminder of the importance of firewalls.

Checking for packages with security vulnerabilities

Most days this will only include the date and time that the package database was update (which is done as part of this process). It may also contain a list of packages installed on your system that have known security vuilnerabilities. That indicates it is time to upgrade the identified packages.

Simply put: it is time to upgrade anything that shows up in this section as soon as you can. Especially on servers that are exposed to the Internet.


The security report offers a lot of valuable information. Some sections will contain a lot of detail for which there is no action to be taken. Other sections give warning signs but only once. It is far too easy to miss the important parts of the security report due to the overwhelming amount of detail provided. The security report is much better than no report at all (which is what you get with most operating systems). but the security report is not a good substitute for a proper security monitoring solution, even on small installations. 

Wednesday, March 25, 2015

Reading the FreeBSD Periodic Reports

FreeBSD is a free open source operating system that predates Linux. It is used globally to this day; mostly as a server and in a few appliances. The FreeBSD project is very active and focuses on stability and reliability of the system. You can learn more about FreeBSD from the FreeBSD Foundation website.

One of the many features that FreeBSD includes in the default installation is a set of maintenance and monitoring scripts that run periodically. Unfortunately, many administrators and hobbyists don't fully understand these reports and some aren't even aware of their existence. The reports are mailed to the local root user of the system and if the email subsystem on the server hasn't been configured they sit in the local mailbox.

The reports include:

  • A daily report showing general system health.
  • A security report that runs daily and highlights potential security concerns.
  • A weekly report of system health activities.
  • A monthly report of system login accounting.

These reports offer the system administrator a quick and easy way to monitor their systems with a quick daily glance at a few e-mails (per system). Obviously if you manage hundreds of servers you will want a more robust solution. I will not cover such options in this post.

The Daily Report

This is an e-mail with the subject line: Hostname daily run output (Hostname is the actual short name for the server). Here is an example:
Removing stale files from /var/preserve:
Cleaning out old system announcements: 
Removing stale files from /var/rwho: 
Backup passwd and group files: 
Verifying group file syntax: 
/etc/group is fine
Backing up mail aliases:
Backing up package db directory: 
Disk status:
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ada0p2     15G    3.3G     11G    23%    /
devfs          1.0k    1.0k      0B   100%    /dev
/dev/ada0p5    254G     79M    233G     0%    /data
fdescfs        1.0k    1.0k      0B   100%    /dev/fd
Network interface status:
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
re0    1500 <Link#1>      00:01:2e:bc:bc:6e   239879     0     0    25803     0     0    0
re0    1500  caliban             190041     -     -    19392     -     -    -
re0    1500 fe80::201:2ef fe80::201:2eff:fe        0     -     -        2     -     -    -
ath0*  2290 <Link#2>      e0:b9:a5:66:0c:80        0     0     0        0     0     0    0
usbus     0 <Link#3>                               0     0     0        0     0     0    0
usbus     0 <Link#4>                               0     0     0        0     0     0    0
usbus     0 <Link#5>                               0     0     0        0     0     0    0
usbus     0 <Link#6>                               0     0     0        0     0     0    0
plip0  1500 <Link#7>                               0     0     0        0     0     0    0
lo0   16384 <Link#8>                           75644     0     0    75644     0     0    0
lo0   16384 localhost     ::1                  75580     -     -    75580     -     -    -
lo0   16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -    -
lo0   16384 your-net      localhost               64     -     -       64     -     -    -  
Local system status:
3:01AM  up 4 days, 16:58, 0 users, load averages: 0.34, 0.08, 0.03 
Mail in local queue:
mailq: Mail queue is empty 
Mail in submit queue:
mailq: Mail queue is empty 
Security check:
   (output mailed separately)
Checking for rejected mail hosts: 
Checking for denied zone transfers (AXFR and IXFR): 
Backing up pkgng database: 
-- End of daily output --

Removing stale files from /var/preserve

The first section should always be blank. If anything is found in this section, it indicates that something has gone wrong. Most likely a daemon did not properly start on the last reboot.

It is the output of files that were found in /var/preserve and deleted as part of the job. The /var/preserve directory is intended as a place to save state data between reboots. Normally it would only be used by the operating system. A daemon will write a file to that directory just prior to shutdown and expects to read that same file when the daemon starts up after a reboot. The daemon is supposed to erase the file once read.

Cleaning out old system announcements

This section should also always be blank. If you aren't sure what system announcements are, odds are pretty good you don't use them and there wouldn't be any issues. If you do see something here and you aren't expecting it, investigation will be warranted.

System announcements are an antiquated method of sending a global message to end-users so that they will see it the next time they log-in to the system. It works through the mail sub-system. Announcements are displayed to the user when they log-in. The end-user will see each announcement only once. Since we rarely have end-users on UNIX systems that login to a interactive text shell, this system is rarely used nowadays.

Removing stale files from /var/rwho

As with the previous entries, this should be empty. If there are entries in this section, it indicates something has gone wrong with the rwho subsystem. Generally speaking nobody should be running the rwho subsystem anymore (there may be exceptions).

The rwho subsystem was used back in the days of interactive shells to query information about other users on other computers. You might think of it a little like Facebook from the very early days of the Internet.

Backup passwd and group files

This should be empty, too. If it isn't empty, it is worth investigating what happened.

The passwd and group files store the local user database and the local group (roles) database. If either one gets corrupted or changed unexpectedly, you can use the backup to restore the previous day's version of the files. So, if something goes wrong with the backup it is important to figure out what and fix it.

Verifying group file syntax

If this says anything other than "/etc/group is fine" it means that the group database is corrupt and needs fixing. The easiest fix is probably to restore the previous day's group file but odds are that a system administrator made changes to the file and messed up the syntax somehow. So, it may be worthwhile looking at the difference between the two and making appropriate corrections.

Backing up mail aliases

Similar to the passwd and group backup, this section should be blank. If it is not, the e-mail subsystem will probably not be functioning correctly and e-mail may get lost or misdirected.

The mail aliases file (/etc/aliases) is used to alter where mail for local users is sent. If something goes wrong, the easies fix is to copy the previous day's aliases file back into /etc.

Backing up package db directory

As with passwd, group, and aliases this should be blank. If it is not, something has gone wrong with the backup of the package subsystem.

The FreeBSD package subsystem is used for installing 3rd party software with binary distributions. The distribution repository is maintained by the FreeBSD foundation and updated regularly. The package system (man pkg) provides functionality for installation, upgrades, and removal of 3rd party software.

Disk status

This provides a summary of the mounted filesystems (the output of df -k). It is intended that the administrator glance at this to watch for unexpected changes such as missing filesystems or full filesystems.

Network interface status

This is the output of the netstat command. It is intended that the administrator glance at this to watch for unexpected changes in network status such as missing network interfaces or unexpected networks appearing.

Excessive (or in some cases any) changes to the numbers in Ierrs, Idrop, Oerrs, Coll, and Drop indicate network issues that may need to be addressed.

Local system status

This is the output of the uptime command. It shows you how long the system has been running since the last reboot.

Obviously if the system rebooted since the last report and the administrator wasn't expecting it, there may be a problem. The last 3 numbers are the system load (over 3 different time periods). Since the periodic script runs in the middle of the night, one would expect these to be pretty low. A value of 1 indicates the system is fully loaded in some way.

Mail in local queue

This should generally say "mailq: Mail queue is empty". If it does not, there is something preventing mail from being delivered locally and this should be examined.

It provides a count of the number of messages waiting in the sendmail subsystem queue for local mail delivery. That is mail that is being sent to a user on this system.

Mail in submit queue

If you are running a mail server, this will contain the number of e-mail messages that are queued up for delivery to other systems. It may not be empty in such cases. A large number would indicate a problem but that may or may not be a problem local to this system.

If you are not running a mail server, it may still contain the number of e-mail messages that are queued up for delivery to other systems (but presumably generated from this system). Odds are any value other than "mailq: Mail queue is empty" indicate some sort of a problem with delivery.

Note that these two preceding sections (Mail in local queue and Mail in submit queue) are written assuming that the sendmail daemon is used for the servers e-mail subsystem. Many alternatives provide work-a-like functionality and will result in this report being accurate (both Exim and Postfix do this). Some lightweight alternatives do not provide this functionality and it will be the administrator's responsibility to provide an alternate monitoring solution.

Security check

This only ever says "(output mailed separately)". The output will be found in the mail message with subject line "Hostname security run output" where "Hostname" is the name of the server that ran the script. That report will be covered in a later journal entry.

Checking for rejected mail hosts

If this is a mail server, there may be entries in this section. Excessive entries may indicate a problem with your setup that is causing other organizations mail servers to reject mail from your server.

This will be a list of e-mail that failed to be delivered because the receipent's e-mail server rejected the message. On mail servers, it is not unusual for there to be some entries here.

Checking for denied zone transfers

If this section is not empty, there is a problem with your network's DNS setup and investigation is warranted. Or, someone is trying to take a copy of your DNS Server's entire database without your authorization.

This is an extract of log messages from the DNS subsystem (usually bind) which indicate attempted zone transfers failed. There are several reasons why this could happen.

A zone transfer request from an unauthorized (and unexpected) source may be an early indication of a focused attempt to crack your network but it can also be simple casual curiosity. While it is important to not over-react it would be a good idea to pay attention to other sources of information about unexpected network activity.

Backing up pkgng database

The pkgng subsystem is a replacement for the package subsystem (discussed above). All the same comments from the earlier section apply here as well. As of version 9.3 of FreeBSD (possibly earlier and later as well) either may exist on the server but both should not be used at the same time.


That covers the content of the daily report. Regularly viewing and understanding this report can help an administrator catch problems before they are noticed by end-users.

I have seen people in corporate environments use the output of these reports to satisfy IT audit requirements and as evidence to support the need for more staff or to justify a better annual review with the boss. They provide not only the information needed to be a pro-active system administrator but also the evidence that you are being pro-active and stopping problems before people notice.

Saturday, March 14, 2015

Dual Root Upgrades

Performing operating system upgrades is always a risky task. Obvious issues like unrecognized hardware and incompatible application software. Less obvious risks like subtle bugs or incompatibility with support systems that might not be noted right away. In production environments these problems can lead to extended downtime and failure to stay within published maintenance windows. The dual-root architecture I developed to try and reduce these risks with a very simple recovery process.

The technique may be applied to most operating systems that run on general purpose computers. The remainder of this journal entry will summarize how I have chosen to implement it on FreeBSD.


  1. In addition to the normal filesystem partitioning scheme, add a second partition that is an identical size to the planned root partition. 
    • This must cover / and /boot and 
    • should cover /usr but 
    • may not need to cover /usr/local. 
    • Ideally it should not cover /usr/src or /usr/ports. 
    • It must not cover /home or /usr/home or any other user or application data directories.
  2. Tell the installer to use an alternate location for the root filesystem when upgrading.
    • for source this is done with the INSTALLROOT variable/option.
    • for bsdinstall do a custom install with the new root partition mounted on /mnt.
For reference when defining the size of the root partition(s):
  • FreeBSD 10.1-RELEASE base install from binary used 1.4GB including the ports skeleton
  • FreeBSD 9.3-RELEASE-p10 base, source, ports, and several packages used 3.5GB before compiling the source tree.
  • I currently use 16GB for each root partition. I would reduce this to 8GB each if I shared /usr/src and /usr/ports between the two.
  • In a production environment, I would probably share /usr/local as well to allow for separate upgrading of OS and applications.

Selecting the Root Partition

The FreeBSD boot loader (boot0) scans the partitions in order looking for /etc/fstab and assumes that is the root partition by default. At least, that is what is (or was) in the source code comments. Fortunately for us, the gpart utility allows us to set some attributes that override that default behaviour.

If we assume that our two root partitions are located on /dev/ada0p2 and /dev/ada0p3, then the boot loader will use /dev/ada0p2 by default all the time. To tell the boot loader to use /dev/ada0p3 we need to set a special attribute with the gpart utility:

gpart set -i 3 -a bootme /dev/ada0

and to unset it:

gpart unset -i 3 -a bootme /dev/ada0

for clarity sake, you may want to set the bootme attribute on the first partition. However, the boot loader will boot the first available partition that appears to be a root partition by default.

Tuesday, March 10, 2015

Shell Script Report Skelton

Over the years I have written and re-written a skeleton of a shell script that provide a bunch of basic program like functionality for bourn shell (sh) scripts. Mostly I use this as a sort of wrapper for Cron and At job. It provides:
  • Simple command line argument parsing.
  • Help (-h argument).
  • E-mail the standard out as a report.

Skeleton Shell Script

# Skeleton shell script
# As written the output is mailed to the person defined by the MAILTO line.
# This is primarily intended as a skeleton for cron jobs.
# Copyright (C) 2015, Ean Kingston, All rights reserved.

# Configuration Variables - These should be customized


# Internal Variables - these should not need to be changed
# If PATH is not set, set it to something sane
MYRPTFILE=${WORKDIR}${0%'.sh'}.$ # Temporary file for the report.

# Support subroutines

# Display Usage information for the script. This will need to be edited.
printhelp() {
cat <<EOT
Usage: $0 [-h] [-m={email}]
This is a skeleton shell script. This text should be replaced with usage
information for the completed script.
   -m Send the mail to an alternate address, blank for stdout.
   default is ${MAILTO}
   -h Display this help text.


# Start of Main code

# Proess command line arguements
for ARG ; do
   case $ARG in
      -[mM]=*) MAILTO=${ARG#-[mM]=} ;;
      -[hH]) printhelp ; exit ;;
      *) echo Unexpected command line arguement $ARG. Stopping. >&2 ; exit 1 ;;

# Take everything from stdout and put it in a report file



# Send (or print) the report
if [ -n "$MAILTO" ] ; then

# Cleanup temporary file(s)

Monday, March 9, 2015

Starting Fresh

It seams like every few years I try my hand at blogging. It can be cathartic to have somewhere to express your thoughts to the world and like so many others, I start a blog to do just that. May years ago I created the Random Thoughts of the Lost Admin blog and then, like so many, I stopped using it.

In fact I stopped so long ago that Blogger couldn't handle all the missing configuration bits and pieces from the many updates to the blogger engine and my old blog was no longer visible. I was able pull an XML dump of my old blog and I reviewed it to see if I wanted to re-upload it here but I've decided it just isn't worthwhile. Most of the topics are no longer relevant.

So, here is my blog reboot. We shall see how long this one lasts.