waiting for root device: could be duplicate UUIDs

Posted on by

If you run into waiting for root device when booting in verbose mode or you get a flashing question mark, etc, it could be that your volumes have duplicate UUIDs. The UUID is the primary means that is used to locate the boot volume to root from. If you have duplicate UUIDs, your Mac may not be able to determine which volume to boot from. If you suspect this may be the case, boot from another disk or CD and run diskutil list to get a list of all volumes. Then run diskutil info on each to get the UUIDs. Check for duplicates.

kserver:~ pbuffer$ diskutil list
/dev/disk0
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *74.5 Gi disk0
1: EFI 200.0 Mi disk0s1
2: Apple_HFS int_1 19.8 Gi disk0s2
3: Apple_HFS int_2 19.8 Gi disk0s3
4: Apple_HFS int_3 34.5 Gi disk0s4

kserver:~ pbuffer $ diskutil info disk0s2 | grep UUID
Volume UUID: A1F5866F-8286-359A-B19F-58910918AC5A
kserver:~ pbuffer$ diskutil info disk0s3 | grep UUID
Volume UUID: A1F5866F-8286-359A-B19F-58910918AC5A
kserver:~ pbuffer $ diskutil info disk0s4 | grep UUID
Volume UUID: CBB0B1F8-07D5-3BFC-9B50-4F99033B01D6

If you do find a duplicate, you can generate a new UUID and set it using the following command:

unmount the disk first

diskutil unmountDisk disk0

/System/Library/Filesystems/hfs.fs/hfs.util -s disk0s2

substitute your own device for disk0s2

Once you’ve done this, re-bless the volume.

Duplicate UUIDs shouldn’t happen, but somehow it happened to one of our Xserves with a hardware raid card.

No creation dates from stat

Posted on by

I was surprised to learn that creation dates are not included in stat output. I really thought stat would include something so basic. You get time of last access, time of last data modification, and time of last file status change, but creation date isn’t in there.

On the Mac, this info is stored in the resource fork. You can get it from the command line using GetFileInfo which is installed when you install Apple’s free Developer Tools.

/Developer/Tools/GetFileInfo -d path/to/file

In Applescript you can use info for:

creation date of (info for (choose file))

Notes on Leopard AD Plugin 10.5.2

Posted on by

The Active Directory plugin is finally usable in 10.5.2, but some environments require workarounds.

1.) Your domain must resolve to the ip address of a domain controller. This was not a requirement in previous versions, but Apple is apparently making it a requirement as they closed my bug stating that it was a configuration issue with Active Directory since creation of a domain sets up this dns info by default. If your domain does not resolve to an ip, you need to fix it, or as a workaround, edit your /etc/hosts file to point the ip of one of your domain controllers.

for example if you know you have a domain controller at 10.3.1.23 and your fully qualified domain is domain.forest.com, you’d add this line to /etc/hosts

10.3.1.23 domain.forest.com

2.) Allow Authentication from any domain in forest does not work. Uncheck this box in Directory Utility or using the corresponding flag in dsconfigad. If you don’t do this, the join may succeed, but you won’t be able to lookup or authenticate users or even use dscl on Active Directory. When you uncheck this option, just be sure to add the correct domains to your authentication search path in Search Policy of Directory Utiltity.

3.) Allow Administration by Active Directory Groups does not seem to work. In 10.4, this option seems to nest the AD group you want to allow for administration into the local admin group, so the workaround is to do the same in 10.5 manually using dseditgroup.

sudo dseditgroup -o edit -a “DOMAIN\group name” -t group admin

replacing DOMAIN\group name with your domain and group that you want to give admin access.

This group nesting method gives members of your AD group admin access for both Apple’s Authorization APIs and sudo.

These workarounds got me working, logins are painfully slow, but that may be due to the hosts hack.

Update: Under 10.5.3, most of these problems are resolved. If you are still having slow logins/joins, there are possible workarounds.

Bootable Software RAID 0+1, 1+0 in Mac OS X

Posted on by

Software RAID striped mirrors or mirrored stripes are possible in Mac OS X. They may be 10.5 only. I haven’t tested other versions.

Simply create your striped sets using DiskUtility or diskutil, then use the command line diskutil and supply the /dev/diskx entries of the stripe sets when creating your mirror.

First run diskutil list to get the /dev/disk entries for each stripe, then create your mirror using:

diskutil createRAID mirror MirrorName JHFS+ disk1 disk2

where disk1 is the device for the first stripe set and disk2 is the device for the second stripe set.

The result is also bootable and will display correctly as a nested RAID in DiskUtility.app.

convert lines to comma separated items with tr

Posted on by

If you have output that is separated by new lines, but you really want it formatted into a single line with commas as separators or maybe a space as a separator, just pipe to tr.

Here is a simple example:

kserver:~ patternbuffer$ diskutil list | grep ^/dev
/dev/disk0
/dev/disk1

If we want them separated by spaces, we could do:

kserver:~ patternbuffer$ diskutil list | grep ^/dev | tr '\n' ' '
/dev/disk0 /dev/disk1

Note that there is a newline on the end of the output, so that trailing newline is also translated. So if you don’t want it there, you’ll have to chomp it off. I use sed, but use what you like.

Here is the same as above, but with commas, with sed to remove the trailing comma.

kserver:~ patternbuffer$ diskutil list | grep ^/dev | tr '\n' ',' | sed 's/.$//'
/dev/disk0,/dev/disk1

No ACL granularity for Extended Attributes

Posted on by

Recently I thought I had a great use for storing extended attributes. I needed to store the time a file had been updated by my file management system without effecting the actual file modifications times. Storing this information directly in the file metadata would ensure that it would not become out of sync with some external database tracking this information.

I wanted this data to be modifiable only by a specific user or group. Unfortunately, setting an ACL on extended attributes applies to all extended attributes. I didn’t want to disallow users from setting their own extended attributes on these files and folders, so I abandoned the idea.

I wonder if we’ll ever be able to set permissions on specific attributes.

rsync 3.0 looking very promising

Posted on by

A little while back, I mentioned the sad state of rsync in Mac OS X. Well I’ve had a chance to test rsync 3.0 prerelease 7 and I am very pleased with the results.

To test for yourself, download the source, extract the source, cd into the extracted directory, run ./configure, run make, and then make install. You’ll find the new rsync installed at /usr/local/bin/rsync. View the man page using man -M /usr/local/share/man rsync.

Use -X for extended attributes ( and resource forks )
Use -A for ACLs
Don’t use -E, which was Apple’s flag for these as it is used for executability

Here are my test results:

Resource Fork Handling: For testing resource forks, I used to keep a copy of SimpleText around as it is all resource fork and no data fork, but a text clipping, which is also all resource fork is a great test. rsync 3 syncs the resource fork exactly the way you’d like. It retains the fork during the initial copy, and only re-copies the file if the fork has changed. This is quite awesome since Apple’s bundled rsync always copies the fork causing major speed hits even when no files have changed. I haven’t looked at the source code to see how rsync determines if the resource fork changed, but it seems efficient so far, though they may be doing checksums because using the -X flag adds time to the compare phase.

I dragged a text clipping into my source folder. I verified its resource fork using cat somefile/rsrc. I ran rsync and verified the destination’s resource fork. I re-ran rsync and no files were copied. I appended some data to the fork and rsynced again. rsync detected the file as changed and re-copied it with the changed resource fork. You can test it yourself. Using the –stats or -v option, try running Apple’s rsync, then rsync 3 and you’ll see the number of files transferred.

Metadata tests: I recently found backup bouncer, which runs a multitude of tests on metadata preservation.

------------------ rsync-3.0pre7 ------------------
This copier exited with error code 23
This copier produced log output in:
/Volumes/Dst/11-rsync-3.0pre7/log
Verifying: basic-permissions ... ok
Verifying: timestamps ...
Sub-test: modification time ... ok
ok
Verifying: symlinks ... ok
Verifying: symlink-ownership ... ok
Verifying: hardlinks ... ok
Verifying: resource-forks ... ok
Verifying: finder-flags ... ok
Verifying: finder-locks ... FAIL
Verifying: creation-date ... FAIL
Verifying: bsd-flags ... ok
Verifying: extended-attrs ...
Sub-test: on files ... ok
Sub-test: on directories ... ok
Sub-test: on symlinks ... ok
ok
Verifying: access-control-lists ...
Sub-test: on files ... ok
Sub-test: on dirs ... ok
ok
Verifying: fifo ... FAIL
Verifying: devices ... FAIL
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... ok
Sub-test: lots of metadata ... ok
ok

Overall it fares pretty well.

The lack of support for finder-locks (aka flags like uchg) is unfortunate. Files on the destination are simply unlocked. This solves the problem of rsync not being able to delete locked (uchg) files, but it is kind of worse because that metadata is just plain lost and a lot of old time Mac users are accustomed to using that feature since it is right there in the GUI Finder Get Info panel.

I suppose a workaround would be to find all the files that have the flag and create a transcript of them before running rsync for later re-locking.

I need to do some performance testing compared to rsyncx, but so far, rsync 3 looks good to me.

Finding your Active Directory Site and Domain Controllers

Posted on by

The AD Plugin uses information in Sites in your Active Directory Configuration to get a list of Domain Controllers to use for LDAP and Kerberos connections. Sites are a method of configuring Active Directory based on a physical or network based location. Clients connecting to Active Directory use this information to determine which domain controllers are nearby and therefore likely to respond the fastest. Plus if you have Domain Controllers in remote or distant locations, without using Sites, you are pushing traffic over a WAN unnecessarily.

This is one of the steps you’ll see when joining a machine to AD. Locating Domain Controllers… or something like that.

First it uses DNS to lookup SVR records to locate ANY domain controller. You can do the same using the dig command:

dig any _kerberos._tcp.yourdomain.yourforest.com

This will give you a list of domain controllers to choose from. Once you have one of your domain controllers to talk to, you can read the same information that the AD Plugin does to figure out your Site by querying AD with ldapsearch and your subnet:

ldapsearch -x -h "somedomaincontroller.yourforest.com" -b "CN=Subnets,CN=Sites,CN=Configuration,dc=yourforest,dc=com" -D "username@yourdomain.yourforest.com" -W "(cn=10.31.0.0/16)" siteObject

From this you will get your Site, which you can then use with ldapsearch again to get a list of domain controllers for your site:

ldapsearch -x -h "somedomaincontroller.yourforest.com" -b "CN=SERVERS,CN=YOURSITENAME,CN=Sites,CN=Configuration,dc=yourforest,dc=com" -D "username@yourdomain.yourforest.com" -W dNSHostName | grep dNSHostName

Pretty cool. Takes some more of the mystery out of the AD plugin.

Notes: Enter each command as a single line. Substitute ‘yourdomain’ with your own domain and ‘yourforest’ with your own forest and ‘username’ with a your username for your domain and ‘10.31.0.0/16’ with your own subnet and ‘YOURSITENAME’ with the site name you found in the previous step. Also, using ldapsearch in this way does a simple BIND authentication which sends your password in CLEAR TEXT. Change your password after using this command if you are in a secure environment.

Update: I was using this method recently and realized I was getting domain controllers that were in my site, but weren’t hosting my specific domain. If you want to get just the servers hosting your domain, you need to filter the ldapsearch like so, but you won’t get the fully qualified dNSHostName:

ldapsearch -x -h "somedomaincontroller.yourforest.com" -b "CN=SERVERS,CN=YOURSITENAME,CN=Sites,CN=Configuration,dc=yourforest,dc=com" -D "username@yourdomain.yourforest.com" -W msDS-HasDomainNCs="DC=subdomain,DC=forest,DC=com" dn | grep dn:

A history of rsync on Mac OS X

Posted on by

I think Apple included the rsync binary with Mac OS X since at least version 10.2 Jaguar, but it couldn’t be used for backups of most Mac data due to its lack of support for resource forks, which were and are prevalent on the Mac.

Rsync first got attention on Mac OS X when Kevin Boyd of University of Michigan added support for resource forks to rsync with a port he called RsyncX. RsyncX also included a simple GUI for setting up backups. The GUI generated shell scripts and scheduled them in cron. The GUI was buggy and generated scripts with minor, but serious flaws. The scripts also lacked some necessary features that a backup solution would provide. But with the lack of a good backup solution (just say no to Retrospect), some admins including myself took the time to customize and extend the scripts that RsyncX created, or just created new ones from scratch. I added features to my scripts like checking for free disk space, rotating incremental backups with hard links, checking for a mounted volume before running, failure notifications, etc.

RsyncX had other flaws though. It was based on older rsync code which contained bugs and at least one serious security vulnerability. One of the bugs results in rsync getting stuck in an endless loop if files are changing while rsync is running. If you have logging enabled, rsyncx will quickly fill your drive while it is stuck in this loop — really bad news. My workaround for this was to spawn rsync off and watch its log every few seconds for telltale signs that it was stuck in a loop. I would then kill rsync and restart it. Ugly, but it actually works. Oh yeah, it also doesn’t know how to handle locked files (uchg), so I have to unlock all files on the destination before the sync. Ugh. Oh and it throws lchown errors on symlinks. Just grep -v them out.

When Apple began talking about the features of Mac OS X 10.4 Tiger, one of the big features for command line unix geeks was support for resource forks in all command line tools, including rsync. By this time those of us using RsyncX were getting pretty tired of all of the workarounds and bugs, so an Apple supported rsync that handled resource forks was awesome news.

When rsync was released with 10.4.0, it has serious bugs. It was basically unusable. It would crash unexpectedly with large data sets. It would incorrectly set modification dates, which would then cause all future syncs to re-copy all files. I stayed with rsyncx to handle production syncs with the additional slap in the face of deprecated warnings filling the system log. With each new release, I would test Apple’s included rsync. I filed bugs. I used a developer support incident. Finally around 10.4.9 and later, rsync did actually seem to work, at least well enough. In real world tests though, Apple’s rsync speed was and is dismal on large data sets. The problem lies in the fact that resource forks do not have a modification date to compare when syncing. Without this key piece of data, there is no easy way of knowing whether the resource fork of a file has changed or not. Apple’s solution to this problem was to ALWAYS COPY the resource fork. That’s right, if your data has resource forks, you copy the resource data every time rsync runs. Yes, resource forks are typically small, but they add up, and for terabytes of small files, the I/O causes an rsync of unchanged data that should take about 30 minutes to instead take 4 hours.

So again I stick with RsyncX.

Leopard’s rsync appears to be virtually unchanged from Tiger’s at least to me. It still copies the resource fork every time.

Mike Bombich apparently likes rsync too. He is using it for the sync engine in Carbon Copy Cloner 3. He includes an updated patched version in the bundle here: Carbon\ Copy\ Cloner/Carbon\ Copy\ Cloner.app/Contents/Resources/ccc_helper.app/Contents/Resources/rsync

One nice thing about this binary is that it has a patch to optionally checksum resource forks (–ea-checksum) to prevent them from being copied unnecessarily. Cool, but checksumming adds time, so I’ll need do some real world tests.

Rsync version 3.0 is in prerelease and includes built-in ACL and extended attribute support. Hopefully this includes some way of handling unchanged resource forks.

Until Apple adds some sort of date stamp to resource forks, checksums may be the only safe way to handle them. I suspect that addressing the issue properly will have to wait for a new filesystem.

Installing Leopard on Unsupported Hardware

Posted on by

I installed Leopard on two older machines that are not officially supported by Apple.

In both cases I resorted to installing from a supported machine with the machine in target disk mode or asr restoring to the drive in the unsupported machine. There may be other ways of getting it installed. I looked at the javascripts that check system requirements in the .mpkg in /System/Installation/Packages of the installer DVD using PackageMaker.app. It is possible that one could edit the OSInstall.mpkg javascripts and build a new installer DVD, but for me it was easier to install from a supported machine.

Results:

PowerMac G4 533Mhz 354MB RAM: This machine works okay with 10.5. It is somewhat slow with graphics and the low memory causes it to hit the disk a lot. The most serious issue is waking from sleep, which results in a kernel panic. I tried adding more RAM, removing SCSI card, disconnecting an older drive, but the kernel panics continued. I’ve set energy saver to never put the machine to sleep for now.

iMac G4 1Ghz 256MB RAM: This machine only has half of the required 512MB RAM for 10.5. It seems to run fine, but swaps to disk a lot.

When spotlight first indexes the drive on first boot, the machines crawl because the disk is so busy with swapping and indexing, but once that is finished, the machines are usable.