-
Notifications
You must be signed in to change notification settings - Fork 0
Building a Microkernel ISO
The Microkernel used by Razor is actually a variant of the Tiny Core Linux (TCL) kernel.
The Microkernel is built using the 'Core' release of TCL, which can be found here. In order to run the tools that are needed by the Microkernel (Facter for discovery), a number of TCL Extensions need to be added to our Microkernel.
Building the Microkernel creates a "bundle" - a single gzipped tarfile - that contains an 'overlay file' (another gzipped tarfile), as well as the scripts and dependencies are needed to merge that overlay file with the 'standard' TCL Core ISO (which is also included in the bundle file) in order to build a new version of the Razor Microkernel ISO.
The components for the overlay file (and the dependencies that it needs) are all either copied over from the Razor-Microkernel project or downloaded from online sources to local directories (if they are not part of the Razor-Microkernel project) before being included in the bundle file.
Building a new ISO image is reasonably simple, but there are a few things you should probably consider before you get started:
If you will be rebuilding the bundle file for the ISO image more than once, it really pays to have a standard HTTP proxy in place to cache the files you are downloading. This is an enormous speedup for the second and subsequent runs.
The "out of the box" configuration of squid and other proxies is typically fine; you are dealing with binary files no larger than 12MB in general.
You will need to be on a Linux system to rebuild the ISO image, but you can download the bundle file more or less anywhere that has POSIX-ish tools.
You will need to run the rebuild as root, or to use fakeroot. In the later
case you also need the 7z executable on the system doing the rebuilding -
check for p7zip-full or p7zip in your distributions.
We strongly recommend the rebuild is done without real root privileges, but they are both officially supported.
The standard bundle can be built by running this command:
./build-bundle-file.sh \
-b additional-build-files/builtin-extensions.lst \
-m additional-build-files/mirror-extensions.lst
Don't forget to pass --build-prod-image or --build-debug-image if you want
something other than a "development" build of the ISO image.
The script that is used for this process (the 'build-bundle-file.sh' script) can be found in the top-level directory of the Razor-Microkernel project itself. The usage for this script is readily available by adding a '-h' (or '--help') flag to the command line when invoking the script:
test@server:~/Razor-Microkernel$ ./build-bundle-file.sh -h
Usage: ./build-bundle-file.sh OPTIONS
This script builds a gzipped tarfile containing all of the files necessary to
build an instance of the Razor Microkernel ISO.
OPTIONS:
-h, --help print usage for this command
-b, --builtin-list=FILE file containing extensions to install as builtin
-m, --mirror-list=FILE file containing extensions to add to TCE mirror
-p, --build-prod-image build a production ISO (no openssh, no passwd)
-d, --build-debug-image build a debug ISO (enable automatic console login)
-t, --tc-passwd=PASSWD specify a password for the tc user
Note; currently, the default is to build a development ISO (which includes the
openssh.tcz extension along with the openssh/openssl configuration file changes
and the passwd changes needed to access the Microkernel image from the command
line or via the console). Also note that only one of the '-p' and '-d' flags
may be specified and the '-t' option may not be used when building a production
ISO (using the '-p' flag).
test@server:~/Razor-Microkernel$
When invoking this script, the '-b' and '-m' flags (or their more verbose equivalents) MUST be provided (if they are not, then an error will be thrown and the command usage will be printed), and the files that are included as arguments to these two flags must be readable. These files should contain a list of builtin extensions and local TCE mirror extensions (respectively) that should be downloaded and included in the overlay file we are creating. The builtin extensions are those that should be installed as part of the boot process for the Microkernel (making them available when tasks like network initialization take place, for example), while the TCE mirror extensions are simply placed into a local TCE mirror in the Microkernel's filesystem (where they can be used as part of the Microkernel Controller initialization process, post-boot). Examples for these two files are included in the 'additional-build-files' subdirectory in the Razor Microkernel project (the files are called 'builtin-extensions.lst' and 'mirror-extensions.lst', respectively). The other arguments to the script are optional, and default values are used if they are not specified (by default, the previous downloads are not re-used and the system builds a development Microkernel, not a production one). A simple example of the usage for this shell script that uses the default lists of extensions provided as a part of the Razor-Microkernel project is as follows:
./build-bundle-file.sh -r -b additional-build-files/builtin-extensions.lst \
-m additional-build-files/mirror-extensions.lst -d -t test1234 2>&1 | tee t.t
The 'build-bundle-file.sh' script must be run from the top-level directory of the Razor-Microkernel project. When it is invoked, this script will first create a directory structure that looks something like the following:
tmp-build-dir/
└── build_dir
Here, 'tmp-build-dir' is the working directory created by the script and the 'tmp-build-dir/build_dir' subdirectory is the directory we will be using to create the final bundle. This 'bundle' file will contain the overlay file itself (in the 'razor-microkernel-files.tar.gz' file that is placed into the 'tmp-build-dir/build_dir/dependencies' directory), along with all of the files and scripts that are needed in order to turn that to turn that overlay file into a working Razor Microkernel instance.
Since the build-bundle-file.sh script downloads a number of dependencies from the network, many of them from a rather slow (and bandwidth-limited) external server, the bundle build process can take quite some time. Previously, we had an additional flag to this script that could be set to "reuse the existing downloads", but use of that flag could often lead to difficulties that resulted from the "old content lingers" problem. As such, that additional flag was removed from the bundle build script. To obtain the same "instant bundle build" effect that the "reuse existing downloads" flag used to give, we now recommend the use of a local caching web proxy.
In our own testing (and in the builds that are made using the CI environment at Puppet Labs), the caching web proxy that is we use is the Squid caching proxy. Given the complexity of configuring a local squid install, and given the usefulness of this as an option for Microkernel developers, we thought that we would quickly go through the configuration parameters that we had to change in our own local environment in order to get a local Squid proxy configured such that it was able to deliver the "instant bundle build" effect that we've found so useful in the past.
First, some assumptions. We're going to assume that you know where to go to get a copy of the Squid proxy (either in source-code form or a binary distribution), and that you've managed to install Squid locally on the machine that you'll be using to build the bundle file. If you have difficulties with this part of the process, there are a number of resources online that can help you with your particular distribution. We're also going to assume that you are comfortable with managing packages that are installed as services in your particular distribution. The build environment we'll be using as an example here will be an Ubuntu Server environment running a squid3 server, and that squid3 server was installed from the binary package available in the standard Ubuntu repositories using the apt-get command. If your environment is different, then you may have to adjust some of the pathnames shown in this example to reflect the preferred locations of the files in question for your own environment. In particular, if you build the Squid caching proxy from source, many of these paths may have to be changed to reflect the paths that you provided when you configured your build.
With those qualifying statements and assumptions out of the way, let's talk about the changes (there were only two of them necessary) that we had to make to the default Squid configuration in order to get a local cache that would cache all of the dependencies needed for the bundle build process. First, we had to change the caching mechanism that is used by the squid server. By default, the squid server uses an in memory cache, and this has a couple of downsides to it. First, if we are to support a cache of any significant size, we'll have to provide the squid server with a lot of memory (which might be problematic in some environments). Without that memory, the OS will start paging and we'll end up using the disk to manage our cache (in a none-too-efficient way). Second, any downloaded contents that are cached locally in an in-memory cache will be lost when the squid server is restarted. If we want to have our cache persist across reboots (or across restarts of the squid server) then we'll have to change the squid server so that it uses the local filesystem for caching. Fortunately, this is an easy change that can be made in the configuration file using the cache_dir configuration parameter.
Next, we had to modify the default configuration so that the squid server would cache larger objects than it typically caches by default. In particular, there are two files that are downloaded by the build-bundle-file.sh script (the Core-current.iso and perl5.tcz files) which are larger than the default (4MB) maximum object size that the squid3 server will cache (those two files are 8MB and 13MB in size, respectively). Fortunately, the maximum object size that will be cached is also easy to change, and this is accomplished using the maximum_object_size configuration parameter (which we reset to a much more reasonable value of 32768 KB.
So, with those two changes in place, this is what our squid configuration looks like:
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access deny all
http_port 3128
cache_dir ufs /var/spool/squid3 500 16 256
maximum_object_size 32768 KB
coredump_dir /var/spool/squid3
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern (Release|Packages(.gz)*)$ 0 20% 2880
refresh_pattern . 0 20% 4320
(Note, that only the cache_dir ufs /var/spool/squid3 500 16 256 and maximum_object_size 32768 KB lines have been changed from the values contained in the default configuration)
With these two changes in place, we simply had to stop the squid3 server (via a /etc/init.d/squid3 stop command), build out the directories needed for the squid3 disk-based cache (via a /usr/sbin/squid3 -z command) and start out squid3 server back up (via a /etc/init.d/squid3 start command). Once the server was restarted, it only took a small change to how we invoked the build-bundle-file.sh script in order to take advantage of our new caching proxy. Instead of invoking the script the way we used to:
./build-bundle-file.sh \
-b additional-build-files/builtin-extensions.lst \
-m additional-build-files/mirror-extensions.lst
We just define an HTTP Proxy to use for the wget commands used to download the bundle file dependencies from the web (either as an environment variable, or directly on the command line if your login shell supports it). Here we show an example of defining the variable on the command line when invoking the bundle build script:
http_proxy='http://127.0.0.1:3128' ./build-bundle-file.sh \
-b additional-build-files/builtin-extensions.lst \
-m additional-build-files/mirror-extensions.lst
With this small change, we successfully cached all of the dependencies downloaded by our build-bundle-file.sh script, and the result was that all successive builds of bundle files (after the first) were significantly faster (taking only a few seconds to build with the caching proxy, rather than nearly 10 minutes without the caching proxy, in our test build virtual machine).
Finally, it should be noted that this approach is significantly better than the previous approach to this same problem. By simply re-using whatever was downloaded previously (in our old implementation), we were losing the benefits of whatever bug fixes may have been made in newer versions of the dependencies that had already been downloaded. The Squid proxy (or any other "real" caching proxy) will automatically download new versions of these files as they become available.
It should be noted that the sub-directories of the 'tmp-build-dir' directory (with the exception of the 'build_dir' subdirectory) will eventually be mapped into the root-level directories in the overlay file that we are building here (eg. the 'tmp-build-dir/usr/local/bin' directory in the local filesystem will correspond to the 'usr/local/bin' directory in our overlay). We should also note that the final directory structure that results from this build process will be left in place at the completion of the bundle build process (so that the downloads that the directory structure contains can be re-used, if necessary, to build another bundle (with additional extensions?).
After creating the initial 'build_dir' subdirectory (and the 'build_dir/dependencies' subdirectory), the 'build-bundle-file.sh' script copies over the scripts that will be needed to build the Razor Microkernel ISO directly from the 'iso-build-files' subdirectory of the Razor-Microkernel project. These scripts are placed directly into the 'tmp-build-dir/build_dir' subdirectory. If we are building a production ISO, the copy of the 'rebuild_iso.sh' script that is made via this process will also be modified (on the fly using 'sed') so that it outputs an ISO file with the string 'prod' in its name (rather than the string 'dev') when it is invoked. The next step in the process is to copy over the modified DHCP client scripts from the Razor-Microkernel project to the corresponding locations in the 'tmp-build-dir'. After that, the script copies over the Razor Microkernel Controller script (and its dependencies) into the appropriate locations in the 'tmp-build-dir/usr/local/bin' and 'tmp-build-dir/usr/local/lib/razor_microcontroller' subdirectories. Finally, the script copies over the list of gems from the 'opt/gems/gem.list' file of the Razor-Microkernel project into the 'tmp-build-dir/opt/gems' subdirectory. With that step, the process of copying files from the Razor Microkernel project into the directories that we will be creating our overlay from is complete and we are ready to start downloading some dependencies.
The download process starts by using the list of gems that was copied over from the Razor-Microkernel project (above) to determine what gems should be downloaded (so that they will be included in the overlay file). The gems in that list are downloaded to the 'tmp-build-dir/opt/gems' directory using the 'gem fetch...' command (Note; this assumes that the local system has RubyGems already installed on it and that the 'gem' command is accessible by the user running this shell script). With the gems in place, the script then downloads two sets of extensions from the TCL Extension Repository (or TCE repository). The first set of extensions downloaded are used to construct a local TCE mirror, and the second set are placed into a subdirectory from which they will automatically be installed during the boot process. The actual extensions that are installed are read from flat files containing the list of extensions to include in the local mirror and a list of extensions to set up as 'built-in extensions' (which appear in the 'additional-build-files/mirror-extensions.lst' and 'additional-build-files/builtin-extensions.lst' files in the Razor-Microkernel project, respectively). Files from the first list will be placed into a mirror subdirectory under the 'tmp-build-dir/tmp/tinycorelinux' subdirectory, while those from the second will be placed into the 'tmp-build-dir/tmp/builtin' subdirectory. In the second case, the list of extensions that is loaded is also used to construct a 'tmp-build-dir/tmp/builtin/onboot.lst' file that will be used by the Microkernel to determine which of the 'built-in' extensions should be loaded on boot.
After the 'bundle initialization' process is complete, the directory structure will actually look a bit more complicated (currently, this is what it looks like after initializing the temporary build directory but before building the bundle):
build-files
└── razor-microkernel-bundle-dev.tar.gz
tmp-build-dir
├── build_dir
│ ├── add_version_to_mk_fs.rb
│ ├── build_initial_directories.sh
│ ├── build_iso_yaml.rb
│ ├── Core-current.iso
│ ├── dependencies
│ │ ├── mcollective-setup-files.tar.gz
│ │ ├── mk-open-vm-tools.tar.gz
│ │ ├── razor-microkernel-overlay.tar.gz
│ │ └── ssh-setup-files.tar.gz
│ └── rebuild_iso.sh
├── etc
│ ├── init.d
│ │ └── dhcp.sh
│ ├── inittab
│ ├── passwd
│ └── shadow
├── mcollective-2.0.0.tgz
├── opt
│ ├── bootsync.sh
│ ├── gems
│ │ └── ...
│ └── rubygems-1.8.24.tgz
├── tmp
│ ├── builtin
│ │ ├── onboot.lst
│ │ └── optional
│ │ └── ...
│ ├── first_checkin.yaml
│ ├── mk_conf.yaml
│ └── tinycorelinux
│ ├── 4.x
│ │ └── x86
│ │ └── tcz
│ │ └── ...
│ └── ...
├── usr
│ ├── local
│ │ ├── bin
│ │ │ ├── lscpu
│ │ │ └── ...
│ │ ├── lib
│ │ │ └── ruby
│ │ │ └── 1.8
│ │ │ └── razor_microkernel
│ │ │ └── ...
│ │ ├── mcollective -> /usr/local/tce.installed/mcollective-2.0.0
│ │ ├── sbin
│ │ │ └── sfdisk
│ │ └── tce.installed
│ │ └── mcollective-2.0.0
│ │ └── ...
│ ├── sbin
│ │ └── dmidecode -> /usr/local/sbin/dmidecode
│ └── share
│ └── udhcpc
│ └── dhcp_mk_config.script
└── util-linux.tcz
With this directory structure in place, building the actual bundle is a very simple process (consisting of one command in the 'build-bundle-file.sh' script). When this process is complete, a single bundle file will be added to the 'build-files' subdirectory of the Razor-Microkernel project (named 'razor-microkernel-bundle-dev.tar.gz' for a development bundle and 'razor-microkernel-bundle-prod.tar.gz' for a production bundle). That bundle file will contain all of the scripts and files that are needed to build a new Microkernel ISO from the current Razor-Microkernel project (and the latest versions of its dependencies). This is what that current contents of a development bundle look like:
-rw-rw-r-- tjmcs/tjmcs 8290304 2012-06-04 22:24 Core-current.iso
-rwxrwxr-x tjmcs/tjmcs 1385 2012-05-02 13:55 add_version_to_mk_fs.rb
-rwxrwxr-x tjmcs/tjmcs 1263 2012-06-19 16:50 build_initial_directories.sh
-rwxrwxr-x tjmcs/tjmcs 2501 2012-05-02 13:25 build_iso_yaml.rb
drwxrwxr-x tjmcs/tjmcs 0 2012-06-21 16:27 dependencies/
-rw-rw-r-- tjmcs/tjmcs 30176283 2012-06-21 16:27 dependencies/razor-microkernel-overlay.tar.gz
-rw-rw-r-- tjmcs/tjmcs 813074 2012-06-08 14:19 dependencies/mk-open-vm-tools.tar.gz
-rw-r--r-- tjmcs/tjmcs 5723 2012-06-07 19:45 dependencies/ssh-setup-files.tar.gz
-rw-r--r-- tjmcs/tjmcs 374 2012-06-06 15:30 dependencies/mcollective-setup-files.tar.gz
-rwxr-xr-x tjmcs/tjmcs 1414 2012-05-02 13:46 rebuild_iso.sh
To use this bundle file, simply copy it over to another directory on this (or another) machine and unpack it
# tar zxvf razor-microkernel-bundle-dev.tar.gz
From that same directory, the sequence of commands to build a new ISO is actually quite simple. First, build the directory structure needed to for the script that constructs the new ISO:
# ./build_initial_directories.sh
then, invoke the script that actually builds the ISO
# ./rebuild_iso.sh [VERSION_NO]
Note; there is a single (optional) argument to this second script, the version number for the Microkernel ISO that you are building. If specified, three-digit (eg. 0.9.3) or four digit (0.9.3.1) version number should be used for this argument. If this optional argument is left out, then a version number will be constructed for you based on the version of the Razor-Microkernel project that was used to build the bundle file that contained this script (for those who are really interested, this auto-generated version number will be constructed using the latest tag applied to the Razor-Microkernel project and the commit information for any commits that have been made since the project was tagged with that tag).
Once the second shell script command finishes, you should have a new Razor Microkernel ISO build in the current working directory; an ISO that is suitable for use with Razor and that contains all of the dependencies and extensions included in the bundle build process (above). If, for some reason, you find an issue with the ISO you have built, rebuilding a new ISO is as simple as running the same pair of shell scripts that are shown, above; perhaps with a new set of built-in and/or additional extensions declared during the initial bundle build process.
There are a few dependencies that must be met for this build process to run successfully. The bundle file build process (described above) relies on the 'unsquashfs' command to extract the 'lscpu' and 'sfdisk' commands from the 'util-linux.tcz' extension that it downloads from the standard Tiny Core Linux Extension Mirror. That same script also makes extensive use of the 'wget' command to download the dependencies that it needs and uses the 'gem' command to 'fetch' the gems that are needed from the 'Ruby Gems' gem repository. Finally, the dynamic downloading of the dependencies and extensions that need to be included in the overlay file that is being constructed require that an active network connection exists (with access to the internet) and that the local Razor-Microkernel project directory be writeable by the current user. If any of these dependencies are not met, the bundle file build process will fail.
The script that builds the Microkernel ISO also has a number of dependencies on system level commands that may or may not exist on all platforms (and that may or may not be accessible for all users on any given platform). It uses the 'chroot' and 'ldconfig' commands to set up the contents of the 'tmp' subdirectory that it uses when building the 'core.gz' file that will be placed into the Microkernel ISO. If these commands are not accessible (or are not usable by the current user), then the resulting ISO may or may not be usable as a boot image by Razor. This script also uses the 'cpio' command to construct the 'core.gz' file and uses the 'advdef' command to further compress that file before placing it into the Microkernel ISO. We have seen issues on some platforms with the flags for these commands not being consistent (or with these commands not being available at all). The build process has been successfully tested and used under recent releases of Ubuntu (the 64-bit server release, but that's probably not important), but failed for some users who were trying to use the same scripts under OS X. The Ubuntu Server instance we have used most extensively uses the GNU cpio command and advancecomp v1.15 (for the 'advdef' command). This build process hasn't much testing on other platforms to date, so your mileage may vary in terms when it comes to using this build process (depending on the platform that you use).
By following the procedure outlined, above, we were able to create 'bundle file' containing an 'overlay' that can be used to add a number of extensions to the default TCL Core distribution. We were then able to use that 'bundle file' (the 'overlay' along with the additional dependencies and scripts that it contains) to build a new Microkernel ISO from the standard TCL Core distribution ISO.
The current 'build-bundle-file.sh' script can be found here. For convenience, we've also provided a nearly complete view of the contents of the 'tmp-build-dir' directory for reference, below. This view shows what this directory structure looks like after the 'build-bundle-file.sh' script has been run, and can be used to determine what is included in the bundle file itself. We say "nearly complete" (above) because we've truncated the actual directory contents a bit (we are only showing the 'tcz' files for the extensions, not the associated md5, list, info, and dep files, and we've cut out all of the files and subdirectories under the 'tmp-build-dir/usr/local/tce.installed/mcollective-2.0.0' directory). Other than the contents that we are not showing, the rest of the structure is intact.
Followed by the post-build directory structure:
tmp-build-dir/
├── build_dir
│ ├── add_version_to_mk_fs.rb
│ ├── build_initial_directories.sh
│ ├── build_iso_yaml.rb
│ ├── Core-current.iso
│ ├── dependencies
│ │ ├── mcollective-setup-files.tar.gz
│ │ ├── mk-open-vm-tools.tar.gz
│ │ ├── razor-microkernel-overlay.tar.gz
│ │ └── ssh-setup-files.tar.gz
│ └── rebuild_iso.sh
├── etc
│ ├── init.d
│ │ └── dhcp.sh
│ ├── inittab
│ ├── passwd
│ └── shadow
├── mcollective-2.0.0.tgz
├── opt
│ ├── bootsync.sh
│ ├── gems
│ │ ├── daemons-1.1.8.gem
│ │ ├── facter-1.6.9.gem
│ │ ├── gem.list
│ │ ├── json_pure-1.7.3.gem
│ │ └── stomp-1.2.2.gem
│ └── rubygems-1.8.24.tgz
├── tmp
│ ├── builtin
│ │ ├── onboot.lst
│ │ └── optional
│ │ ├── bash.tcz
│ │ ├── dmidecode.tcz
│ │ ├── firmware-bnx2.tcz
│ │ ├── gcc_libs.tcz
│ │ ├── libssl-0.9.8.tcz
│ │ ├── lshw.tcz
│ │ ├── openssh.tcz
│ │ ├── openssl-1.0.0.tcz
│ │ ├── ruby.tcz
│ │ └── scsi-3.0.21-tinycore.tcz
│ ├── first_checkin.yaml
│ ├── mk_conf.yaml
│ └── tinycorelinux
│ ├── 4.x
│ │ └── x86
│ │ └── tcz
│ │ ├── fuse.tcz
│ │ ├── glib2-dev.tcz
│ │ ├── glib2.tcz
│ │ ├── libdnet.tcz
│ │ ├── libffi-dev.tcz
│ │ ├── libffi.tcz
│ │ ├── ncurses-common.tcz
│ │ ├── ncurses.tcz
│ │ ├── perl5.tcz
│ │ └── procps.tcz
│ ├── kmod-install-list.yaml
│ └── tce-install-list.yaml
├── usr
│ ├── local
│ │ ├── bin
│ │ │ ├── lscpu
│ │ │ ├── mcollectived -> /usr/local/mcollective/bin/mcollectived
│ │ │ ├── rz_mk_controller.rb
│ │ │ ├── rz_mk_control_server.rb
│ │ │ ├── rz_mk_init.rb
│ │ │ ├── rz_mk_tce_mirror.rb
│ │ │ └── rz_mk_web_server.rb
│ │ ├── lib
│ │ │ └── ruby
│ │ │ └── 1.8
│ │ │ └── razor_microkernel
│ │ │ ├── logging.rb
│ │ │ ├── rz_host_utils.rb
│ │ │ ├── rz_mk_configuration_manager.rb
│ │ │ ├── rz_mk_fact_manager.rb
│ │ │ ├── rz_mk_gem_controller.rb
│ │ │ ├── rz_mk_hardware_facter.rb
│ │ │ ├── rz_mk_kernel_module_manager.rb
│ │ │ ├── rz_mk_registration_manager.rb
│ │ │ └── rz_network_utils.rb
│ │ ├── mcollective -> /usr/local/tce.installed/mcollective-2.0.0
│ │ ├── sbin
│ │ │ └── sfdisk
│ │ └── tce.installed
│ │ └── mcollective-2.0.0
│ │ └── ...
│ ├── sbin
│ │ └── dmidecode -> /usr/local/sbin/dmidecode
│ └── share
│ └── udhcpc
│ └── dhcp_mk_config.script
└── util-linux.tcz