^--amphioxus--^ DTD-->

AMPHIOXUS

A Management Program for Hosts In Ordered XML Using SCCS Subversion


The pros:

The cons:

(Correction: with a few tweaks, it now takes just over one minute thirty to do a "make" and "make install")

(May 12, 2004: Oy vey. It takes over six minutes for a full build. Blame it on the DNS!)

How it works:

If you're familiar with XML, then you're laughing. No, really. Laugh.
If you're not, but are familiar with HTML, you're only snickering.
If you're not familiar with either... err... just smile and nod.

All the host information is now kept in a single document, residing at the time of this writing in /etc/xml/hosts.xml on csa. It is checked into SCCS, so you would probably do something like the following:

csa% su
Password: ******************
% cd /etc/xml
% sccs edit hosts.xml
1.20
new delta 1.21
12532 lines
% my_favorite_editor hosts.xml
You should then see a large file (12532 lines, according to the text above), with blocks like:
<host>
<name>fsa</name>
<alias>ntp</alias>
<alias>ntp1</alias>
<alias>emgw1</alias>
<alias>submit</alias>
<ip>136.159.2.1</ip>
<hostcomment>Dell 2450</hostcomment>
<netgroup>linux_1</netgroup>
<netperms>* : $STAFF</netperms>
<dns internal="yes" external="yes">
<cname>ntp</cname>
<cname external="no">submit</cname>
<cname>ntp1</cname>
<cname>emgw1</cname>
</dns>
</host>
In XML, like in HTML, we have a bit of terminology. The main one we deal with are elements, highlighted here:
<host>
<name>fsa</name>
<alias>ntp</alias>
<alias>ntp1</alias>
<alias>emgw1</alias>
<alias>submit</alias>
<ip>136.159.2.1</ip>
<hostcomment>Dell 2450</hostcomment>
<netgroup>linux_1</netgroup>
<netperms>* : $STAFF</netperms>
<dns internal="yes" external="yes">
<cname>ntp</cname>
<cname external="no">submit</cname>
<cname>ntp1</cname>
<cname>emgw1</cname>
</dns>
</host>
Each of the elements shown is a container element, so it has the starting tag (<element-name>) and an ending tag (</element-name>), with other things between -- sometimes text, sometimes other elements.
The other important term is attributes, shown here:
<host>
<name>fsa</name>
<alias>ntp</alias>
<alias>ntp1</alias>
<alias>emgw1</alias>
<alias>submit</alias>
<ip>136.159.2.1</ip>
<hostcomment>Dell 2450</hostcomment>
<netgroup>linux_1</netgroup>
<netperms>* : $STAFF</netperms>
<dns internal="yes" external="yes">
<cname>ntp</cname>
<cname external="no">submit</cname>
<cname>ntp1</cname>
<cname>emgw1</cname>
</dns>
</host>
These are contained within the starting tag of an element.

Most of the element names are obvious, I think, but we'll go over what they're for. Note that the order is important -- this is one of the biggest drawbacks of this system, and one I'm trying to work around. If you like, follow along while browsing csa:/etc/xml/hosts.xml

<hostlist> (required)

This is the element that holds all of the host information; XML requires having a containing, single object. The two tags above this in the file are meta-data describing the format of the XML file and the location of the DTD file (mentioned below).

hostlist elements:
<host> (required)

attributes: comment-out

This element contains all of the other elements of a host. If the comment-out attribute is set to yes, the host entry will exist in the hosts and DNS in commented form (good for temporarily disabling an entry but still "claiming" the name and IP). The default value of comment-out is no.

<preamble> (optional; zero or more)

attributes: file, subnet, group

The contents of this element are placed at the beginning of a file, DHCP subnet entry or DHCP group entry, depending on the value of its attributes. If file is set, then the <preamble> contents are placed at the start of the named file, such as
<preamble file="hosts">
#
# Copyright (c) 1992-2003 The Department of Computer Science, University of Calgary.
# All rights reserved.
#

</preamble>
If the subnet attribute is set, then the <preamble> contents are placed at the beginning of that subnet entry in the dhcpd.master.conf file, viz:
<preamble subnet='136.159.2'>
subnet 136.159.2.0 netmask 255.255.255.0
{
  option routers 136.159.2.10;

  #Solaris 8
  group
  {
    filename "inetboot.sun4u.Solaris_8-0701";
    next-server web.cpsc.ucalgary.ca;
</preamble>
If the group attribute is set, then the <preamble> contents are placed around all host entries with the matching group attribute in their <dhcp> element within a given subnet, such as:
<preamble group="solaris9">
  group
  {
    option SUNW.root-server-ipaddress       nsc.cpsc.ucalgary.ca;
    option SUNW.root-server-hostname        "nsc";
    option SUNW.root-path-name              "/export/install/Solaris_9-0403/Solaris_9/Tools/Boot";
    option SUNW.install-server-hostname     "nsc";
    option SUNW.install-server-ipaddress    nsc.cpsc.ucalgary.ca;
    option SUNW.install-path                "/export/install/Solaris_9-0403";
    option SUNW.JumpStart-server            "nsc:/export/install/Solaris_9-0403/jumpstart";
    option SUNW.sysid-config-file-server    "nsc:/export/install/Solaris_9-0403/jumpstart";

    filename "inetboot.sun4u.Solaris_9-0403";

</preamble>
The default group preamble can be set using the default value:
<preamble group="default">
  #Solaris 8
  group
  {    
    filename "inetboot.sun4u.Solaris_8-0701";
    next-server web.cpsc.ucalgary.ca;
</preamble>

<postamble> (optional; zero or more)

attributes: file, subnet, group

Similar to the <preamble> element above, the contents of this element go at the tail end of the named file or subnet entry, such as:
<postamble subnet='136.159.2'>
  }
}
</postamble>

host elements:
<name> (required)

Obviously, the name of the host. Since hosts can have aliases (see below), this is the primary name: the first one listed in the hosts file; the name used as the DHCP host; the A record in DNS; the name referenced in netperms and netgroup.

<alias> (optional; zero or more)

If the machine has other hostnames, they can be listed here. These names will be added to the hosts file. See below for DNS aliases (CNAMEs).

<ip> (required)

The IP address of the machine. This is currently required; if a machine doesn't have an IP address, why is it being managed in networking files? While machines can technically have more than one IP address if they have more than one interface, it's typical that each interface has its own name, and should therefore have its own <host> entry. Perhaps, logically, they should all be grouped together, but...

<user> (optional)

The primary user or group that uses this host. This can be anything, but is inserted, along with the location and description text, as a comment in the hosts and DNS files (provided an explicit comment isn't provided -- see below).

<location> (optional)

The physical (or logical) location of this host. This can be anything, but is inserted, along with the user and description text, as a comment in the hosts and DNS files (provided an explicit comment isn't provided -- see below).

<description> (optional)

A general of this host. This can be anything, such as make or operating system, but is inserted, along with the user and location text, as a comment in the hosts and DNS files (provided an explicit comment isn't provided -- see below).

<hostcomment> (optional)

A general string placed at the end of the hosts file entry. If this element is not defined, the system will build one from the <user>, <location> and <description> tags, if available.

<netgroup> (optional)

The netgroup to which this host belongs. Note that there is no testing done by this system that such a group previously existed, or that there aren't too many entries in that group -- the external programs that checked that before will still do this work.

<netperms> (optional)

The netperms entry for this host. NOTE: This is pasted in as-is into the netperms file. While most hosts would have an entry such as
<netperms>* : $STAFF</netperms>
in fsa, it is possible to have something more complex such as
<netperms>* : $STAFF | { cliff }</netperms>
or even
<netperms>xdm : $STAFF | { lonsdale, argento, peuter } \
                * : $STAFF | { lonsdale, argento, peuter }</netperms>
It is important to note that everything between the start and end tags for <netperms> will be pasted into netperms, so using the line-continuator (\) is important for multi-line entries. Now, whether having entries for more than the "*" resource is necessary is another question... Also, because this is XML, like HTML, certain characters have to be escaped, the main one in this case being the & symbol, which must be written as &amp;, like
<netperms>* : $STAFF | ( { .csus } &amp; ! { csus } )</netperms>
Which will make a netperms entry as:
warthog		* : $STAFF | ( { .csus } & ! { csus } )

<dhcp> (optional; zero or more)

attributes: comment-out, group

This element contains all the DHCP-related information for the host, which is covered below. If the comment-out attribute is set to yes, the host entry will exist in the DHCP file in commented form (good for temporarily disabling an entry for testing purposes).

The default value of comment-out is no. NOTE: Multiple <dhcp> entries can exist, if a given IP address can be given out to multiple MAC addresses. This is often used for laptops that might have a docking station, or test addresses that float around a handful of network interfaces.

The default value of group is default. The value of this attribute is used to collect hosts within each subnet and place them in the same DHCP group; <preamble> and <postamble> tags with the group attribute define the blocks in which the hosts appear -- see the <preamble> entry for more information.

<dns> (optional)

attributes: internal, external

This element contains all the DNS-related information for the host, which is covered below. The internal and external attributes define whether this host is listed on the internal and external DNS servers. The default value for internal is yes, and for external it's no. Note that this is for the main A record (the main hostname); CNAME entries can be told individually about their internal and external appearance. If no dns element is provided, it's assumed that the default values of the two attributes are assigned and correct.

dhcp elements:
<dhcpcomment> (optional)

Similar the hostcomment, this text gets put along with the dhcp entry in the dhcpd.master.conf file. If this element is not defined, the system will build one from the <user>, <location> and <description> tags, if available.

<mac> (required)

The MAC address for this host. Because multiple <dhcp> entries can exist for a host, multiple MAC addresses can be associated with that host, like the following:
<dhcp>
<dhcpcomment> Departmental floater 3com NIC </dhcpcomment>
<mac>00:60:97:36:f2:27</mac>
<dhcpname>whizfloat17</dhcpname>
</dhcp>
<dhcp>
<mac>00:06:5b:b9:42:c3</mac>
<dhcpname>whiz17</dhcpname>
</dhcp>
which would generate these two entries in dhcpd.master.conf:
    #  Departmental floater 3com NIC 
    host whizfloat17 { hardware ethernet 00:60:97:36:f2:27; fixed-address whiz17; }
    #
    host whiz17 { hardware ethernet 00:06:5b:b9:42:c3; fixed-address whiz17; }

<dhcpname> (optional)

This can be used to provide a different name for the host, such as the first entry in the example above. I think the DHCP system complains if you have more than one entry for the same dhcpname, so any secondary (and beyond) dhcp entries will require this. If not supplied, the value of the <name> tag is used.

<dhcphost> (optional)

dhcphost allows you to override the value of the fixed-address in the DHCP file entry. It is very unlikely you will ever use this. The only entries that you might see with this value defined are entries culled from the existing dhcpd.master.conf file that used IP addresses instead of hostnames. If not supplied, the value of the <name> tag is used.

<preamble> (optional)

The contents of this element are placed before a DHCP entry. This can be used for special case entries that exist in the DHCP file for individual hosts (see "mithril" in /etc/xml/hosts.xml). If you don't know what this is for, you probably don't need it.

<postamble> (optional)

Similar to <preamble> above, the contents of this element are placed after a DHCP entry.

dns elements:
<dnsname> (optional)

Provides an alternate name for reverse-DNS lookup, if <hostname>.cpsc.ucalgary.ca isn't what is desired. It's rare that this would be used, but an example is calgarysciencenetwork.ca being returned instead of csn.cpsc.ucalgary.ca. If not supplied, the value of the <name> tag is used, appended with .cpsc.ucalgary.ca..

<cname>

attributes: internal, external

Defines DNS aliases (CNAMEs) for the host. Note that having an <alias> element does not automatically imply that that alias is used in DNS; it must be defined here also. The internal and external attributes define whether this CNAME is listed on the internal and external DNS servers. The default value for internal is yes, and for external it's yes.
If you think it'll help, you can take a look at this DTD (Document Type Definition) file, which is used by the system to enforce the definitions above.

Examples

A good example of a "full" host entry is the following:
<host>
<name>cobol</name>
<alias>dict</alias>
<alias>webster</alias>
<alias>answerbook</alias>
<alias>jabber</alias>
<ip>136.159.2.6</ip>
<hostcomment>Ultra2 dict and answerbook server(in need of a better name)</hostcomment>
<netgroup>sun4</netgroup>
<dhcp>
<mac>8:0:20:85:f2:eb</mac>
</dhcp>
<dns internal="yes" external="no">
<cname external="no">dict</cname>
<cname external="no">answerbook</cname>
<cname external="no">jabber</cname>
<cname>webster</cname>
</dns>
</host>
All that's really missing are <user>, <location> and <description> tags and an example of more than one <dhcp> entry. A simpler host entry could be:
<host>
<name>stilton17</name>
<ip>136.159.17.170</ip>
<user>crwth</user>
<location>ms625b</location>
<dhcp>
<mac>00:04:75:9d:f2:4b</mac>
</dhcp>
<dns internal="yes" external="no">
</dns>
</host>
Or, even simpler:
<host>
<name>localhost</name>
<ip>127.0.0.1</ip>
<dns internal="yes" external="yes">
</dns>
</host>

And once you've done some XMLing...

% sccs delget hosts.xml
comments? Added new host -crwth
No id keywords (cm7)
1.22
0 inserted
0 deleted
12536 unchanged
1.22
12536 lines
No id keywords (cm7)
% make

...
Running make in /etc/xml does the following: Running make install in /etc/xml does the following: (Note: Until it's decided that we're going with this system, the make install actually "installs" everything into /tmp on csa for testing purposes; take a look at /etc/xml/Makefile)

Issues

Todo


Please let me know if there's anything wrong, missing, confusing or offensive about this system. Give it a whirl; try checking out hosts.xml in csa:/etc/xml, editing it, checking it back in and running make. The files are all created in that same directory (with DNS in the /etc/xml/internal and /etc/xml/external directories for Internal and External, respectively.
Also, if you're keen, you can look at the handful of .xsl files in there to see how the raw XML data gets transformed into pretty datafiles.

--
Wayne

May 20, 2003
The system has been put into place. Regarding the Issues above, running make will do the checkout-edit-checkin step, run propagate in /etc/dhcpd, then run make in /usr/etc/named, and finally HUP the named daemon on fsa. This means that running ship doesn't cause a rebuild of the files, so any changes made to /etc/xml/hosts.xml WILL NOT be reflected system-wide unless make is run from within /etc/xml.

June 16, 2003
To allow for upgrades to external DNS, named.*.rev is now generated for the external DNS server(s) as well as internal (where previously the external server cached the information from the internal server). The changes were:

June 19, 2003
Added the group attribute to the dhcp element. This allows grouping of dhcp hosts with similar options and such, as mithril used to have. Also added was the group attribute to the preamble element to provide this support.

August 12, 2003
Created a utility called hfinger (host finger), renamed to findhost, which can be given a hostname, a user, a MAC address, an IP, etc., and it'll look it/them up in the hosts.xml file. You can also give it a slew of parameters to restrict what info you're given.

August 21, 2003
Added a new element, platform, with attributes os and version, such as

        <platform os="Windows" version="XP"/>
or
        <platform os="Linux" version="Fedora"/>
Oh yeah, and some time before this I added user, location and description. Oops.

October 30, 2003
Created a small C++ executable called generate which takes a datafile (/etc/xml/tasklist or /etc/xml/etclist) of processing tasks. This changes the Makefile from using nearly 50 calls to the Xalan executable to just a few, as most are done within the generate program. This speeds things up immensely now that internal and external DNS is so large.

February 9, 2004
Added make all as the new target for the full system, where make only builds the etc/ files and not DNS. The majority of changes people need don't require DNS to be updated right away, so this allows them to get on with their day a little quicker.

April 15, 2004
Added support for generation of the /etc/ethers file.
Added the ethers attribute to the mac element, to allow a specific MAC entry to NOT be inserted into /etc/ethers. The default value is "yes".

May 12, 2004
Added support for subnets larger than /24.

The dhcpsubnet element has a new attribute, realsubnet. This is used like this:

<dhcpsubnet subnet='10.0.4'>
  <preamble>
  option routers 10.0.4.1;
  </preamble>
  <subnetmask>255.255.252.0</subnetmask>
</dhcpsubnet>
<dhcpsubnet subnet='10.0.5' realsubnet='10.0.4'/>
<dhcpsubnet subnet='10.0.6' realsubnet='10.0.4'/>
<dhcpsubnet subnet='10.0.7' realsubnet='10.0.4'/>
This configures the network 10.0.4/22, so all hosts are placed in the correct dhcp subnet section.

Don't tell anyone, but I had to cheat and use Perl to keep performance from going through the toilet; building the dhcpd.master.conf went from a 32 second task to a 4 minute one using just XSLT, but using a little Perl helped keep the increase to 1 minute. Time for a complete build, as of this date, is currently at 6:15 (yuck), where leaving DNS-building out reduces it to 2:30. And 45 seconds of both of those runs are the ship command running -- not my fault!

July 15, 2004
Added propertytag and servicetag optional elements.

June 13, 2005
Name change! Amphioxus now stands for "A Management blah blah blah Using Subversion"! That's right, no more SCCS. I left the original instructions in the body of this text for historical purposes. The new method for making changes can be found here.

Along with the repository change, Amphioxus is now run on a much faster machine. I don't have any numbers on crunching time, but since it's done during the check-in process, it can't be too long!

With the move to the new machine, we're no longer using the Apache Xerces/Xalan libraries, either; we've moved to the GNU libxml/libxslt pair, since they don't require any building (there by default), and do the job just as well. In fact, the errors from xsltproc are better than Xalan's.


p.s. And since it was asked how Amphioxus was pronounced... Bummer... the link's dead.
^--amphioxus--^ DTD-->
©2002-2017 Wayne Pearson