^--Soigan--^

Soigan - a Multicast XML monitoring system - plugin howto

Introduction

This is a quick version of How To Write a Soigan Plugin; it will be replaced with a better version at a later date.

Planning

Purpose

So what's needed? First, having some kind of information that you want to produce, generate, collate or make up is good. Examples are "who's logged in?" or "how much disk space is available?".

Structure

Next you need to figure out the structure of the data you want to send back. This is up to you, as you'll be providing a way for the system to query this structure as well.

XML-RPC, upon which Soigan is written, supports integers, doubles, booleans, strings, dates, arrays and structures. In Java they map to int, double, boolean, java.util.String, java.util.Date, java.util.Vector and java.util.Hashtable, respectively. You will need to figure out what the equivalent is for your implementation language.

Language

Anything you like, provided it can be executed at a command line, can take command-line parameters, and can return output to console.

Options

A minimal Plugin needs to support just two things: returning data and returning a schema. Returning data should be possible with no command-line flags supplied (but possibly parameters), and the schema should be retrievable with a -S flag.

Other optional functionality might include a -h for a usage; -X for XML output; -n for no processing (raw output, if the content is from another file or command).

Writing a Plugin

The internal logic of the Plugin is immaterial; however you want to go about retrieving or generating your output is up to you, be it from a file, from another command, over a network or from thin air. What's important is the output generated.

A Soigan Plugin outputs its data in a format that makes it easy for a Soigan Worker (the daemon that runs on host machines) to process and convert into an XML-RPC format. At a minimum, this must include

The contents of the structure are left to the developer, but are restricted to the datatypes supported by XML-RPC.

Output

All output from the Plugin must follow to following rules. Each keyword and data item must be on its own line (line-delimited).

string

All the simple data types are the same -- the name of the datatype, followed by the data itself on a new line.
	string
	<string>
For example, the first bit of data any Plugin should spit out is the name of the Service:
	string
	Plugin@host
Where Plugin is the name of your Service, and host is the host it's running on. An example would be who@nsd or last@csl.

date

This should be the second bit of data output by every Plugin, which represents the time the data was collected.
	date
	<num>
The date is in milliseconds-since-the-epoch format, which most programming languages support. If not, I'm afraid you've got some calculating to do, or a language to ditch. Sorry!

boolean

	boolean
	<true/false>
Only the strings "true" and "false" are supported, not "0" and "1", or "t" and "f".

int

This is a signed int, so any value from negative 2 billion to positive 2 billion. This is the 4-byte signed integer value in your programming language.
	int
	<num>

double

I won't guess at the range of a double, but it's big. And decimal. Unless you're doing high-energy physics, I think you're fine. And if you are, I apologize.
	double
	<num>

structure

A structure consists of the string "struct" on a line, ends with "end" on a line, and contains zero or more of the following: A name for the member, followed by
	<struct entry>	 (this would be a nested structure)

	<array entry>

	<string entry>

	<double entry>

	<int entry>

	<date entry>

	<boolean entry>
In BNF, this would be
	structentry := "struct\n" structcontent "end\n"
	structcontent := structcontent membername entry |
			 nil
	membername := string
	entry := structentry | arrayentry | stringentry | intentry |
			doubleentry | booleanentry | dateentry

	string := STRING "\n"
Order is not important in a structure, as all members are referred by their <name>s.

array

A structure consists of the string "array" on a line, ends with "end" on a line, and contains zero or more of the following:
	<struct entry>

	<array entry> (a nested array, thus an array of arrays)

	<string entry>

	<double entry>

	<int entry>

	<date entry>

	<boolean entry>
And in BNF:
	arrayentry := "struct\n" arraycontent "end\n"
	arraycontent := arraycontent entry |
			 nil
Note that arrays do not have named members as structures do. Order counts in an array. This is the full BNF, if it makes things clearer:
	pluginoutput := stringentry dateentry structentry

	structentry := "struct\n" structcontent "end\n"
	structcontent := structcontent membername entry |
			 nil
	membername := string

	arrayentry := "struct\n" arraycontent "end\n"
	arraycontent := arraycontent entry |
			 nil

	stringentry := "string\n" string
	intentry := "int\n" num
	doubleentry := "double\n" num
	dateentry := "date\n" num
	booleanentry := "boolean\n" boolean

	entry := structentry | arrayentry | stringentry | intentry |
			doubleentry | booleanentry | dateentry


	string := STRING "\n"
	num := NUM "\n"
	date := NUM "\n"
	boolean := "true\n" | "false\n"

An example

Let's look at some sample output, to get a feel for all of the above. This is from a "who" Plugin that shows who's logged into the machine.

The data I've chosen to return is: a structure (since that what it MUST return) that contains two things -- an array of "who entries", and an integer with the number of entries in that array (the number isn't really necessary, since any system should be able to count the entries, but this provides quick info in case the user just wants to know how many people are logged in, but not who).

The array contains multiple structures, each which contain the following: the login name of the user; the tty they're using; when they logged in; and what host they came from.

First we output the name of the Service and the time/date we gathered the data:

	string
	who@stilton
	date
	1092068694322
Then we start the structure, and add in our user count:
	struct
	count
	int
	3
And now we start the array of actual data, using whoentries as the name of our array of structures.
	whoentries
	array
And finally each of the structures representing our logged in users:
	struct
	login
	string
	crwth
	tty
	string
	tty0
	when
	date
	1090507800000
	remotehost
	string
	:0
	end
And here are the rest:
	struct
	login
	string
	crwth
	tty
	string
	tty1
	when
	date
	1092073740000
	remotehost
	string
	:0
	end
	struct
	login
	string
	crwth
	tty
	string
	tty4
	when
	date
	1090515660000
	remotehost
	string
	:0
	end
	end
	end
Notice we had to close off the array tag with an end, as well as the struct tag, so the last entry will have those right after it.

Schema

The output was easy enough to generate, mainly because it's up to the plugin's author to decide what data to return and in what shape. This flexibility is allowed because the plugin must support the ability to return its schema -- that is, to generate out that describes what its data looks like.

If you remember, I wrote a text description of the who plugin up above:

The data I've chosen to return is:  a structure (since that what it
MUST return) that contains two things -- an array of "who entries",
and an integer with the number of entries in that array (the number
isn't really necessary, since any system should be able to count the
entries, but this provides quick info in case the user just wants to
know how many people are logged in, but not who).

The array contains multiple structures, each which contain the following: the login name of the user; the tty they're using; when they logged in; and what host they came from.

We need to describe this to the Soigan system, using the same method that we returned the data. This is done quite simply, as we shall see.

The first thing to note is that the first two fields we outputted before -- the Service name and the time/date -- stay there. This is still a Response from a Plugin, and so it must be structured accordingly. What changes, though, is that the structure that used to contain the data now contains a description of the data structure.

Let's look at the data we returned in the example above, but from a structural point-of-view:

{
	count(int)=5
	whoentries(array)=[
		(struct)={
			login(string)=crwth
			tty(string)=tty0
			when(date)=1090507800000
			remotehost(string)=:0
		}
		(struct)={
			login(string)=crwth
			tty(string)=tty1
			when(date)=1092073740000
			remotehost(string)=:0
		}
		(struct)={
			login(string)=crwth
			tty(string)=tty4
			when(date)=1090515660000
			remotehost(string)=:0
		}
		
	]
}
Here I've used { and } for structures and [ and ] for arrays, in the notation of various languages. We want to describe the above. Let's start from the inside-out, and look at the "whoentry" structure. We need to describe that we have a structure that contains four members: "login", "tty", "when" and "remotehost". They are of types "string", "string", "date" and "string", respectively, which we see from the parentheses. So we describe it in this way:
		(struct)={
			login(string)=string
			tty(string)=string
			when(string)=date
			remotehost(string)=string
		}
This may be a bit confusing at first. Remember, the only thing we need to relate is what each member is called, and its type. All types are described in a textual form, so are all strings. That is why in the above text, all the types (in parentheses), are strings. The "data" of each method above is the text that describes the type in the real data.

I'm not sure how well I explained that. I'll finish describing the rest of the schema and see if that helps.

The above structure is part of an array, and that's all we really need to say -- that we will have an array of them:

	(array)=[
		(struct)={
			login(string)=string
			tty(string)=string
			when(string)=date
			remotehost(string)=string
		}
	]
That array, however, is part of a structure (our "main" structure), and has a name, whoentries. There's also a sole integer with the name count:
{
	count(string)=int
	whoentries(array)=[
		(struct)={
			login(string)=string
			tty(string)=string
			when(string)=date
			remotehost(string)=string
		}
	]
}
And there we have a schema. The above is:

The actual output would then look like this:

	string
	who@stilton
	date
	1092075600493
	struct
	count
	string
	int
	whoentries
	array
	struct
	login
	string
	string
	tty
	string
	string
	when
	string
	date
	remotehost
	string
	string
	end
	end
	end	
With the above, any Soigan Client can ask what your plugin returns, and with luck, the member names you've chosen are descriptive enough that the user can then decide how to interpret the data you return.

In general, the schema will have the same layout as your data output, in that if you have a structure with an array of structures, so will your schema. Schemas, though, always have just one entry in an array, because they only have to describe the array element once, whereas the data output from your plugin might have many entries in a given array.

Other examples

Here are a few more examples of some schemas (and a corresponding diagram to what kind of data they represent) for other Plugins.

last

Return the last so-many logins to a machine
	string
	last@stilton
	date
	1092086519926
	struct
	count
	string
	int
	lastentries
	array
	struct
	login
	string
	string
	tty
	string
	string
	remotehost
	string
	string
	from
	string
	date
	to
	string
	date
	duration
	string
	int
	end
	end
	end

{
	count(int)
	lastentries(array)=[
		(struct)={
			login(string)
			tty(string)
			remotehost(string)
			from(date)
			to(date)
			duration(int)
		}
	]
}

ps

Return a process list on the machine
string
ps@stilton
date
1092086693164
struct
count
string
int
psentries
array
struct
login
string
string
tty
string
string
pid
string
int
ppid
string
int
pcpu
string
double
pmem
string
double
when
string
date
command
string
string
end
end
end

{
	count(int)
	psentries(array)=[
		(struct)={
			login(string)
			tty(string)
			pid(int)
			ppid(int)
			pcpu(double)
			pmem(double)
			when(date)
			command(string)
		}
	]
}

ping

Returns a list of machines that were pingable.
string
ping@stilton
date
1092086840920
struct
count
string
int
pingentries
array
struct
hostname
string
string
ip
string
string
end
end
end

{
	count(int)
	pingentries(array)=[
		(struct)={
			hostname(string)
			ip(string)
		}
	]
}

Conclusion

As mentioned before, a minimal Plugin must be able to do two things: generate output when executed (possibly with flags or parameters), and generate a schema (when passed -S). For testing purposes, the Plugins mentioned above were also written with a couple of other options, which are certainly not required, and don't add anything to the final product.

The -n option spits out data formatted to look like the original source, such as the who command or the last command. For instance, the who Plugin mentioned above generates the following with -n:

crwth      tty0         Jul 22 08:50              (:0)
crwth      tty1         Aug  9 11:49              (:0)
crwth      tty4         Jul 22 11:01              (:0)
and last returns something like:
bart       smb/1        163.195.6.51     Mon Aug  9 09:06   still logged in
crwth      pts/6        gb               Fri Aug  6 11:46   still logged in
crwth      pts/6        gb               Fri Aug  6 09:52 - 11:45  (01:53)
joe        pts/5        daffy            Fri Aug  6 09:22 - 12:32  (03:10)
crwth      pts/1        gb               Fri Aug  6 09:21   still logged in

The -X option generates XML output of the same format that the Soigan Worker eventually creates when talking to a Soigan Server. This XML is functionally equivalent to that produced from the Plugin's usual output, again for debugging purposes. who:

<?xml version="1.0" encoding="ISO-8859-1"?>
  <methodCall>
    <methodName>plugin.results</methodName>
    <params>
      <param>
        <value>who@stilton</value>
      </param>
      <param>
        <value><dateTime.iso8601>20040809T03:38:13</dateTime.iso8601></value>
      </param>
      <param>
        <value><struct>
        <member><name>count</name><int>3</int></member>
        <member><name>whoentries</name>
        <array><data>
          <value><struct>
            <member>
              <name>login</name><value>crwth</value>
            </member><member>
              <name>tty</name><value>tty0</value>
            </member><member>
              <name>when</name><value><dateTime.iso8601>20040722T08:50:00</dateTime.iso8601></value>
            </member><member>
              <name>remotehost</name><value>:0</value>
            </member>
          </struct></value>
          <value><struct>
            <member>
              <name>login</name><value>crwth</value>
            </member><member>
              <name>tty</name><value>tty1</value>
            </member><member>
              <name>when</name><value><dateTime.iso8601>20040809T11:49:00</dateTime.iso8601></value>
            </member><member>
              <name>remotehost</name><value>:0</value>
            </member>
          </struct></value>
          <value><struct>
            <member>
              <name>login</name><value>crwth</value>
            </member><member>
              <name>tty</name><value>tty4</value>
            </member><member>
              <name>when</name><value><dateTime.iso8601>20040722T11:01:00</dateTime.iso8601></value>
            </member><member>
              <name>remotehost</name><value>:0</value>
            </member>
          </struct></value>
        </data></array>
        </member></struct></value>
      </param>
    </params>
  </methodCall>
You'll notice that XML-RPC uses a different format for time/date when in XML form.

last:

<?xml version="1.0" encoding="ISO-8859-1"?>
  <methodCall>
    <methodName>plugin.results</methodName>
    <params>
      <param>
        <value>last@nsd</value>
      </param>
      <param>
        <value><dateTime.iso8601>20040809T03:40:22</dateTime.iso8601></value>
      </param>
      <param>
        <value><struct>
        <member><name>count</name><int>5</int></member>
        <member><name>lastentries</name>
        <array><data>
          <value><struct>
            <member>
              <name>login</name><value>bart</value>
            </member><member>
              <name>tty</name><value>smb/1</value>
            </member><member>
              <name>remotehost</name><value>163.195.6.51</value>
            </member><member>
              <name>from</name><value><dateTime.iso8601>20040809T09:06:00</dateTime.iso8601></value>
            </member><member>
              <name>to</name><value><dateTime.iso8601>null</dateTime.iso8601></value>
            </member><member>
              <name>duration</name><value><int>394</int></value>
            </member>
          </struct></value>
          <value><struct>
            <member>
              <name>login</name><value>crwth</value>
            </member><member>
              <name>tty</name><value>pts/6</value>
            </member><member>
              <name>remotehost</name><value>gb</value>
            </member><member>
              <name>from</name><value><dateTime.iso8601>20040806T11:46:00</dateTime.iso8601></value>
            </member><member>
              <name>to</name><value><dateTime.iso8601>null</dateTime.iso8601></value>
            </member><member>
              <name>duration</name><value><int>4554</int></value>
            </member>
          </struct></value>
          <value><struct>
            <member>
              <name>login</name><value>crwth</value>
            </member><member>
              <name>tty</name><value>pts/6</value>
            </member><member>
              <name>remotehost</name><value>gb</value>
            </member><member>
              <name>from</name><value><dateTime.iso8601>20040806T09:52:00</dateTime.iso8601></value>
            </member><member>
              <name>to</name><value><dateTime.iso8601>20040806T11:45:00</dateTime.iso8601></value>
            </member><member>
              <name>duration</name><value><int>113</int></value>
            </member>
          </struct></value>
          <value><struct>
            <member>
              <name>login</name><value>joe</value>
            </member><member>
              <name>tty</name><value>pts/5</value>
            </member><member>
              <name>remotehost</name><value>daisy</value>
            </member><member>
              <name>from</name><value><dateTime.iso8601>20040806T09:22:00</dateTime.iso8601></value>
            </member><member>
              <name>to</name><value><dateTime.iso8601>20040806T12:32:00</dateTime.iso8601></value>
            </member><member>
              <name>duration</name><value><int>190</int></value>
            </member>
          </struct></value>
          <value><struct>
            <member>
              <name>login</name><value>crwth</value>
            </member><member>
              <name>tty</name><value>pts/1</value>
            </member><member>
              <name>remotehost</name><value>gb</value>
            </member><member>
              <name>from</name><value><dateTime.iso8601>20040806T09:21:00</dateTime.iso8601></value>
            </member><member>
              <name>to</name><value><dateTime.iso8601>null</dateTime.iso8601></value>
            </member><member>
              <name>duration</name><value><int>4700</int></value>
            </member>
          </struct></value>
        </data></array>
        </member></struct></value>
      </param>
    </params>
  </methodCall>

Finally, they also support a -R flag, which is redundant, and just emphasizes that we want (XML-)RPC output. I added this in case someone needed to explicitly specify the output for some reason.

^--Soigan--^

©2002-2017 Wayne Pearson