Each context associated with a particular log file is constituted of a series of components. Each component has a role to play in processing a log file.
The following diagram shows the structure of a typical Generic Log Adapter configuration.
The context describes the ordered grouping of components as they are chained together for log file processing. Each log file has a separate context, and multiple contexts can be defined in a single configuration. The Generic Log Adapter provides two context implementations:
A context has the following properties:
Property | Type | Behavior |
---|---|---|
Description | string | A text description for the context. This value is used only by the Adapter Configuration Editor. |
Unique Id | string | A unique identifier for this context. The context refers to this string to identify which component instance to use. |
Is Continuous Operation | boolean | Whether or not this log file should be monitored for changes based upon the Maximum Idle Time and Pause Interval properties. |
Maximum Idle Time | integer (milliseconds) | The maximum amount of time the context will wait for a log file to change before shutting itself down. |
Pause Interval | integer (milliseconds) | The amount of time the context waits after reaching the end of the log file before trying to read again. |
ISO language code | The language code of the Locale that the log file was generated in. These codes are the lower-case, two-letter codes as defined by ISO-639. You can find a full list of these codes at a number of sites, such as: http://www.ics.uci.edu/pub/ietf/http/related/iso639.txt. If this value is not specified, the language of the current Locale where Generic Log Adapter is executing will be used. This value is used when parsing timestamps in the log file. | |
ISO country code | The country code of the Locale that the log file was generated in. These codes are the upper-case, two-letter codes as defined by ISO-3166. You can find a full list of these codes at a number of sites, such as: http://www.chemie.fu-berlin.de/diverse/doc/ISO_3166.html. If this value is not specified, the country of the current Locale where Generic Log Adapter is executing will be used. This value is used when parsing timestamps in the log file. | |
Encoding | The character encoding of the log file to be parsed. The encoding value specified must be a value listed in the IANA Charset Registry (http://www.iana.org/assignments/character-sets) and be supported by the JVM that Generic Log Adapter executes in. If this value is not specified, the log file will be parsed using the default file encoding of the JVM that Generic Log Adapter executes in, as specifed in the file.encoding JVM system property. |
The sensor provides the mechanism to read the content of the log for processing. The Generic Log Adapter provides the following sensors:
The following table lists the attributes that are common among the sensor types.
Sensor attributes
Attribute name | Type | Behavior |
---|---|---|
Sensor Type | String | The sensor type to be used by the adapter. |
Description (required) | String | A text description for the sensor. This value is used only by the Adapter Configuration Editor. |
Unique ID (required) | String | A unique identifier for this component. The sensor refers to this string to identify which component instance to use. |
Maximum Blocking (required) | Integer | The maximum number of lines the sensor reads before forwarding to the extractor for processing. For the StaticParserSensor type, this field indicates the maximum number of Common Base Event objects created by the static parser before forwarding to the outputter for processing. |
Confidence Buffer | Integer | The last n bytes of a file to be saved and used for determining if an append has occured to the file since the last time it was read. |
Footer size | Integer | The n bytes of a file that contains static content. Some log files append a footer to the end of the log file. This part of the log file is not considered log data. The footer portion of a file will not be used when determining if an append has occured since the last time the file was read. |
The following table lists the properties that are specific for each type of sensor. The following properties are known by the component code and must be spelled exactly as indicated in the table for them to be recognized and used by the component.
Sensor type | Properties | Application |
---|---|---|
SingleFileSensor |
|
Used when creating an adapter for parsing a log file with regular expression rules |
StaticParserSensor |
|
Used when creating an adapter for parsing a log file with a static parser Java class. |
The extractor takes a collection of input lines provided by the sensor and separates them into message boundaries.
The Generic Log Adapter provides the following extractors:
The following properties are provided as part of the configuration for the extractor:
Property | Type | Behavior |
---|---|---|
Description | string | A text description for the extractor. This value is used only by the Adapter Configuration Editor. |
Unique Id | string | A unique identifier for this component. The extractor refers to this string to identify which component instance to use. |
Contains Line Breaks | boolean | Indicates whether the messages in the log file span multiple lines. |
Replace Line Breaks | boolean | Replace the line breaks in this file by another string segment. The replacement is
made before the extractor searches for Start Pattern or End Pattern. Note: The side effect of using this option is that the values parsed from the log file will not contain line breaks unless explicitly added by the parser substitution rules. |
Line Break Symbol | string | The string segment to replace line breaks with. This is used only if Replace Line Breaks is true. |
Include Start Pattern | boolean | Whether to include the StartPattern as part of the message to be forwarded to the next component. If the value is false, the matching portion of the message is stripped. |
Start Pattern | string or regular expression | The string segment that identifies the start of a message. You can always use the RegularExpressionExtractor class. However, if your start pattern is a simple string, the SimpleExtractor class incurs less overhead. |
Include End Pattern | boolean | Whether to include End Pattern as part of the message to be forwarded to the next component. If the value is false, the matching portion of the message is stripped. |
End Pattern | string or regular expression | The string segment that identifies the end of a message. If End Pattern is not specified, then the message is delimited by the next Start Pattern. |
The parser takes the messages that have been delimited by the extractor and builds a set of mappings of string values to a data structure. In the case of the Generic Log Adapter the data structure is Common Base Event 1.0.1
The Generic Log Adapter provides the following parser class:
The parser has two phases of execution:
The two phases of execution allow the message to be tokenized into a series of attribute values during the global processing phase. The attribute values can then be referred to by attribute name or index during the attribute processing phase.
The following global properties are specified as part of the parser:
Property | Type | Behavior |
---|---|---|
Description | string | A text description for the parser. This value is used only by the Adapter Configuration Editor. |
Unique Id | string | A unique identifier for this component. The parser refers to this string to identify which component instance to use. |
Separator Token | string or regular expression | A token, or a regular expression that describes a series of tokens, that breaks the
message up into attribute-value pairs. For example, if each new line of your log file contains an attribute and its value, you
could specify a Separator Token of \n. The advantage of breaking the message up is that the string segments
can be referred to in the attribute processing phase. |
Designation Token | string or regular expression | A token, or a regular expression that describes a series of tokens, that breaks the string segments created by the Separator Token into attribute-value pairs. For example, if each attribute and value are separated by a = sign , and the value of Designation Token is = , the attribute and its value can be referred to in the attribute processing phase. If a separator token is specified but a designation token is not specified then the message is broken up into a list of attribute values which can be referred to in the substitution rules by index. |
The individual attributes of the Common Base Event have the following properties:
Note: The values parsed during the global processing phase will be trimmed of leading and trailing white space characters. Therefore, substitution rules that use position values can be written assuming the values have no leading or trailing white space characers.
Property | Type | Behavior |
---|---|---|
Use Previous Match Substitution As Default | boolean | If the current set of substitution rules for this attribute returns a positive match, then store the match value so that it can be used as a default to process the next record if there is no positive match. |
Default value | String | If none of the substitution rules for this attribute provide a positive match or there are no substitution rules for this attribute then this attribute will be assigned the default value specified here. If the same value is desired for all records for a particular attribute, this property can be used instead of creating a default substitution rule for that attribute. Note, the Use Previous Match Substitution as Default will take precedence if there is a saved value available. |
Required by parent | boolean | Allows you to specify whether or not this element must have a valid value in order for the parent element to be valid. For example, if this option is selected for the situation element, this indicates to the parser that the Common Base Event object must have a valid situation element. During processing, if the situation element returns a null value, the parser will not process any of the other child elements of the Common Base Event object since it already knows that this Common Base Event object is invalid because a required child element returned an invalid value. |
Child choice | boolean | Allows you to specify whether or not processing for this element should stop once a valid child attribute or element returns a valid value. For example, consider the situation with three child elements, ReportSituation, RequestSituation and StartSituation, each with their own subtrees. If the Child Choice option is selected for the situation element, parsing will stop at its first child subtree that returns a valid value. |
Substitution rules are run as part of the attribute processing phase. They are processed in order from top to bottom. Once a rule is a match, no further rules are run against this property. If no match is found, then no value is placed in the attribute unless the Use Previous Match Substitution as Default or Default Value properties of the attribute are specified.
Substitution rules contain the following properties:
Property | Type | Behavior |
---|---|---|
Match | regular expression | The regular expression that identifies the value to be applied to this attribute if a match occurs in the scope of the string segment being processed by this Substitution Rule. If you do not provide a regular expression, the rule is considered a match. |
Positions | name-value attribute pairs |
Positions let you refer to the name-value attribute pairs created during the global
processing phase.
In the Match property, you can add the separator to indicate which regular expressions should be applied to which attribute value. |
Substitute | string | The value to place in the attribute. This can be a string or it can refer to the regular expression in the Match property. The string segments that are extracted by the Match property can be referred to using the groupings in the regular expression. $1 indicates the first grouping, $2 the second, and so on. |
Use built in function | check box |
Use built-in function when selected for an attribute, if none of the previous substitution rules,
if any exist, are matched, will cause the Generic Log Adapter runtime to generate
a value for that attribute using its own internal functions.
This feature is only supported for the following CBE attributes:
|
Substitution Class | string |
Specifies the name of a user created Java class that can perform custom parsing
logic on the value matched by the Positions, Match and Substitute specifications
of the rule.
If the substitution rule matches data in the record, this class will be called and the result
will be used to create the Common Base Event object.
Specifications:
|
Time Format | string |
This field only applies to substitution rules for the creationTime element
and is used to simplify date time parsing.
For example, some log files contain timestamps that include the
names of a month instead of its numerical representation.
This makes it difficult to create substituion rules to generate the
string value in the form of the XML Schema dateTime data
type required by the CBE creationTime element. In this case the user can specify a java
simple date format string as documented in the javadoc for java.text.SimpleDateFormat
(java.sun.com/j2se/1.4.2/docs/api/java/text/SimpleDateFormat.html) in the Time Format field.
The value matched by the Positions, Match and Substitute specifications of the rule will be parsed
based on the Time Format string and the resulting value will be used to generate the CBE creationTime value.
Note: The Locale, as represented by the ISO language and country code values specifed in the Context Instance configuration, is used when parsing the timestamp with the time format string. |
The formatter takes the mappings of attributes to their values provided by the parser and builds the correct Java object instance. The adapter runtime contains a single formatter that creates Common Base Event objects conforming to the Common Base Event version 1.0.1 specification.
The formatter contains the following properties:
Property | Type | Behavior |
---|---|---|
Unique Id | string | A unique identifier for this component. The formatter refers to this string to identify which component instance to use. |
Description | string | A text description for the formatter. This value is used only by the Adapter Configuration Editor. |
The outputter externalizes the resulting Common Base Event records provided by the formatter. Outputters provide, or wrap, the mechanism for storing the final outputs of the context. The Generic Log Adapter provides the following outputters:
Properties of the outputter:
Property | Type | Behavior |
---|---|---|
Unique Id | string | A unique identifier for this component. The outputter refers to this string to identify which component instance to use. |
Description | string | A text description for the outputter. This value is used only by the Adapter Configuration Editor. |
Outputter Type | enumeration | The type of outputter this is. |
Outputter types and required properties:
Outputter type | Required properties |
---|---|
StandardOutOutputter | None |
SingleFileOutputter | directory, fileName |
LoggingAgentOutputter | agentName - name given to the logging agent when it is created waitUntilLoggingTime(optional)- the time the outputter will wait for the user to attach and start monitoring the agent before it starts writing to the agent. If this property is not specified, the outputter will not wait for the agent to be monitored before writing to it. |
Related concepts
Overview of the Hyades Generic Log Adapter
Common Base Event format specification
Related tasks
Creating a log parser
Creating a rules-based adapter
Creating a static adapter
Related references
Adapter Configuration File structure
Common Base Event format specification
Adapter Configuration Editor
Regular expression grammar
(C) Copyright IBM Corporation 2000, 2005. All Rights Reserved.