Identify objects (JSON,YAML) and elements (XML)
The JSON log from above has 2 kind of objects. The log object and the inner time-stamp (instant) object. Both together build one log entry.
Very similar, the Log4j 1.2 XML element, an outer log entry and an inner message element:
<log4j:event logger="de.toem.impulse.test.primary.Log1" timestamp="1589182416309" level="DEBUG" thread="main"> /> <log4j:message><![CDATA[ rcu: ]]></log4j:message> </log4j:event>
And very differnt the next example. The java logging XML has a structure of elements for all information and does not contains any attributes.
<log>
<record>
<date>2009-12-09T10:04:57</date>
<millis>1260371097880</millis>
<sequence>0</sequence>
<logger>com.sun.deploy</logger>
<level>FINE</level>
<class>com.sun.deploy.util.LoggerTraceListener</class>
<method>print</method>
<thread>10</thread>
<message>Reading certificates from 11108 http://pub.admc.com/modeler/modeler-pro.jar | /home/blaine/.java/deployment/cache/6.0/36/363eb424-1a174301.idx
</message>
</record>
...
</log>
The log configuration dialog shall reflect this structure. The first element of the JSON example starts a new log sample. The second one (Timestamp) is part of this log entry, thus the log objects actions is set to "Add to previous sample" (Previous in terms of what was detected before). The last configured object is the "Ignore" pattern. This one is optional and make sense if you have other objects in the log, that shall not be captured. Otherwise an error is thrown for objects that are not part of the configuration.
- impulse supports a lot of different file formats. To get an overview of your installed readers, open the Preferences view and navigate to impulse->Serializers.
- To open a Workspace resource, double-click the record file (or use the context menu of the file).
- Some readers require a configuration (e.g. CSV, PatternLog,..).
- To add a configuration,
- Open the Preferences view and navigate to impulse->Serializers (or use the "Reload" button menu);
- Double-click the reader to open the readers preferences page;
- Press the "Add" button to add and configure the configuration.
- To re-load a file with a different configuration,
- Enter the "Reload" button menu in the viewers toolbar;
- Select the configuration to re-load the file.
- If your file is not loaded (not identified as an impulse record file), first check our website if you need to install another extension (Products) to get the file supported.
- If the file type is supported but not identified, please check if your file has the correct extension. See H102 to change the file extension.
- If the file type is not supported, there are 4 options:
- If your file is a text log file with multiple text pattern, try the Pattern Log Reader (defines the content, using regular expressions).
- You can implement a reader in Java language (check the extension documentation)
- Contact us to get a reader implemented.
- In case of reading signal data from a signal port (e.g. TCP, Memory, Serial Interface,..), you can use the Scripted Reader.
impulse identifies a resource file by its extension and its content data. To change the file extension for a support record type:
- Open the Preferences view and enter General->Content Types;
- Open Element->Record and select the impulse content type (e.g. VCD Record)
- Press "Add" beside the field "File Associations";
- Enter the extension and press "OK".
- Click the "Reload" button in the viewers toolbar;
- Enter the "Reload" button menu in the viewers toolbar;
- Select the configuration to re-load the file.
Screen Cast: How to read custom XML, YAML or JSON logs
Identify Attributes (XML) and Variables (JSON, YAML) and map to members
In the next step, it is required to identify the attributes and variables.
JSON and YAML logs only have variables, XML structures have attributes and text elements.
In the JSON example, there are 'thread, level, loggerName, message, endOfBatch, loggerFqcn, threadId, threadPriority' for the main object. The timestamp has: 'epochSecond' and 'nanoOfSecond'.
Log signals are struct signal with multiple members.
Select member names and types for all variables that shall have a member in you log samples (e.g. the message).
- Select the member names. You can select a predefined member names or enter custom ones.
- For custom members you can define the member type and a descriptor. If custom members are used over multiple log elements, the type need to be identical.
Valid member types are:
- Float,
- Integer,
- Text ,
- or Enumeration.
The descriptor field can optionally contain a content or member descriptor:
- a simple content descriptor e.g. "default", "state", event" or "label", ...
- a complete member descriptor e.g. "default<df=Hex>" or "state<df=Text>"
For JSON values that shall not find their way into the signal structure, leave the the name blank or set type to 'None'.
The table below shows the extracted members.
Define the time-stamp
To define the time-stamp, first look at the domain base settings in the reader configuration dialog. For logs, you typically have 2 meaningful options; 'Time' or 'Date'. With 'Time' you focus on some kind of duration (e.g. seconds or nonoseconds); 'Date' gives you an absolute position in time using Year, Month, Day, Hours, Seconds and Milliseconds.
- Domain Base: Select the domain base. This represents is smallest domain change (e.g. 1 ms).
Define the signal naming scheme
Each single sample belongs to one signal. You can organize your samples into one or multiple signals (e.g. a signal per log source). The "Signal/Scope name" section allows to select the signal and scope naming scheme for each log element.
In most cases it makes sense to get the signal name from the log sources - Thus all logs are organized by their source and it is easier a navigate and understand.
- Undefined : Either all log samples get into one signal with the name "log" -- or this log element is part of a hierarchical multi-object log and the name information is not part of this element.
- Explicit name : Define a signal name for all samples of this log element.
- Explicit hierarchy : Define a signal/scope hierarchy for all samples of this log element. The signal and scope names are generated by splitting the value into its contents (e.g. the value "de.toem.logger" creates the scope/signal hierarchy de/toem/logger).
- Name from source value : The value of the selected source is taken to define the signal name (.e.g. from the logger source).
- Hierarchy from source value : The value of the selected source is taken to define the signal and scope names by splitting the value into its contents (e.g. the value "de.toem.logger" creates the scope/signal hierarchy de/toem/logger).
You may extend the naming scheme with another source value. If enabled, the value of the source will be put after the signal name (in parentheses).
XML vs JSON and YAML
The XML, JSON and YAML log readers are very much the same from the UI side. But there are slight differences.
The XML reader has this optional setting:
- Read XML fragments: If checked, the reader will wrap the input content into a dummy root element.
If not checked, the reader will only accept well-formed XML. If checked, the reader wraps the received XML data into a dummy root element.
<dummy> <log4j:event logger="de.toem.impulse.test.secondary.Log3" timestamp="1589182416306" level="WARN" thread="ThreadB">
<log4j:message><![CDATA[get Sinus Wave 0.9629702887498031]]></log4j:message>
</log4j:event> </dummy>
Another difference is the text element. Text content can be mapped to members using this first psoido attribute ("XML Text").
