Contents
What is Tachyon Agent Historic Data Capture?
The Tachyon Agent, while running, continuously captures details of certain events as they happen, in a similar way to Windows Task Manager or Perfmon. During startup, the Agent is able to detect some events that occurred when it was not running. Data is regularly written into a local, compressed and encrypted persistent store, called an inventory. The inventory is accessible to the Agent language as SQL tables. The Agent periodically aggregates data in order to minimise the amount of storage required, so that each capture source has a live, hourly, daily and monthly table. The whole process is designed to minimise impact on device performance, storage and security.
Configuration options for each capture source are described in the public documentation reference for Tachyon Agent configuration properties. The defaults are the same for each capture source, but aggregation and retention settings can be set individually for each table.
The Agent has two key mechanisms of knowing when an event occurs that is of interest - event-based and polling-based
- Event-based relies on a source external to the Agent (normally the operating system) providing a notification to indicate that something has happened
- Polling-based is where the Agent will periodically check a source of data and work out what has changed by looking at differences in the data returned. Polling intervals means some brief events that occur between polls can be missed.
On Windows, the Agent is able to use Event Tracing for Windows (ETW). However, the Agent can be configured to use polling instead of ETW for individual capture sources.
The table below lists currently supported capture sources, on which OS they are supported, and which capture method is used by default. See Agent Historic Data Capture regarding Windows XP, Vista and Windows Server 2003.
Historic data source | Description | Introduced | Windows | MacOS | Linux | Solaris | Android |
---|---|---|---|---|---|---|---|
ARP cache entries | The Agent captures translations between IP addresses and MAC (physical) addresses, known as ARP (Address Resolution Protocol). ARP cache polling is every 30 seconds. |
| Not yet available | Not yet available | Not yet available | Not yet available | |
DNS resolutions | The Agent captures whenever a DNS address is resolved. When using the polling method, the polling interval is every 30 seconds. |
| Polling | Not yet available | Not yet available | Not yet available | |
Process executions | The Agent captures whenever a process starts on the device. When using the polling method, the polling interval is every 30 seconds. |
| Polling | Polling | Polling | Not yet available | |
Process stabilizations | The Agent captures the time taken for a process to be considered stable. This is captured when a process starts on a device, provided that process is in a list of processes selected for monitoring in the Agent configuration file. |
| Not yet available | Not yet available | Not yet available | Not yet available | |
Software installations | The Agent captures which software is present on a device, and when it is installed and uninstalled. Software polling is every 120 seconds. |
| Polling | Polling | Polling | Not yet available | |
TCP outbound connections | The Agent captures whenever an outbound TCP connection is made. When using the polling method, the polling interval is every 30 seconds. |
| Polling | Polling | Polling | Not yet available |
How do I retrieve the data from the Tachyon Agent devices?
Live and aggregated historic data is available in inventory tables.
Historic data source | Live tables | Hourly tables | Daily tables | Monthly tables |
---|---|---|---|---|
ARP cache entries | $ARP_Live | $ARP_Hourly | $ARP_Daily | $ARP_Monthly |
DNS resolutions | $DNS_Live | $DNS_Hourly | $DNS_Daily | $DNS_Monthly |
Process executions | $Process_Live | $Process_Hourly | $Process_Daily | $Process_Monthly |
Process stabilizations | $ProcessStabilization_Live | $ProcessStabilization_Hourly | $ProcessStabilization_Daily | $ProcessStabilization_Monthly |
Software installations | $Software_Live | $Software_Hourly | $Software_Daily | $Software_Monthly |
TCP outbound connections | $TCP_Live | $TCP_Hourly | $TCP_Daily | $TCP_Monthly |
/* Sum the number of connections made per process today */ SELECT SUM(ConnectionCount) AS Connections , ProcessName FROM $TCP_Daily WHERE TS = DATETRUNC(STRFTIME("%s", "now"), "day") GROUP BY ProcessName;
Note the below example uses LIKE because the inventory tables are not created with COLLATE NOCASE, and need to be queried in a case-sensitive fashion. If ProcessName = "chrome.exe" is used then it will not match "Chrome.exe" or "chrome.EXE".
SELECT * FROM $Process_Live WHERE ProcessName LIKE "chrome.exe"
How is the data managed?
The Tachyon Agent automatically aggregates and grooms data in each inventory table, according to aggregation intervals and data retention settings which are configurable in the Agent configuration file.
- Default aggregation cycle interval is every 60 seconds, therefore it may take up to a minute before an event appears in an aggregated table
- Default retention for live tables is 5000 entries provided at least 3 aggregation cycles have occurred (older entries are deleted to make room for new entries)
- Default retention for hourly tables is a rolling 24 hours.
- Default retention for daily tables is a rolling 31 days.
- Default retention for monthly tables is a rolling 12 months.
Each aggregated table is built from the live table, and does not have a dependency on other aggregated tables. For example, Monthly is fed by Live, not fed by Daily. This allows retention settings to be configured independently for each table.
Data is stored in a local, compressed and encrypted persistent store, which persists during an Agent upgrade, uninstall and re-installation, unless specifically deleted.
If the Agent is unable to write to storage (out of disk space or other file-system problems), it will fail but continue monitoring in the hope this situation will improve later.
Historic data capture inventory schema
The following table shows the fields which exist only in the Live and Aggregated (Hourly, Daily, Monthly) tables.
Historic data source | Fields that exist only in Live tables | Fields that exist only in Aggregated tables |
---|---|---|
ARP cache entries | n/a | n/a |
DNS resolutions | n/a | LookupCount |
Process executions | CommandLine, ProcessId, ParentProcessId | ExecutionCount |
Process stabilizations | ProcessId, StabilizationTimeMs | ExecutionCount, TotalStabilizationTimeMs |
Software installations | IsUninstall | InstallCount, UninstallCount |
TCP outbound connections | ProcessId | ConnectionCount |
Timestamps
The timestamp column (TS) in each table is stored in Unix Epoch format. Defined as the number of seconds that have elapsed since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970
To convert to a readable text format use the datetime SQL operation.
SELECT Fqdn, datetime(TS, "unixepoch") AS TS_ FROM $DNS_Hourly WHERE Fqdn LIKE "%facebook%";
Timestamps are truncated in the aggregated tables.
- Hourly - time is truncated to each hour - so an event that occurred at 2017-01-27 18:03:54 would be included in the summary for 2017-01-27 18:00:00
- Daily - time is truncated to midnight on each day - so an event that occurred at 2017-01-27 18:03:54 would be included in the summary for 2017-01-27 00:00:00
- Monthly - time is truncated to midnight on the first day of each month - so an event that occurred at 2017-01-27 18:03:54 would be included in the summary for 2017-01-01 00:00:00
ARP cache entries
ETW | Polling | Not supported |
---|---|---|
|
|
|
Field | Datatype | Description | Sample value | Tables |
---|---|---|---|---|
CacheCount | integer | The number of times that the combination of IpAddress, MacAddress and Subnet were seen in the ARP cache for this time period. | 1234 | All |
IpAddress | string | The IP address that was resolved using ARP. | 192.168.11.12 | All |
MacAddress | string | The MAC (physical) address to which the IP address was resolved. | 58-82-a8-93-4c-da | All |
Subnet | string | The CIDR-format IP subnet to which the resolved IP address belongs. | 192.168.11.0/8 | All |
TS | integer | See Agent Historic Data Capture. | 1500756083 | All |
The Agent polls the operating system ARP cache periodically. Since the lifetime of an entry in the ARP cache can be variable, if an entry in the ARP cache is encountered which is already present in the Agent's database, the Agent will increment the CacheCount field on the table for that row, and update the timestamp (TS) field to the current time. To that end, the CacheCount field can be used to determine how frequently a particular entry was observed in the operating system's cache.
DNS resolutions
ETW | Polling | Not supported |
---|---|---|
|
|
|
Field | Datatype | Description | Sample value | Tables |
---|---|---|---|---|
Fqdn | string | The FQDN which is being resolved. | All | |
LookupCount | integer | Sum of resolutions per FQDN within the hour, day, month. | 1234 |
|
TS | integer | See Agent Historic Data Capture. | 1500756083 | All |
When using polling, the local DNS cache is queried for all unique FQDNs. This includes an initial scan of cache entries created before the Agent starts, which are stored with the same timestamp. New entries that appear in the cache are deemed to correspond to new resolutions and stored with the timestamp of when the polling occurred.
When using ETW, the Agent attempts to capture DNS queries at the point that they are made. The query is captured, not the result of that query. That is, the Agent will capture a request to resolve an FQDN which may ultimately not be resolvable. The DNS cache is not scanned.
Process executions
ETW | Polling | Not supported |
---|---|---|
|
|
|
Field | Datatype | Description | Sample value | Tables |
---|---|---|---|---|
CommandLine | string | The full command-line of the process, including (on Windows) the executable name. Sometimes the executable name part of the command-line is quoted, sometimes it's not - it's arbitrary based however the parent process launched the child; so you may see a mix of command-lines like...
| "C:\Windows\system32\VmConnect.exe" "1EUKDEVWKS1231" "TCH-CLI-WXPX86" -G "B2C72520-BBC6-4736-BBBC-5CCF50FE6666" -C "0" |
|
ExecutableHash | string | The MD5 hash of the process executable. | dae0bb0a7b2041115cfd9b27d73e0391 | All |
ExecutableName | string | The filename (including extension) of the process executable. | vmconnect.exe | All |
ExecutablePath | string | The path and filename of the process executable. On Windows, this is the NT-device format version of the path (as a process does not necessarily need to have been launched from a device which has a drive-letter mapping). | \device\harddiskvolume8\windows\system32\vmconnect.exe | All |
ExecutionCount | integer | Sum of executions per executable within the hour, day, month. | 1234 |
|
ParentExecutableName | string | The filename (including extension) of the executable of the process which spawned this one. | mmc.exe | All |
ParentProcessId | integer | The process ID of the process which spawned this one. | 2088 |
|
ProcessId | integer | Operating-system dependent process ID. | 178 |
|
TS | integer | See Agent Historic Data Capture. | 1500756083 | All |
UserName | string | The name of the user in whose session the process was launched (or blank if it is a system-launched process) | 1E\bill.gates | All |
On Windows, the Agent runs as LOCAL SYSTEM, therefore details of almost every process will be available; however some processes may not be accessible because of permissions.
The Agent captures process starts; it does not track how long the process has been running, or how much CPU-time (or user/kernel/active time) the process has used.
Each time the Tachyon Agent starts it does an initial scan of processes before it starts capturing, and will use that time to record when these processes started.
The UserName field is derived from the session in which the process was executed, and doesn't necessarily reflect the user in whose context the process was executed.
Process stabilizations
ETW | Polling | Not supported |
---|---|---|
| Not yet available | MacOS, Linux, Solaris, Android, Windows XP |
Field | Datatype | Description | Sample value | Tables |
---|---|---|---|---|
ExecutableName | string | The filename (including extension) of the process executable. | vmconnect.exe | All |
ExecutionCount | integer | Sum of executions per executable and username within the hour, day, month. For example, vmconnect.exe run by 1e\user1 and vmconnect.exe run by 1e\user2 will have separate rows and thus will be summed separately. | 53 |
|
ProcessId | integer | Operating-system dependent process ID. | 178 |
|
StabilizationTimeMs | integer | The time taken for the process to be considered stable, measured in milliseconds. This will be a multiple of 100. | 4500 |
|
TotalStabilizationTimeMs | integer | Sum of the time taken to be considered stable per executable and unsername within the hour, day, month. For example, vmconnect.exe run by 1e\user1 and vmconnect.exe run by 1e\user2 will have separate rows and thus will be summed separately. | 864300 |
|
TS | integer | See Agent Historic Data Capture. | 1500756083 | All |
UserName | string | The name of the user in whose session the process was launched (or blank if it is a system-launched process) | 1e\bill.gates | All |
On Windows, the Agent runs as LOCAL SYSTEM
, therefore details of almost every process will be available; however some processes may not be accessible because of permissions. The Agent captures only information that can be accessed by LOCAL SYSTEM
- as such it does not check the UI responsiveness of a process.
By default, process stabilization monitoring is not active. To enable, the process names must be specified in the Agent configuration file as follows:
- Add M
odule.Inventory.ProcessStabilization.MonitoredProcesses=<string>
to the Agent configuration file. - This is a list of comma separated values, and the case is not significant. For example,
winword.exe
andWINWORD.EXE
are treated the same. - The list of monitored processes does not currently have a limit, however adding a large list of processes to monitor can cause performance degradation and the process stabilization time will become less accurate.
A process' resource usage is tracked, and it will be considered stable once it's resource utilisation has stopped fluctuating. The margin in which a process is considered stable can be modified in the Agent configuration file. Changing from default is not recommended.
- This margin is controlled by the
fuzziness
configuration setting. - Add
Module.Inventory.ProcessStabilization.Fuzziness=<integer>
to the agent configuration file. It cannot be lower than 1, and cannot exceed 66. The default is 5.
A process that exits before it is considered stable is not be recorded. Currently, such processes are discarded. A warning is logged when this occurs.
The accuracy of process monitoring decreases if more processes need to be monitored concurrently. For example, accuracy will decrease if many processes are started at the same time. Warnings are logged when this occurs.
The accuracy of the process monitoring decreases if the system is under considerable load, for example high disk or CPU stress.
Aggregation is grouped by the UserName
and ExecutableName
fields. Unlike process executions, process stabilization values for UserName
and ExecutableName
are lower case.
Software installations
ETW | Polling | Not supported |
---|---|---|
|
|
|
Field | Datatype | Description | Sample value | Tables |
---|---|---|---|---|
Architecture | string | The platform architecture of the software product. | x64 | All |
InstallCount | integer | Sum of installs per software product version within the hour, day, month. 0 if uninstalled, or present but not detected as installed. | 1234 |
|
IsUninstall | integer | 0 = install, 1 = uninstall. | 0 |
|
Product | string | The title of the software product that was installed/uninstalled. | Google Chrome | All |
Publisher | string | The publisher of the software product that was installed/uninstalled. | Google Inc. | All |
TS | integer | See Agent Historic Data Capture. The Agent assumes a "new" installation/uninstallation occurred at the point of polling. | 1500756083 | All |
UninstallCount | integer | Sum of uninstalls per software product version within the hour, day, month. 0 if installed, or present but not detected as installed. | 1233 |
|
Version | string | The version of the software that was installed/uninstalled. | 55.0.2883.87 | All |
Each time the Tachyon Agent starts it does an initial scan of install software before it starts capturing. Since the Agent has no way of knowing when this install/uninstall happened, it will mark the event as having occurred "now".
On Windows, software installations are read from the registry key HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall
and HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall
Per-user installations are not yet supported.
Linux does not distinguish between O/S packages (even the kernel) and application packages; they are all software.
TCP outbound connections
ETW | Polling | Not supported |
---|---|---|
|
|
|
Field | Datatype | Description | Sample value | Tables |
---|---|---|---|---|
ConnectionCount | integer | Sum of connections to an IP Address and Port by a process within the hour, day, month. | 123 |
|
IpAddress | string | The target remote IP address of the connection, either an IPv4 or IPv6 address. Windows support for IPV6 is limited; the Agent will capture the connections, but the format used to represent the target IPV6 may differ slightly depending on the mechanism used, and may be subject to change in future versions of the Windows Agent. | 132.245.77.18 [2001:4860:4860::8888] | All |
Port | integer | The target remote port of the connection. | 443 | All |
ProcessId | integer | The operating-system specific identifier of the process which instigated the connection. Not supported for Mac OSX earlier than Mac OSX Lion (10.7). | 11828 |
|
ProcessName | string | The executable filename of the process which instigated the connection Connections originated from system-oriented processes are captured as "(system)" | chrome.exe | All |
TS | integer | 1500756083 | All |
The Agent captures TCP connections, not UDP connections - as UDP is inherently connectionless (each packet sent is effectively a new connection).
Each time the Tachyon Agent starts it does an initial scan of connections before it starts capturing. A limitation of the Windows API is means that all established TCP connections, whether inbound or outbound, are captured; there is no way to distinguish between the two. This means that it is possible for the Agent to double-capture a connection if that connection was established before the Agent stops monitoring, and still exists when the Agent starts monitoring again, for example between Agent restarts. Unlike other capture sources, there is no persistent storage setting to prevent double-counting.
The Agent captures initial "connect" requests, not just successful connection establishment. This means that an attempt to perform a connection will be captured, even if that connection does not complete, for example, because of a timeout, or the server-side does not permit the connection.
Constraints of Legacy OS
In this documentation, the following are referred to as legacy OS. 1E does not provide support for the Tachyon Agent on these OS. This is because Microsoft has withdrawn support for these OS or they are not significantly used by business organizations.
|
|
Please contact 1E if you require support for these legacy OS.
If you experience an issue on these OS, then please try replicating the issue on a supported OS.