1E 9.x (on-premises)

1E stores its Device Deduplication configuration in: <INSTALLDIR>\Tachyon\Coordinator\Tachyon.Server.Coordinator.exe.config

The time and frequency of when the Device Deduplication process runs is changed by adding and editing the DeviceDeduplication setting:

<module assemblyName="Tachyon.Server.Coordinator" providerName="Tachyon.Server.Coordinator.Scheduling.SchedulingModule">
  <settings>
    <add key="Crontab1" value="50 23 * * * LogDevicesSeenInTheLast7Days" />
    <add key="Crontab2" value="15 * * * * MaintainPolicyRulePartitioning" />
    <add key="Telemetry" value="40 23 * * 5 SendTelemetryStats" />
    <add key="DeviceDeduplication" value="0 1 * * 0 DeviceDeduplication {DeleteDevices:'false',UpdateDeviceData:'false',DeleteOrphanedDeviceData:'false',DeviceIdentifier:['FQDN']}" />
  </settings>
</module>

The numbers that you see after value is a crontab schedule expression. The schedule 0 1 * * 0 means "at 01:00 hours, any day of the month, any month of the year, on weekday 0 (Sunday)". In other words, this configuration runs the Device Deduplication process at 1AM UTC every Sunday night.

You can use an online tool such as https://crontab.guru/ to verify your crontab schedule expression.

Running with all "Update..." and "Delete..." arguments set to false will make no database changes and is considered a diagnostic/practice run. Following a run such as this the DeviceDeduplication and DeviceDeduplicationLog tables could then be analyzed to figure out how many duplicates are in the environment based on the DeviceIdentifier input. This would provide a projected impact for a subsequent run with one or more of these arguments set to true. Alternatively, the user could then also choose to modify the DeviceIdentifier value to include more/less/different columns that would provide a more fine tuned target of what the user expects a truly unique device to be.

The following table describes the system behavior for every permutation of boolean arguments passed to DeviceDeduplication:

*1 - This could lead to continued inflated device counts in the system

*2 - Be careful as this data will be deleted and cannot be recovered or merged at a later date even if the TachyonGuid that reported it comes online again.

If any of the Boolean arguments are set to true, Device Deduplication may affect existing data in the system, whether through relinking to a new TachyonGuid or removing as orphaned data. For this reason, when the Device Deduplication process completes (with any settings set to true), it triggers a full Experience Synchronization. By default, the regular Experience Synchronization process is incremental, and runs at 2AM UTC daily.

The full Experience Synchronization includes a full process of the TachyonExperience aggregated data, which is initiated regardless of the current ProcessMode option specified (GlobalSetting table in the TachyonExperience database). A full process is required because data which typically 1E handles as additive can have historical data modified by the DeviceDeduplication process, which could otherwise lead to inaccurate counts and aggregations.

For large environments the full process of the aggregable data is likely to be a long running background process, it will not impact data displayed in the UI until it has completed. If it conflicts with an ongoing Experience Synchronization process it will be skipped. Server resource demand is likely to increase during an aggregation process. Be aware that depending on the resources available, this could affect the query response time of the UI.

The latest Accumulated Hotfix for 1E Platform Server allows you to add the ExperienceSync setting to the Coordinator configuration file. You do not need to add or modify this unless advised by 1E.

<module assemblyName="Tachyon.Server.Coordinator" providerName="Tachyon.Server.Coordinator.Scheduling.SchedulingModule">
  <settings>
    <add key="Crontab1" value="50 23 * * * LogDevicesSeenInTheLast7Days" />
    <add key="Crontab2" value="15 * * * * MaintainPolicyRulePartitioning" />
    <add key="Telemetry" value="40 23 * * 5 SendTelemetryStats" />
    <add key="DeviceDeduplication" value="0 1 * * 0 DeviceDeduplication {DeleteDevices:'false',UpdateDeviceData:'false',DeleteOrphanedDeviceData:'false',DeviceIdentifier:['FQDN']}" />
    <add key="ExperienceSync" value="0 2 * * * ExperienceSynchronization" />
  </settings>
</module>

The numbers that you see after value is a crontab schedule expression. The schedule 0 2 * * * means "at 02:00 hours, any day of the month, any month of the year, any day". In other words, this configuration runs the Experience Synchronization process at 2AM UTC daily.

You can use an online tool such as https://crontab.guru/ to verify your crontab schedule expression.

The Device Deduplication process creates logs in the Tachyon.Coordinator.log file.

The DeviceDeduplicationLog table - in the TachyonMaster database - logs more granular detail including the arguments used for the instance, details of each pass if more than one is specified, numbers of duplicates found, as well as rows affected in specific tables by various actions.

The DeviceDeduplication table - in the TachyonMaster database - stores a list of devices to be kept and their duplicates for the instance. The ToBeDeleted column is either 0 (false) for a device to be kept, and 1 (true) for a duplicate to be deleted. Kept and duplicate devices are associated by the Hash column.

With all arguments set to false you can review whether the specified DeviceIdentifier is able to uniquely identify devices and genuine duplicates. Once you are sure the DeviceIdentifer is suitable, you can then change other arguments from false to true according to your needs.

When the DeleteDevices argument is set to true, the Device table is updated for each device where older duplicates of it have been identified. Each device will have a timestamp set for LastDeduplicationUtc column, as well as an Experience event logged. The timestamp is updated when one or more new duplicates are found. There is also a DeduplicationCount column which represents the number of complete Device Deduplication processes where a device has been found to have duplicates. These values will only be set on the device identified as being the most recent device that will be kept, not it s duplicates.

Devices with a LastDeduplicationUtc value can be found using the below SQL query.

Genuine devices that have had duplicates identified

SELECT * FROM [TachyonMaster].[dbo].[Device] WHERE LastDeduplicationUtc IS NOT NULL and DeduplicationCount IS NOT NULL

The Experience event can be viewed within Experience Analytics (Devices → {Select a device} → Logs).

The details of the log will show what attributes were used when it was identified as having duplicates, if multiple passes were specified they will be delimited with a '#' in the order of execution. The IdentifyingHash value can be used to query the DeviceDeduplication table Hash column, in the TachyonMaster database.

This will provide all device details of the device itself as well as all of it's duplicate devices.