Contents

AI Powered Auto-curation

Introduction

Describes the AI Powered Auto-curation feature that improves software inventory results.

Using the AI Powered Auto-curation feature you'll be able to increase the total amount of normalized software in your Inventory repositories. This is done using AI that integrates with the inventory consolidation process. By using AI your organization will benefit from significant numbers of normalized software and a reduced manual effort needed to normalize software.

The Inventory app captures data from different sources, then consolidates, normalizes and stores it in Inventory repositories.

Inventory uses connectors to capture software, hardware and other data from sources such as Configuration Manager, Big Fix and vCenter where it is consolidated, de-duplicated and normalized. Normalization uses software and hardware entries in the 1E Catalog.

Inventory repositories are used by the following applications:

  • Patch Success - reports on and ensures successful patching of your enterprise
  • Application Migration - automates the migration of applications during a Configuration Manager OS deployment
  • AppClarity - manages entitlements, compliance and reclaiming unused software.

The Inventory app is used to view and export inventory, and manage associations.

To find out more about 1E Catalog please refer to 1E Catalog.

On this page:

In previous versions of Tachyon and SLA Platform, 1E Catalog handled normalization. This used rules that map source Programs and Features data into an accurate standardized form of Vendor, Title, Version and Edition (VTVE) fields. These normalized records can then be referred to with the same signature. Although the rules in the 1E Catalog cover most common software, there's always new software that needs to be manually mapped.

AI Powered Auto-curation first extracts the VTVE from varied raw Programs and Features data by removing excess data and extracting the relevant strings. The AI then standardizes the VTVE fields to a single, accurate form of representation. It also produces a numeric value called the confidence score. This score represents how confident the AI is in its prediction. This score is set to a high threshold through testing and validation across a large data set. This can be calibrated according to the variety of software data at a customer site to optimize the normalization rate, this is controlled by 1E through collaboration with customers.

You can enable AI Powered Auto-curation either during or after installation of Tachyon.

AI Powered Auto-curation is used by all Inventory connectors. Please refer to the Connectors page for more information about how Connectors are used to connect to other 1E and third party systems.

Preparation

Before starting, ensure you've met the Tachyon 5.0 - Requirements and completed the Tachyon 5.0 - Preparation steps.

The minimum total disk space required for the downloaded AI Package ZIP is approximately 12.5 GB at installation. This includes:

  • Binary and support files for AI and the directory structure is 8 GB
  • AIPackage ZIP file is 4 GB

While the downloaded AI Package is 5 GB, the unzipped contents are approximately 12.5 GB and are extracted to C:\ProgramData\1E\SLA Platform\AI\

Please note, the required disk space reduces to 8.2 GB after the first Catalog sync with the cloud. During the sync a check is made for any newer AI packages. If there is one, it's downloaded and a hash check of the ZIP file is made to make sure it's not corrupted. Once all checks are complete, the AI Package ZIP is deleted from C:\ProgramData\1E\SLA Platform\AI\ leaving an 8 GB disk space requirement.

1E Catalog 2.0 or later is required for the AI to work. 1E Catalog is included in the Tachyon Platform zip, and installed by Tachyon Setup.

Memory requirements

Distinct Software Titles

Extra GB RAM for AI engine

1 to 5000

15

10000

17

20000

20

30000

24

40000

28

50000

33

60000

38

70000

43

80000

49

90000

55

100000

61

110000

67

120000

74

130000

82

140000

89

150000

97

160000

106

170000

114

If you're only using a System Center Configuration Manager (SCCM) connector then use the SCCM database as a guide for how much extra RAM you need before installation. We recommend using the SLA-Data database query after installation to double-check memory requirements. However, additional RAM will be required if you're going to use a Tachyon connector (for Inventory or Patch Success) either on its own, or with other connectors including SCCM. In this case, use the SLA-Data database query after installation and when the connectors are collecting data.

Physical memory for AI Powered Auto-curation is calculated on the basis of the number of records the AI has to process. Use the accompanying table as a quick reference for the memory requirements needed by the AI.

For virtual machines, the AI Engine action will not work if you're using a dynamic memory configuration. You'll need to allocate dedicated memory.

You can also run the following query on your SCCM database to calculate your memory requirements. This is primarily aimed at people who have not already installed Tachyon and want to estimate the required RAM:

SCCM database query 
/*SCCM database query to get number of distinct software titles from SCCM database*/
USE [YourSCCMDBName]
GO
SELECT COUNT(*) AS 'Distinct Software Titles'
FROM (
	SELECT DISTINCT DisplayName0 AS Title FROM [v_ADD_REMOVE_PROGRAMS] WITH (NOLOCK)
	UNION
	SELECT DISTINCT [Caption0] AS Title FROM [v_GS_OPERATING_SYSTEM] WITH (NOLOCK)
) AS Titles

Run this query on SLA-Data if you already have Tachyon installed and are considering enabling AI:

SLA-Data database query 
/*SLA query to get number of distinct software titles from SLA-Data database*/
USE [SLA-Data]
CREATE TABLE #Software
       (
              [SoftwareID]                      INT IDENTITY(1,1) PRIMARY KEY,
              [DataSource]                      NVARCHAR(255) COLLATE DATABASE_DEFAULT,
              [SoftwareIdent]                   NVARCHAR(MAX) COLLATE DATABASE_DEFAULT,
              [SoftwareIdent_Hash]		        VARBINARY(128),
              [Vendor]                          NVARCHAR(MAX) COLLATE DATABASE_DEFAULT,
              [Vendor_Hash]			            VARBINARY(128),
              [Title]                           NVARCHAR(MAX) COLLATE DATABASE_DEFAULT,
              [Title_Hash]                      VARBINARY(128),
              [Version]                         NVARCHAR(MAX) COLLATE DATABASE_DEFAULT,
              [Version_Hash]                    VARBINARY(128),
              [ColloquialVersion]               NVARCHAR(MAX) COLLATE DATABASE_DEFAULT,
              [ColloquialVersion_Hash]          VARBINARY(128),
              [Edition]                         NVARCHAR(MAX) COLLATE DATABASE_DEFAULT,
              [Edition_Hash]                    VARBINARY(128),
              [NormalizedProductID]             INT
       );
SELECT * FROM #Software
EXEC [usp_ReportDataEx_se] 1, N'Software', N'#Software';
  
SELECT count(*) AS 'Distinct Software Titles'
FROM
(
SELECT DISTINCT [Title] FROM #Software
) a

DROP TABLE #Software

Enabling AI Powered Auto-curation

You can enable AI Powered Auto-curation either during or after installation of Tachyon.

Enabling AI Powered Auto-curation during installation of Tachyon

AI is switched off by default in the Tachyon Setup, this is because the feature requires additional memory to run correctly. For more details please refer to the AI Installation requirements on this page.

During Tachyon setup you'll need to check the Enable AI-based auto-curation checkbox on the SLA and Catalog screen of the Tachyon Setup wizard. For more details please refer to the Tachyon Setup and Tachyon Setup: SLA and 1E Catalog.

The minimum total disk space required for the downloaded AI Package ZIP is 12.5 GB at installation.

While the downloaded AI Package is approximately 5 GB, the unzipped contents are approximately 12.5 GB and are extracted to C:\ProgramData\1E\SLA Platform\AI\

Please note, the required disk space reduces to 8.2 GB after the first Catalog sync.

Enabling AI Powered Auto-curation after installation of Tachyon

You can enable AI Powered Auto-curation after you've installed Tachyon, to do this you'll need to enable it in both 1E Catalog and Tachyon.

Reconfiguring 1E Catalog

The following script can be used to modify the EnableAIPackageSync setting in the Settings table of the 1ECatalog database . You can set the new value to be one of the following:

ValueDescription
TrueDownload will be enabled.
FalseDownload will be disabled.

After changing the value you will need to restart the 1E Catalog Update Service for your changes to take effect.

SQL script to configure EnableAIPackageSync setting
/* Script to change 1ECatalog setting */
USE [1ECatalog]
GO 
DECLARE @setting nvarchar(max), @oldvalue nvarchar(max), @newvalue nvarchar(max);; 
SET @setting = 'EnableAIPackageSync' 
SET @newvalue = 'True' 

SET @oldvalue = (SELECT [Value] FROM [dbo].[Settings] WHERE [Key]= @setting) 

UPDATE [dbo].[Settings] SET [Value]=@newvalue WHERE [Key]=@setting 

SELECT @setting AS 'Setting', @oldvalue AS 'Before', [Value] AS 'After' FROM [dbo].[Settings] 
WHERE [Key]=@setting 
GO

Reconfiguring Tachyon

The following scripts can be used to modify the DisableAIProductPrediction setting in both:

  • ProjectSetting table of the SLA-Shared database - the settings template for new inventory repositories
  • InstanceSetting table of the SLA-Data database - the settings for each of the existing inventory repositories.

You can set the new value to be one of the following:

Value

Description

0AI will be enabled.
1AI will be disabled.


SQL script to configure DisableAIProductPrediction setting
/* Script to change SLA setting */
DECLARE @setting nvarchar(max), @oldvalue1 nvarchar(max), @oldvalue2 nvarchar(max), @newvalue nvarchar(max)
SET @setting = 'DisableAIProductPrediction' 
SET @newvalue = 0 

SET @oldvalue1 = (SELECT [Value] FROM [SLA-Data].[dbo].[InstanceSetting]  WHERE [Name]= @setting) 
SET @oldvalue2 = (SELECT [Value] FROM [SLA-Shared].[dbo].[ProjectSetting] WHERE [Name]= @setting) 

UPDATE [SLA-Data].[dbo].[InstanceSetting]  SET [Value]=@newvalue WHERE [Name]=@setting 
UPDATE [SLA-Shared].[dbo].[ProjectSetting] SET [Value]=@newvalue WHERE [Name]=@setting 

SELECT '[SLA-Data].[dbo].[InstanceSetting]' AS 'Database', @setting AS 'Setting', @oldvalue1 AS 'Before', [Value] AS 'After' 
FROM [SLA-Data].[dbo].[InstanceSetting] 
WHERE [Name]=@setting 
UNION
SELECT '[SLA-Shared].[dbo].[ProjectSetting]' AS 'Database', @setting AS 'Setting', @oldvalue2 AS 'Before', [Value] AS 'After' 
FROM [SLA-Shared].[dbo].[ProjectSetting] 
WHERE [Name]=@setting


Verifying AI Auto-curation is working

The AI feature consists of an 1E-AIEngine.exe and AI model files contained in an AIPackage_<version>.zip. The 1E-AIEngine.exe uses the associated AI package files to run the AI. Because of the size of the package, it initially resides on the Catalog cloud to reduce the installer size. 

When the AI feature is enabled:

  • 1E Catalog checks that it has the latest AI package synced from the 1E Catalog cloud. This is downloaded and saved in: C:\ProgramData\1E\Catalog\CatalogSynchronization\PredictionModel
  • Tachyon Inventory checks with 1E Catalog that it has the latest AI package ZIP which is then unzipped to: C:\ProgramData\1E\SLA Platform\AI

The minimum total disk space required for the downloaded AI Package ZIP is approximately 12.5 GB at installation. This includes:

  • Binary and support files for AI and the directory structure is 8 GB
  • AIPackage ZIP file is 4 GB

While the downloaded AI Package is 5 GB, the unzipped contents are approximately 12.5 GB and are extracted to C:\ProgramData\1E\SLA Platform\AI\

Please note, the required disk space reduces to 8.2 GB after the first Catalog sync with the cloud. During the sync a check is made for any newer AI packages. If there is one, it's downloaded and a hash check of the ZIP file is made to make sure it's not corrupted. Once all checks are complete, the AI Package ZIP is deleted from C:\ProgramData\1E\SLA Platform\AI\ leaving an 8 GB disk space requirement.

You can review the Catalog sync progress in the following logs located at: C:\ProgramData\1E\Catalog\

  • Catalog.Integration.UI.log 
  • Catalog.UpdateService.log 

A succesful sync should be similar to the example Catalog.UpdateService.log opposite.

Catalog.UpdateService.log 
[Info] T11 2020-01-31 15:45:48.567: Syncing AI Package with cloud
[Info] T11 2020-01-31 15:45:48.756: Downloading AI Package version 1.3.0.700 with cloud of size 4804685823 from url  api/sync/aiengine?Version=1.3.0.700
[Info] T11 2020-01-31 15:45:48.772: Indexes re-creation started after Sync.
[Info] T12 2020-01-31 15:58:25.929: Indexes are created successfully after Sync.
[Info] T5 2020-01-31 16:30:47.930: Download complete. Comparing file hash..
[Info] T5 2020-01-31 16:31:07.120: Successfully synced AI Package version 1.3.0.700 with cloud

When enabled, a new action called Process AIEngine is created. This is automatically run after the Sync Data action for a supported connector. The sync time for the Process AIEngine action depends on the number of distinct software records being processed, hardware and environment configuration.

AI Powered Auto-curation is used by all Inventory connectors. Please refer to the Connectors page for more information about how Connectors are used to connect to other 1E and third party systems.

You can see the Process AIEngine action in Settings→Monitoring→Process log.

Process AIEngine

You can review its progress using the Engine.log located at C:\ProgramData\1E\SLA Platform\ 

In the log you should see something similar to the opposite example.


Engine.log 
2020-03-31 08:46:25.0274 INFO  Action Process AIEngine starting! Jobs: 1/5
2020-03-31 08:46:25.0274 INFO  ID=612d1b08-7eb7-4bf1-9c23-46dea29d9b6d Starting action: 11065
2020-03-31 08:46:25.0274 DEBUG  [HubConnection] Adding notification '11065' to queue. Queue count:3.
2020-03-31 08:46:25.0274 INFO  Folder's available in PreProcessAsync: output- True, input- True, tempDir- True, training- True
2020-03-31 08:46:25.0500 INFO  Catalog web address: http://LAB.LAB.local/CatalogWeb/
2020-03-31 08:46:25.0704 INFO   Is higher version False, localVersion 1.3.0.700, serverVersion 1.3.0.700
2020-03-31 08:46:25.0704 INFO  serverFileDetails.Base64filehash 0x853E29401B209301C9F9C4D381202C931816C871, modelFileDetails.Base64filehash 0x853E29401B209301C9F9C4D381202C931816C871
2020-03-31 08:46:25.0724 INFO  Condition for download before checking: False
2020-03-31 08:46:25.0724 INFO  Saving Model file hash (0x853E29401B209301C9F9C4D381202C931816C871) and version (1.3.0.700)
2020-03-31 08:46:25.1124 DEBUG  [HubConnection] Notification sent '11064'
2020-03-31 08:46:25.1484 DEBUG  [HubConnection] Adding notification '0' to queue. Queue count:2.
2020-03-31 08:46:25.1484 INFO  Folder's available in ProcessAsync: output- True, input- True, tempDir- True, training- True
2020-03-31 08:46:25.1504 INFO  Folder's available in Before Invoking AI EXE: output- True, input- True, tempDir- True, training- True
2020-03-31 08:46:25.1504 DEBUG  [HubConnection] Notification sent '0'
2020-03-31 08:46:25.1504 INFO  process started - 9636process name is 1E-AIEngine
2020-03-31 08:46:25.1504 INFO  in wait for exit async
2020-03-31 08:46:25.1504 INFO  In Read Async
2020-03-31 08:46:25.1724 DEBUG  [HubConnection] Notification sent '11065'
2020-03-31 08:46:25.1967 DEBUG  [HubConnection] Notification sent '0'
2020-03-31 08:51:05.5709 INFO  Folder's available in After Invoking AI EXE: output- True, input- True, tempDir- True, training- True
2020-03-31 08:51:05.5709 INFO  AI process completed with exit code: 0

If you see a red warning triangle in the Process log, this indicates a problem.

The most common reason for the Process AIEngine to fail is due to insufficient memory when it runs, refer to Increase in memory requirements on this page for details. 

In this example, clicking on the red warning triangle displays an error message explaining that the Process AIEngine has failed.

AI Process Engine failed

Process AIEngine failed

You can review the AI logs at: C:\ProgramData\1E\SLA Platform\AI\Log_*.txt

The picture opposite shows an example of a failure logged due to insufficient memory. Log_*.txt

If the AI processing step does fail, the overall inventory consolidation will not. In the event of a failure, the AI processing step is skipped, making sure the rest of the process takes place as expected.


 Log_*.txt > Expand

AI log memory error example

Differences in SLA Inventory screens when AI Powered Auto-curation is enabled 

There are some changes to the SLA Inventory UI screens compared with Tachyon 4.1 and SLA Platform 3.3. These screens also differ if AI Powered Auto-curation is enabled or disabled.

The process of setting up connectors, creating inventory repositories and running the basic inventory consolidation remains the same as previous versions of Tachyon.

Basic Summary screen

You can access the Basic Summary screen using the Inventory app and by choosing SLA Inventory from the left-hand menu. The data in this screen is updated by the Basic Inventory Consolidation process.

The Summary table now includes software records curated by AI. The first column in the table shows the total Records Collected. The next three columns are:

  • Normalized Records-Direct Catalog Matches (Distinct Products) - corresponds to software records identified using existing rules in the 1E Catalog. Exploring these will show you the normalized entries with enriched data regarding licensability, end-of-support date etc.
  • Normalized Records-AI Assisted Matches (Distinct Products) - the software records which AI has automatically matched and normalized. The AI recognizes the vendor, title, version and edition and normalizes them. Over time, the 1E Catalog reviews records identified by AI and enriches them with data regarding licensability, end-of-support date etc. for inclusion in future downloads.
  • Non-Normalized Records (With AI Suggestions) - software records which AI has low confidence in automatically matching. AI makes predictions for all source data, but predictions with a low confidence score should be manually reviewed by drilling down to the Unidentified Software Records screen.  

The numbered links in brackets allow you to drill down to more detailed screens.

Basic Summary Screen

Unidentified Software Records screen

You can access this screen by clicking on the number link in the Non-Normalized Records (With AI Suggestions) column of the Basic Summary screen. This screen shows AI predictions that have a low confidence score preventing them being automatically matched and normalized. These predictions should still be close to accurate, therefore making the task of manually adding rules to the Catalog much easier.

This screen has the following elements:

Overview 

Shows a count of distinct software records to be identified, so you can keep track of progress.

  • Software Products left to identify - show all records
  • AI predicted - records the AI was unable to identify, but was able to provide suggestions
  • AI predicted and partially matched - where the vendor or title was identified through previous knowledge from the 1E Catalog, these records have a default high confidence score
  • Cannot be identified - these records have insufficient information to recognize the software or make a prediction.

Application Type

Filters by application type.

  • Show only Secondary Applications - filters to only secondary applications (for example, updates, hotfixes or language packs)
  • Newly discovered (Last 30 days) - filters to applications discovered within the last 30 days
  • New to Catalog - filters to applications that are either new vendors or titles previously unknown to the 1E Catalog
  • Exclude Secondary Apps - removes any applications considered secondary, like updates, hot fixes or language packs.

AI Prediction Confidence

Confidence scores for predictions are divided into three equal parts below the threshold. High, Medium and Low confidence relate to the ease at which the AI engine suggestions can be manually reviewed and added as rules.

FILTER BY

Use this to add conditional filters for various fields on the Unidentified Software Records Table, for example you could filter on "Vendor Contains Micro" to return results based on Microsoft as a software vendor.

Table Filters

AI Predictions for Unidentified Software Records

If the AI feature is disabled the Basic Summary screen will differ to when AI is enabled, note the difference in the Overview element.

AI Predictions for Unidentified Software Records AI disabled

Unidentified Software Records Table

Each row on this table corresponds to a distinct set of software records. There are four values Install Count,  Source Data ,  AI Prediction ,  Confidence  and a number of actions to perform for each distinct record. The values are:

ActionExplanation
Install CountThe total number of records found with the same source data across the estate.
Source DataARP data extracted into Vendor, Title, Version, Edition, Colloquial Version and Licensable fields.
AI PredictionThe AI suggestion after attempting to extract Title, Version, Vendor, Edition, Colloquial Version and Licensable fields from source data, Licensability is set to a default of No on the suggestions table, this can be edited before adding.
ConfidenceAbstracted to high, medium or low based on the exact numeric value for prediction confidence, these confidences values will always be lower than the threshold required to automatically normalize records.

For each record the  Actions  are:

ActionExplanation
AddWhen clicked this will add records directly to 1E Catalog. This option is enabled if AI suggestions for a source Vendor, Title, Version and Edition (VTVE) are generated. In case the product is not Edition based then Vendor, Title and Version (VTV) suggestions are enough. After the record(s) is added to 1E Catalog then whenever you run the next Basic Inventory Consolidation then the same record(s) will be available in the Catalog Matched items.
EditEdit functionality is available for all unidentified records. When clicked, it opens a dialog where you can provide details for the selected software record. You also have the option to add this information to 1E Catalog.
ExcludeExclude is available for all Unidentified Software Records except for Partially Matched and Secondary Application types of records. Once excluded, the items are moved to Secondary Applications. You can also Include it again until the next Basic Inventory Consolidation is run, afterwards the Include option will be unavailable and the item will become a part of Secondary Applications.

You can explore these records to see what the AI has curated by clicking on the related links in the  Basic Summary Report .

We have also introduced a new screen for AI suggestions for any software records that remain non-normalized. This is a replacement for the Best Match Screen which allowed you to add rules.