On this page:

Offloading

Offloading responses is the process of directing the responses from your request from Tachyon into a different location or application. There are times that I want my responses to stay in Explorer and there may be times I want my responses to be sent to a different location or application. This exercise will show you how to configure offloading using 1E's Offloading Client. You will then be able to use Offloading in your own consumer. Remember that once you offload the responses they will no longer be available to the Explorer application. The offloading client (also called the Offloading Relay) can be used to forward responses to other third party applications (such as Splunk). The offloading client can also be extended because it uses a mechanism of pluggable providers. This Offload Client is meant to be an example for you to see offloading of responses in practice.  The concepts in this exercise will transfer to the application that you will be writing after learning the concepts.

Copy the Application Files

1ETRNAP
  1. Open the SkyTap Shared Drive shortcut on the desktop and Open the 1E Tachyon - Course Content\1E Tachyon v5.1 - Course Content folder. Download Tachyon-OffloadingClient.v3.3.0.396.zip
  2. When the download finishes move the file to c:\temp and Extract the files from the ZIP
  3. Open the Folder that contains the Extracted files and copy the OffloadingClient folder to c:\Windows

Create the Web Application

1ETRNAP
  1. Open IIS Manager
  2. We need to use an Application Pool that is configured for CLR 4.0 and has an Integrated Pipeline so we will create one now.
  3. Right click on Application Pools and select Add Application Pool
  4. In the Name field type in Offloading Client
  5. Click Ok
  6. Select the Offloading Client app pool and click on Advanced Settings in the far right menu
  7. In the Process Model section - Identity line click on ApplicationPoolIdentity then click the elipses so that we can change that value.
  8. In the Built-in account field select Network Service. Click Ok and click Ok again
  9. Right click on Sites and choose Add Website

  10. In Site Name field type in OffloadingClient
  11. Click Select next to the Application Pool field and choose OffloadingClient click Ok
  12. In the Physical Path field click the ellipses button and browse to c:\windows\offloadingclient
  13. Click on Test Settings. You should get green Authentication and The Application Pool identity is valid. Authorization will not be able to be verified. Click Close
  14. In the Binding section change the port to 8080
  15. Click Ok

Configure the Offloading Client


1ETRNAP
  1. Navigate to c:\Windows\OffloadingClient and Open the Web.Config in Notepad
  2. Scroll down to the section called <dataProviders>
  3. We have written three providers for use by the Offloading Client. The providers are what will do the work to get the responses and either display them or send them to another system.
    ProviderDecription
    Memory

    For Demo only and not intended for production use.

    The memory provider stores all received responses in the Application object of the Tachyon.Server.OffloadingClient application. The application provides a set of browsable pages where you can view the number of results received, their timing, and their raw content. The links to these pages are removed when the Memory provider is not enabled in the .config file.

    The memory provider accepts a single <setting> called "LimitEntries" (set to 1000 in the example above). This establishes the maximum number of responses that will be kept in memory in the server. Once the limit is reached, further responses are discarded, until the existing ones are cleared (the Home page provides a link for doing so).

    FileSystem

    This provider sends all the responses that are received to CSV files on the local disk.

    The <settings> are as follows:

    • RootFolder sets the root of the folder hierarchy under which all the files will be saved. You need to ensure that the account configured for the IIS Pool that runs the offloading client has sufficient permissions for writing into that folder.
    • FolderNameFormat establishes the pattern for the folders that will be created to store the CSV files. The Filesystem provider creates one folder for each instruction, and then inside the folder it adds one CSV file for each batch of responses received from Tachyon. Several optional tokens within the FolderNameFormat allow you to determine the naming for these folders. The example above, "Instruction_{Id}" causes the folders to be named with the word "Instruction" followed by the Id that Tachyon assigned to the instruction.
    • FileNameFormat determines the names for the files that are created within each folder. The pattern shown in the example, ""{Now}-{Guid}.csv" will cause the files to be named using the current date and time (so that they will be listed in chronological order in the Windows Explorer), followed by a Guid that serves to desambiguate the names in case two batches are received at the same time.

    The following tokens can be used (they are case-sensitive):

    {Id} - The Id of the Instruction as set by Tachyon.
    {Seq} - The Sequence number of the Instruction as set by Tachyon.
    {Timestamp} - The Timestamp of the Response as set by Tachyon.
    {Now} - The current date and time formatted as yyyyMMddHHmmss.fff
    {Guid} - A random GUID generated on-the-fly.

    Syslog

    This provider sends the responses to a Syslog server.

    The <settings> are as follows:

    • DefaultSyslogServer indicates the address or DNS name of the Syslog server. The example uses the local server, but you will usually have your Syslog server installed in a different computer.
    • DefaultSyslogPort is the UDP port used by your Syslog server. The standard port is 514.
    • DefaultSyslogFacility is the value that will be sent to the Syslog server for the "facility" parameter.
    • DefaultSyslogSeverity is the value that will be sent to the Syslog server for the "severity" parameter.
    • DefaultSyslogMsgId is the value that will be sent to the Syslog server for the "Msg Id" parameter. This is an arbitrary piece of text; the example uses "TACHYON".


    Our Offloading Client is configured with three providers for an example scenario in order to show the functionality. In a real-world scenario you would only have a single provider. If there are multiple providers the Offloading Client will deliver the responses to all of them to enable learning.
  4. Notice for each provider we have a Name attribute. The name can be anything as long as it is unique in relation to the other providers.
  5. Notice for each provider we have an assemblyName attribute. This indicates the DLL where the provider can be found. Note that our assemblyName is the same for all the providers and is the same DLL that contains the Offloading Client website.
  6. Notice for each provider we also have a providerName attribute this indicates the Class within the DLL that implements the Provider and must be its complete namespace.
  7. Each provider can also contain various entries under <settings>. The specfic values allowed depend on each individual provider type.
    By default IIS limits the maximum size of the HTTP requests that it receives. The limit depends on the version of IIS you are using. You need to understand the potential sizes of your responses because if a response is larger than the limit, IIS will reject the request and the Offloading Client will never see that response.
  1. Locate the section in the file that contains the Filesystem provider. We need to change the location of the .CSV file that the Offloading Client will create for our responses.
  2. Change the RootFolder value to c:\temp\responses
  3. Save and Close the Web.config
  4. In File Explorer create a Responses folder in c:\temp. Give Network Service full control to the folder
  5. The identity that your application pool is running under will be creating the file and saving it to the folder location.
  6. Restart the OffloadClient website in IIS Manager

Using Tachyon RunInstruction Command Line Tool

We will be using the RunInstruction Command Line tool to learn how offloading the responses works.  We must first install the tool.

1ETRNAP

  1. Navigate to c:\temp\TachyonPlatform.v5.1.0.663\Installers
  2. Double-click TachyonToolkit.msi to install the toolkit (select all the defaults)

Add the Offloading Client Consumer

In order to configure our Offloading Client we must create a consumer in Tachyon and point to our Offloading URL. To find your Offloading URL we will open the Offloading Client Web Page

1ETRNAP
  1. In Google Chrome navigate to our Offloading Client Web Interface by going to http://Tachyon.1etrn.local:8080
  2. Note the Offloading Target URL. Copy this value.
  3. In a new tab navigate to https://Tachyon.1etrn.local/Tachyon and open the Settings Application and navigate to Configuration - Consumers
  4. Create a new consumer by clicking Add in the far right
  5. In the Name field type in TachyonRunInstructionOffloadingClient
  6. In the Maximum Simultaneous Instructions field type in 25
  7. In the Offload Target URL field paste the Offloading Target URL from the Offloading Client Website
  8. Check the boxes for Use Windows Authentication and Enabled
  9. Click Save

Using the Offloading Client

In order to see the Offloading Client in action first we will ask a question using the Instruction Runner and offload the results to our Offload URL

1ETRNAP
  1. Open a Command Prompt and change directory to c:\Program files (x86)\1e\Tachyon\Toolkit\TachyonRunInstruction and run the following command line
  2. Tachyon.RunInstruction.exe -I 1E-Explorer-TachyonCore-AllInstalledSoftware -C https://Tachyon.1ETRN.local/Consumer -N TachyonRunInstructionOffloadingClient -O -D 1etrnW71.1etrn.local -U Tachyon_AdminPP Passw0rd
    This submits the instruction 1E-Explorer-TachyonCore-AllInstalledSoftware. The -C is used to designate the Tachyon Consumer API URL. The -N parameter specifies the Consumer Name that we configured in Tachyon in the prior steps. The -O indicates the results should be offloaded (to whatever address was configured for the consumer) instead of saved to the Tachyon Responses database. The -D is our coverage. The -U is the username and password that we want to use to issue the instruction.
  3. Open the Offloading Client Website and view the results
  4. Open file explorer and navigate to C:\temp\responses
  5. Note the names of the folders. The number after Instruction is the Instruction ID (this naming convention is controlled by the FolderNameFormat value in the Offloading Client's web.config file. The file name format is also set in the web.config file and by default is the date and time (formatted yyyyMMddHHmmss.fff followed by a randomly generated GUID.
  6. Navigate to C:\ProgramData\1E\Tachyon\Tachyon.RunInstruction.log to view the logging.
  7. By default the Tachyon Instruction Runner is configured to Info level logging. This can be increased for troubleshooting purposes by editing the Program Files\1E\Tachyon\Toolkit\TachyonRunInstruction\Tachyon.RunInstruction.Exe.Config in the Log4Net section level - current value is INFO. You can change to DEBUG or TRACE for increased amounts of logging.

Exercises On Your Own

Extending the Offloading Client

You can write your own provider and add it to the offloading client using the following steps:



  1. Create a class library project in .Net,
  2. Add a Reference to Tachyon.Server.OffloadingClient.dll and any dependencies such as Tachyon.SDK.Offloading.dll.
  3. Add a class that implements the Tachyon.Server.OffloadingClient.Providers.IDataProvider interface.
  4. Compile, and install the resulting DLL into the bin folder for the Offloading Client web application.
  5. Add an entry to the dataProviders section of the web.config file. Point it to your assembly and your class.

Lab Summary

In this lab we learned how to configure a consumer for Offloading of responses.