Content

Overview

Installing Talend

Creating a LabVantage Workspace

Adding the sapphire.jar

Context Variables

Attachment Handler Contexts

Report Handler Contexts

Obtaining Set Up Variables

Context Variables

Attachment Handler Contexts

ReportHandlerContexts

Obtaining Set Up Variables

Building a Job Using Talend
Development Mode
 

Standard Component Documentation

tLabVantageAction

tLabVantageQueryInput

tLabVantageSDIInput

tLabVantageFileOutput

tLabVantageLinkSDI

tLabVantageMetaData

tLabVantageDocumentParser

tLabVantageResultGrid

LIMSCI Migrations

 

 

 

 

 

Overview

Top ../images/arwup.gif (846 bytes)

Talend is third-party software that can be used to graphically design a parser. The purpose is to be able to extract, transform, and load data from files stored as attachments. Attachment Handlers will employ jobs that you have compiled in Talend. Run File Reports can also take advantage of this technology. This allows you to design the output needed from LabVantage in order to send data to an instrument in the form of a file.

This document will cover installing Talend, getting a workspace set up and using all the LabVantage components. To complete these instructions, you will benefit from a Talend Workspace provided by LabVantage. This will include components for implementations that are using Talend for the first time. Migrating Jobs from LIMSCI may require you to add legacy components also available from LabVantage.

 

Installing Talend for Data Integration

Top ../images/arwup.gif (846 bytes)

 

Begin by downloading the installer from https://www.talend.com/products/data-integration/data-integration-open-studio/

Once downloaded, install to your C:\Talend folder and provide the rest of the defaults for the installation.

Obtain the sapphire.jar file from the application server where you installed LabVantage. This will be in the LabVantage home path.

 

Creating a LabVantage Workspace

 

 

If you have launched the Talend program, exit the program now.

Obtain the SDMSWorkspace.zip from LabVantage and extract it to your "C:\Talend\workspace\" folder (where C:\Talend\" is the Talend root installation). This creates a "C:\Talend\workspace\LABVANTAGE_SDMS\" folder. Take the "SDMSComponents.zip" and extract it into the "components" folder under the "LABVANTAGE_SDMS" workspace folder.

Prior to LabVantage 8.5, complex instruments were integrated with LabVantage using a component called LIMSCI. If you used this and you desire to migrate your drivers to LabVantage 8.5, you might want to optionally import additional components. If you require the LIMS CI migration components then take "SDMSLIMSCIComponents.zip" and extract it into the same "C:\Talend\workspace\LABVANTAGE_SDMS\components" folder.

Now launch Talend and select switch projects/workspace.

Select the "LabVantage_SDMS" project and hit Finish.

In some cases, you may have to use the "Import an existing project" option to import that workspace before you can select it on your list.

 

 

A successful import of the LabVantage workspace will give you seven example job designs in your repository (left panel) and eight LabVantage components (right panel). If you have optionally imported the LIMS CI components, six additional LabVantage components are available to you for a total of 14.

 

 

Adding the sapphire.jar

 

 

Using your Talend menus, navigate to Window > Preferences

Within the Preferences dialog, expand Java then Build Path and select User Libraries

 

 

On the right, click the New button. Provide a name for this library. Do not tick "System library"

 

 

Click OK.

On the right, click the Add External JARs... button.

Browse your file system for the place where you have copied LabVantage's sapphire.jar

Once you are finished, your User Libraries should look similar to this:

 

 

Click Apply and Close

 

Context Variables

Top../images/arwup.gif (846 bytes)

When a Talend Job is executed inside LabVantage as an Attachment Handler or Report, there are several context variables sent to the Job which can be accessed with the "context.variable" syntax in Talend. Here is an example:

 

 

Attachment Handler Contexts

 

 

This is a list of context variables that you can pass to Talend. These can be used when testing a job in developer mode using the Talend before loading the code into LabVantage. The purpose of these is primarily to support the extraction, transformation, and loading of data from a file that is attached to an SDI in LabVantage. These files are usually instrument test results but could be of other origins.

 

Report Handler Contexts

 

 

This is a list of context variables that you can pass to Talend. These can be used when testing a job in developer mode using the Talend before loading the code into LabVantage. The primary purpose for these is to be able to extract data from LabVantage, transform it to an output that can be read by an instrument, and supply a file to an SDMS Collector.

 

Obtaining Set Up Variables

 

 

When executing the Job inside LabVantage any Attachment Handler/Operation set up variables or Report Properties are passed into the Context as property_variableid where variableid is the set up variable name or report property name. You are then free to access them through the context. You can simulate this to test your Job in Talend. Te limitation of this is:

  1. Names are case sensitive
  2. You need to provide the "property_" prefix
  3. Executing the Job outside of LIMS and in dev mode means that these context variables are not loaded up until the first Labvantage Component is executed in the Job (which might be the last component in your job).
    For example, if your job parses a file to process data, the context values will not be available when reading the file until the end. Therefore we recommend using our provided API to do retrieval of any set up or report variables.

Use the LabVantageUtil class (sapphire.talend.LabVantageUtil) provided in Sapphire.jar. Inside Talend you will need to either import Sapphire.jar as an external library or do this as part of the install above when setting up our LabVantage Components. You can then use "sapphire.talend.LabVantageUtil.getVariable" static method when required (please also note that you may chose to import the "sapphire.talend.LabVantageUtil" class by using the standard tJava Component and under the advanced settings adding the "sapphire.talend.LabVantageUtil" as an import. All components downstream of this will then allow just LabVantageUtil.getVariable to be used. Here are a couple of examples:

Remember that in development mode, no properties will be available unless you provide a report id or attachment handler id as context variables for the Job. At run time in LabVantage these are passed in.

The LabvantageUtil also has two other static methods, one to fetch a dataset based on query and another to execute an action.

Building a Job for LabVantage

Top../images/arwup.gif (846 bytes)

 

All LabVantage components will build and run outside of LabVantage. However when run in Talend, the components will run in debug mode and output log information, but will not perform any functionality. For the LabVantage components to function the Job needs to be built then imported into an Attachment Handler or Report following these instructions.

First open the Job file you wish to compile and make sure it runs using the Run button in the toolbar:

Observe the run is successful and the log shows no errors:

 

 

From the left hand Repository Tree right click on your job file and select "Build Job". In the Build Job dialog specify the following:

 

 

Click the Finish button and wait until the build is complete and the dialog disappears.

Before continuing take a note of the class full name by visiting the "Code" tab in the editor, taking the package name and appending the generated class name:

 

 

In the example above the full class name would be "labvantage_sdms.autotasterresults_0_1.AutoTasterResults".

Now navigate to the folder specified in the Output path for the export Job. Inside this path you should see a folder by the job name and a "lib" folder.

First take the Jar file from the Job name folder which has the package name in the filename and copy to a centralized location:

 

 

Next if there are any 3rd party libraries the Job requires which are found in the Lib folder then copy them to the centralized location also. You have the following choices to distribute your additional third-party jars:

In the example below we only require talendcsv.jar for the Job to work as all others are provided:

 

 

Dynamic class loading is disabled system wide by default. To use it, you need to go into the Security Policy and find the "Java Based Attachments" section. Set "Enable On Attachment Handlers" and "Enable on Reports" to Yes. See Policies Help Topic for more details. You will also need to check that a role is specified in the "Java Attachments Role" and that your user has that role.

 

 

In this document we will cover adding your Jars dynamically.

 

Using the Upload File button, attach the library files you are executing.

 

 

Add any example files that you know the Talend Job works with in the Example Files tab and click Save.

 

 

To validate your new Attachment Handler, click the Test Handler button and click OK on the prompt without selecting an attachment class. This button will create an adhoc Data Capture SDI, attach the example file, add itself as the operation on the Data Capture and process it.

 

 

When the page refreshes with the green information box, click the link to the adhoc Data Capture to open it. Jump to the executions tab on the Data Capture to view the execution information on that Talend Job.

 

 

You can click the "Log" column to see the detailed log from Talend. You can also change the attachment and click the re-execute if you wish to retry the execution.

 

Development Mode

Top../images/arwup.gif (846 bytes)

 

Some of the LabVantage components support being run outside of LabVantage while developing and building the driver/report/parsing logic. However this is for development purposes and restricts certain functionality which is only available when executed inside LabVantage from an Attachment Handler. Development mode is just to aid the building of instrument and parsing drivers and not to fully execute functionality outside of LabVantage.

The following components support full operation in development mode:


The following components support limited functionality in development mode:


The following components do not operate outside of LabVantage and in development mode:


To place the Job in development mode you simply need to set the "devtoken" and "lvendpoint" context parameters. These are prompted for when executing the Job from Talend using the LabVantage default context library. The devtoken is an encrypted authentication token obtained from Labvantage following these steps:

To begin, register a new external application in the External Apps area in LabVantage. If you already have an External App called "TalendDeveloper", please review the settings to verify they match what is shown here. You can find more details about this in the Authentication Tokens for External Applications help topic.

 

 

Make sure you set the Status to Active and check the checkbox next to Requires SOAP WebServices. Next, register a new external user in LabVantage.

System Admin > Security > Users... Add

 

 

Add a new Token in the Auth Tokens area in LabVantage.

System Admin > Security > Auth Tokens:

 

 

Set the External App and External User to the ones you set up earlier and change the Token status to active. You now need to copy the Token Value and use this as the "devtoken" in the context parameter in Talend.

The "lvendpoint" is simply the URL to the LabVantage application including the server, port and application name. For for example: https://server_host:8080/labvantage

Certain processing might not be allowed according to your Security Policy. Navigate to System Admin > Configuration > Policies... and edit the SecurityPolicy. Enable WebServices, Full Action Processing, and Unregistered SQL. Please note that these settings are not recommended for a production system and should be used for development only. This can be turned off again once development is complete and the attachment handler is running inside LabVantage. Also, it is recommended once the Talend Job is built and deployed to LabVantage that you also deactivate the authentication token used for development mode.

Now when you execute a Job directly from Talend, the LabVantage components will use SOAP based webservices to communicate with Talend and allow functionality such as querying Labvantage.

Another feature of Development Mode is the ability for the framework to fetch additional context variables that would normally be available to you when running the Job inside LabVantage. This includes set up variables for the Attachment Handler and meta data for the Data Capture. To enable this simply provide the context parameter "attachmenthandlerid" when executing the Job inside Talend Studio. The Development Mode framework will then fetch the set up variables from the provided attachment handler and populate the context parameters (logging out the variables). You can then also pass an "sdcid" and "keyid1" context parameters which would be the Data Capture details (sdc being LV_DataCapture" and the keyid1 being the data capture id). When provided it additionally retrieves any operation level overrides to the set up variables as well as any meta data on the data capture and populates the context parameters (again logging out the parameters). This then means you can use the context parameters in your driver development.

 

Standard Component Documentation

Top../images/arwup.gif (846 bytes)

LabVantage supplies several components that are designed specifically for building Jobs that can insert information into LabVantage using the API. You have these components if you imported a workspace or the individual components in the first section of this document Creating a LabVantage Workspace

 

tLabVantageAction

 

 

This component is used for executing a single action in LabVantage. The action is executed directly in the LIMS and all at the end once the Job has finished working. Internally an action block is constructed from any call to the Action component and that action block is then executed as the Job returns to LabVantage. This entire block is therefore in the same transaction. However by checking the "Synchronous Execution" checkbox in the component properties it will force the action to be executed immediately in its own transaction in the run of the Job. This latter option is useful if you need to run an action to use its output to drive the properties of another component in the Job.

 

tLabVantageAction Basic Settings

 

tLabVantageAction Development Mode Settings

Works in limited use outside of LabVantage. When run in Development mode from Talend OS it will only allow synchronous actions to be processed.

 

tLabVantageQueryInput

 

 

This is a very simple component that will simply execute a query into the LabVantage database using the SQL provided. Please note that this bypasses any SDI and Departmental security and therefore it is recommended to use the tLabVantageSDIInput instead if you wish to execute queries on SDI's. Also note that because all our components are executed inside LabVantage automation service, that you do not need to relax your security policy for unregistered SQL.

 

tLabVantageQueryInput Basic Settings

 

tLabVantageQueryInput Development Mode Settings

Works fully outside of LabVantage. When run in Development mode from Talend OS it will use SOAP webservices to query the LabVantage system.

 

tLabVantageSDIInput

 

 

This is a component that will execute and SDI Request for LabVantage SDI's using our full security layers.

 

tLabVantageSDIInput Basic Settings

 

tLabVantageQueryInput Development Mode Settings

Works fully outside of LabVantage. When run in Development mode from Talend OS it will use SOAP webservices to query the LabVantage system.

 

tLabVantageFileOutput

 

 

This component will return a file back to LabVantage. When run as an Attachment Handler this will have the effect of adding another Attachment to the Data Capture. When run from a report this will have the result of returning the file as the report content. To use this component any Talend components in the Job creating files need to output the file to the defined temporary file path which is provided by the Global Map property "FILEPATH". For instance if you add a tLabvantageFileOutput to your Job and it is named "tLabVantageFileOutput_1" then you will have a global map variable "tLabVantageFileOutput_1_FILEPATH". Also if you have more than one file to output you will need multiple tLabvantageFileOutput components.

 

tLabVantageFileOutput Basic Settings

 

tLabVantageFileOutput Advaned Settings

 

tLabVantageFileOutput Development Mode Settings

Works fully outside of LabVantage. However no files would be written back to LabVantage.

 

tLabVantageLinkSDI

 

 

This component will return back a set of links to create the primary (Data Capture) to the provided SDI's (e..g Sample). This means you can link any processed data to the data capture.

 

tLabVantageLinkSDI Basic Settings

 

 

tLabVantageLinkSDI Development Mode Settings

Does not work outside of LabVantage. No linking would occur back in LabVantage.

 

tLabVantageMetaData

 

 

This component allows you to return key value pair attributes back to LabVantage which are then added to the Attachment Handler primary (e.g. Data Capture). As through context you are passed meta data through metadata_xxx (where xxx is the attribute id) this is a way of passing data that's not LIMS data between Jobs or Attachment Handlers or just a way of storing non-LIMS data which can still be searched and queried.

 

tLabVantageMetaData Basic Settings

 

tLabVantageMetaData Development Mode Settings

Does not work outside of LabVantage. No meta data would be created in LabVantage

 

tLabVantageDocumentParser

 

 

Talend does not provide a way to parse and process PDF or Post Script files. Therefore, LabVantage provides this component which allows you to input a PDF or PS document and extract tabular data. The document parser does not rely on LabVantage internal functionality. This component can be run outside of LabVantage and from a standard Job.

 

tLabVantageDocumentParser Basic Settings

 

tLabVantageDocumentParser Example Use Case

Here is an example PDF which we require to extract the second table with the forca [ibf] column:

 

We want the columns from a table containing the [IBF] column and want to end the table collection as soon as we encounter the string "Pagina" which is the end of the page. The number column also produces two columns due to the white space so we want to merge them together. Finally we want to output the columns in the order they are found as Number, Force, Result and Date. The settings of the component are shown above.

 

Example Output:

 

We then map the Number, Date and Result to a new XLS file:

 

tLabVantageDocumentParser Development Mode Settings

Works fully outside of LabVantage.

 

tLabVantageResultGrid

 

 

The result grid component is the SDMS component that allows SDI's and Data to easily be created and updated in the LIMS with minimal effort. It automatically fills in missing information such as missing replicate ids and allows through property driven settings to define how the data should be imported and saved. The functionality of this component wraps the Result Grid API.

 

tLabVantageResultGrid Basic Settings

 

 

tLabVantageResultGrid Advanced Settings

 

tLabVantageResultGrid Development Mode Settings

Does not work outside of LabVantage. No meta data would be created in LabVantage.

 

LIMS CI Migrations

Top../images/arwup.gif (846 bytes)

 

SDMS Also provides a set of components that are aimed to support migration of Talend Jobs written for LIMS CI to new SDMS Jobs. As such, some components are deprecated and do not perform any function other than allowing the Job to compile without error. These components show a yellow warning triangle on their icon in the lower right corner. Other components still function but its recommended to migrate them to the newer SDMS components. These components are shown with a green warning triangle in the bottom right corner of the icon.

Please refer to this document on how to migrate from LIMS CI to SDMS: LIMS CI Component Migration

tLabVantageConnection - deprecated

This component used to be used by LIMS CI to create a connection object to LabVantage as LIMS CI used to execute the Job outside of the Labvantage environment and use Webservices to communicate with LabVantage. In the new SDMS this is not required and hence is deprecated. The Job would still compile and build with this component inside it, but it will not do anything. The recommendation is to remove it when rebuilding the job.

tLabVantageCloseConnection - deprecated

Like the connection component this is no longer required and is deprecated functionality. The job would still compile and build with this component inside it, but it will not do anything. The recommendation is to remove it when rebuilding the job.

LIMSCI/tLabVantageLIMSCIFileInput - deprecated

This LIMS CI component managed import of CSV and Excel data into Talend. However in the later editions of Talend this functionality is provided with core Talend functionality which also provides nice graphical parsers. Therefore this component will no longer function in the Job and should be swapped for the tFileInputDelimited component.

LIMSCI/tLabVantageLIMSCIPostData - deprecated

This LIMS CI component managed the export of data from the instrument result parsing into a LIMS table ready for input into the system. The SDMS does no longer use this mechanism or staging area and instead directly executes the functionality in the LIMS with full traceability and the ability to examine the changes and re-execute them. Hence this component is no longer supported and should be migrated to the new tLabVantageResultGrid component which aims to do the same functionality and more.

tLabVantageExec - functioning

This LIMS CI Component will still continue to function in the SDMS built jobs what with more limited functionality than the new tLabVantageAction component. You should be able to do a direct migration from the tLabVantageExec to the new tLabVantageAction and make benefit of the more powerful support as well as the auto linking of created SDI's to the data capture.

Component Basic Settings:

Development Mode Settings - Partially works outside of LabVantage. Only inline synchronous actions will be executed using SOAP based webservices.

tLabVantageInput - functioning

This LIMS CI Component will still continue to function in the SDMS built jobs what with more limited functionality than the new tLabVantageQueryInput or tLabVantageSDIInput components. You should be able to do a direct migration from the tLabVantageInput to the new tLabVantageQueryInput and make benefit of the more powerful support.

Component Basic Settings:

Development Mode Settings - Works fully outside of LabVantage. Queries will be executed using SOAP based webservices.