Overview
One of the possible ways to integrate your data to the smartApps and synchronize it with your systems is using import pipelines. You can upload and maintain data, such as users, jobs, skills, posts, etc.
Before creating a pipeline and associating it with a definite import type, you need to set up a connection.
|
|
| |
 |  | |
| Description | Provides file access and file transfer to smartApps over a reliable data stream. | Sends data via API to a smartApp dropspace. |
| Synchronization | Checks for new files every hour. | Once data is transferred to a dropspace. |
| Pros and cons | Requires to give access to the client infrastructure as well as might require firewall configuration changes and approvals. | Requires to create, schedule, and maintain a script to transfer a file via API. |
To import data of any possible type, you need first fill out a document with this data. It is recommended to create this document based on the template. Depending on the file format (EXCEL, CSV, or XML), the template contains a set of columns or tree elements you need to supply with values to successfully import data to smartApps.
Among those columns and tree elements, you can also see the ones corresponding to the custom fields. These fields are added in Ludi Cloud by the administrator. Once added, they are automatically implemented to the template. As a result, you can add values to the custom fields in addition to the standard ones.
To import data via the SFTP connection you can use one of the supported file formats: EXCEL, CSV, and XML. Whereas for the API connection, you can create a file either in CSV or XML format.
You can download a template per each data type from the Import templates section on the Import pipelines page.
If you use your file for importing data to smartApps via automated pipelines, make sure all the mandatory data is filled in, and they have the correct data format.
Each pipeline setup requires a connection. Thus, creating a connection is a preparation phase of the data import process based on the automated pipelines.
There are two connection types:
- SFTP
- API
To add a new connection, follow the steps listed below:
- On the Setup tab of the Import pipelines page, click +Add connection.
- On the Basic settings stage, first, enter a connection name.
- Select a type of data integration:
- Sync via SFTP. Allows connecting to your file system via SSH and getting data from your server.
To use this connection type for importing data to smartApps, you need to have a definite structure of folders on your server. Per each import type, it is recommended to have a separate folder. Each folder, in its turn, has a file with data to be imported. The following file formats are supported: EXCEL, CSV, and XML. - Upload via API to Dropspace. Allows creating a dropspace where your data is uploaded in the CSV format via API. Once uploaded, your data is continuously synchronized with your database.
To use this connection type for importing data to smartApps, you need to upload a file with data to a dropspace via Postman or via Ludi Cloud. The following file formats are supported: CSV and XML.
- Sync via SFTP. Allows connecting to your file system via SSH and getting data from your server.
- Now that you have selected the type of data integration, click Next to proceed to the Credentials stage.
To navigate back to the previous steps of the add connection process, click the step name shown on the top of the page as breadcrumbs. Navigating away from the current step does not clear data you already filled in in the fields available on the step.
SFTP connection
Let us consider the process for setting up credentials of the Sync via SFTP type.
Firstly, select how to secure your connection:
- Via password: Secure your connection by using a strong password. It should be unique, long, and hard to guess.
- Via SSH key: Public key authentication is an alternative way of identifying yourself to a login server, instead of typing a password. It is more secure and more flexible.
Secondly, specify the credentials for your SFTP server:
- Path to the folder available on the server, where the file from which the data will be further imported is stored.
Use the following format:your.server/folder_storing_the_file_for_data_import
Only one file can be available per directory. A separate connection shall be created for each directory.
If you add more files to the same directory, only the latest file you add will be considered for the import. - Username
Once you have selected to secure your connection via a password, define the one to identifying yourself to a login server.
If you decide using SSH public key authentication, the SSH key is automatically generated. To copy the key, click the Copy to clipboard 
icon.
Once you are done, click Submit. Your connection is now being checked. If the connection is successful, click Continue.
In the Summary step, you can view a summary of the files available on the server.
It is possible to refresh a list of files by clicking a refresh icon.
Once you are done, click Save.
If you select to create a new API key, the API name is automatically generated and shown in API Name. You can edit it if needed.
If you decide to use an available API key, in the Selected API drop-down list, select the API key you need among the API keys created on the API keys page or previously on the current step.
Once you are done, click Save to proceed.
On the Summary step, you get all the access credentials of the new or selected API key.
To copy a value, click the 
icon within a corresponding text box.
To complete the process of adding a connection and save the changes, click Finish.
Now that you have added the connection that meets all your needs, it is shown in the list under Setup.
You can test a connection you have previously added. To this end, click the 
icon. Once the connection is tested, you can see its Success status.
To edit a connection, click the three dots button within that connection entry, and then click Edit.
For both connection types, you can edit a connection name.
Once you edit an SFTP connection, you can also download a file available on the server. To this end, click a download icon within that file entry in the Files on server section.
It is also possible to update a list of files to show the ones currently available on the server. To this end, click a refresh icon. If more than one file is available on the server in the specified directory, only the latest uploaded file is used for importing. The rest of the files, if any, are just shown in the list.
To delete a connection, click the three dots button within that connection entry, and then click Delete. You can also delete a connection from the edit connection page. To this end, first, click the three dots button, and then click Delete.
If a connection of any type is used for the pipeline, it is not possible to delete it.
Once you have created an API connection, you can upload a file to a dropspace via API. There is a great variety of tools for using APIs. We recommend to add a file by using the POST method with the help of Postman. It utilizes a graphical user interface, which simplifies the process of obtaining different types of responses.
To add a file to a dropspace via API, follow the steps listed below:
- Launch Postman.
- Create a new request. To start, click New.
- In the Create new dialog that appears, click HTTP Request.
- Select the POST method, and then enter a request URL. You can copy the URL from the Dropspace access field when creating or editing a connection of the API type.
- Switch to the Body tab and select form-data.
- Under KEY, select File, and then type file in lower case.
- Under VALUE, click Select Files, and select a file to upload. The supported file formats are CSV and XML.
- Click Send.
Make sure you get true as the uploaded value as well as the 200 OK response code. It means that the HTTP request was successful, and the file was uploaded via API.
To successfully upload a file to the dropspace via API using Postman, check the API key settings on the API keys tab in Ludi Cloud. Make sure that the Dropspace API for import pipelines checkbox is selected under Data integration of the Available APIs section.
Now that you have added connections, you can start adding pipelines.
Using pipelines, you can import different types of data, such as users, jobs, skills, etc.
Be aware that it is possible to import users only via one pipeline, and that pipeline can use either SFTP or API connection. Whereas several pipelines can be created for importing skills and jobs. With that, each pipeline can use a different connection type. If these are connections via API, then either a different or the same API key can be used for each connection.
If the connection you have previously added is used for a pipeline, it cannot be used for another pipeline.
To add a new pipeline for importing data of any possible type, follow the steps listed below:
1. On the Import pipelines page, switch to the Pipeline tab.
2. Click +Add pipeline.
3. Define the following basic settings:
-
- Import type: data you want to import (users, jobs, skills, etc.).
- Connection: connections you have previously added (SFTP or API).
- Pipeline name: a descriptive name for the pipeline you are currently adding.
4. Click Next to proceed.
To navigate back to the previous steps of the add pipeline process, click the step name shown on the top of the page as breadcrumbs. Navigating away from the current step does not clear data you already filled in in the fields available on the step.
5. If you create a pipeline based on the API connection and you have not uploaded a file to our server via Postman yet, you can now upload the file.
To this end, drag your file to the Drag & Drop your files here area or click Upload a file to search for the file on your computer. Once the file is uploaded, click Continue to proceed to the Mapping step.
For the pipeline based on the SFTP connection, the file shall be already available on the server, which is why the file upload step is skipped, and you are moved directly to the Mapping step.
The file upload step is skipped as well if you create a pipeline based on the API connection and you have already uploaded a file via Postman.
6. On the Mapping step, first, you can check if the file, from which the data is to be imported, is available. It is shown under File on server.
To fetch new mapping fields, you can upload a new file. To this end, click the upload icon. Be aware that the maximum file size is 2 KB. This file is not used for importing data but for mapping columns only.
To download the currently available file and save it locally, click the download icon.
7. Map columns from your file with smartApp system attributes.
If you assign a column from your file for synchronization, the user cannot then change values in this column anymore.
Depending on the type of data you want to import, attributes are organized in different groups.
In the first column, you can see smartApp system attributes. Next to each system attribute, select a column from your file to map it to the system attribute. Once you map the column from your file, its status is changed to Assigned.
For an attribute, you can turn on the possibility to add new entries within an attribute you need. To this end, select the Add new entries checkbox. Once enabled, new entries from your file can be automatically added to the list for this attribute.
You can also define a custom date format per each mapped field presenting date. It can help you work successfully with your own date formats.
For users added to Ludi Cloud via import pipeline, the following fields will be disabled in Ludi Cloud as well as on the details tab of the user’s profile page in smartPeople:
- First name
- Last name
- Username
It is not possible to edit them after import.
When importing jobs, you can manually select a job type for all job records available in the import file or assign a column from the import file which values will be automatically applied as job types.
For manual job selection, under User jobs, turn on the Manual job selection toggle, and then select a job type in the Type drop-down box that becomes available. As a result, the selected job type will be set to all job records imported via this pipeline.
Once the Manual job selection toggle is turned off, select a column available in your import file and assign it to the Type pipeline attribute. As a result, the values of the assigned column will be automatically applied as job types while importing.
When importing skills, you can select multiple columns and map them to one pipeline attribute, so you can adapt your company import file structure to the smartApp structure. To this end, follow the steps listed below:
- Click
next to the column you want to map to the User skill text attribute. - In the dialog that appears, select a method to process multiple columns:
- Combine columns. Values from multiple columns you add will be combined into a single column.
- Combine multiple skills with proficiency. Multiple skills from one cell will be combined with a proficiency level from another cell.
- Once you select the Combine columns method, you have now the possibility to add up to 5 column drop-down boxes, and in each drop-down box, select a definite column from your import file which value can be used to denote a skill. To add a column drop-down box, click the plus sign, and then select a file column.
If you decide to combine multiple skills with proficiency, select one file column which values can be used to denote a skill and another column—to denote a proficiency level.
To undo the action and come back to the initial column, click
.
As a result, values of the selected columns will be combined for skill extraction while importing.
Import trainings
When importing trainings, you need select a training library as well as select a language. Under Language, you can select a language manually for all training records you want to import. To this end, turn on the Manual language selection toggle, and then select a column to link it to the language attribute. When mapping columns for the pipeline to import trainings, you can also define how to extract skills from the training data. Turn the Skill by skill IDs toggle on to retrieved skills by skill IDs. Once turned off, skills will be extracted based on their names and descriptions.
8. Once you map all the mandatory attributes, click Next to proceed to the Options step.
9. Set up automatic synchronization. Depending on the data type for which you create a pipeline, the settings you can define on the Options step differ.
To update data on a regular basis, first, select a method and a schedule for synchronization. You can choose between the following options:
- Synchronization of all users. Allows updating existing users, adding new ones, and deleting the others.
- Synchronization of signed-in users. Allows synchronizing user data upon users’ sign-in to Ludi Cloud, and thus updating existing user data.
Once you have selected how to synchronize data, you can define what to do if a user does not exist in the file. You have the following options:
- Deactivate user
- Delete user
- Skip user
Import jobs
Here you can set up automatic synchronization. You can choose between the following options:
- One-time import on first sign-in. Allows importing jobs once.
- Continuous import. Allows updating jobs continuously.
Per each possible option, you can see a list of actions and conditions provided for it.
Here you can set up automatic synchronization. You can choose between the following options:
- One-time import on first sign-in. Allows importing skills once.
- Continuous import. Allows updating skills continuously.
Per each possible option, you can see a list of actions and conditions provided for it.
For both job and task post pipeline, the following synchronization option is possible:
- Replace. Allows replacing the existing job or task posts.
A list of actions and conditions for the option is provided on this step.
You can also select metadata to match a post by. To this end, select the value you need in the Compare post database by drop-down list:
- Post ID
- Post title and location
- Post link
For a pipeline to import trainings, the following synchronization option is possible:
- Update. Allows updating the existing trainings with new information from your import file.
A list of actions and conditions for the option is provided on this step.
Once you have selected how to synchronize data, you can define an action to apply to a training if it exists in smartLibrary but is missing in the current import file:
- Delete but keep it in existing profiles
- Delete it everywhere
You can also define whether to publish a library after finishing the import process. To this end, turn the Publish library after import finished toggle on.
10. Once you are done, click Save.
As a result, the pipeline is created. Now you can start importing data from your resource. Data is retrieved either from files available on your server or files uploaded to dropspaces. The import is triggered every hour via a pipeline based on the SFTP connection. Whereas data is retrieved and immediately transferred to smartApps once a file with the data is uploaded to a dropspace via the API connection.
If mandatory fields are missing values in the file, these fields are not processed during the import process and will be declined.
Depending on the data type you import, the import log record becomes available in the Imports table of the smartApp where the data you import via pipeline becomes available. For example, you can view the error details of the user import on the Import error page in Ludi Cloud. The page is accessible from the Imports table on the User import page (Users > Import users).
In Postman, you can also get a response for each import. The response presents the following information:
- import ID
- start and end (only for finished) time
- import status (open, in progress, error, or finished)
- import summary
- error details, if any
Now that you have added pipelines, you can manage them as needed.
You can test a pipeline you have previously added. To this end, click the 
icon. Once the pipeline is tested, you can see its Success status.
To edit a pipeline, click the three dots button within that pipeline entry, and then click Edit. Afterwards, manage the settings as needed.
Once you edit a pipeline based on the API connection, you can also upload a file to fetch the latest fields for mapping. To this end, click an upload icon within that file entry in the Files on server section on the Mapping step. Be aware that the maximum file size is 2 KB. This file is not used for importing data but for mapping columns only.
Editing pipelines based on any connection type, you can download a file to your local computer by clicking a download icon within that file entry in the Files on server section.
To delete a pipeline, click the three dots button within that pipeline entry, and then click Delete. You can also delete a pipeline from the edit pipeline page. To this end, first, click the three dots button, and then click Delete.
To deactivate a pipeline, turn off the toggle button within that pipeline entry.
Next step
Well done! Learn more about identity & access management.
