Migrate from Tableau Native Connector to OCF

Customer Managed Applies to customer-managed instances of Alation

Applies from Alation version 2023.3.5 and Tableau OCF Connector 1.7.1

Note

Alation Cloud Service customers can request the Tableau native source migration to Open Connector Framework (OCF) through Alation Support.

Before you start the migration, we recommend you thoroughly review this guide, as it covers essential prerequisites and considerations. We recommend that the migration is performed by a user with the Server Admin role.

Considerations

Migration Scope

Some BI objects are migrated from native BI sources to OCF-based BI sources seamlessly. However, certain information will not be migrated:

• Information Included Into the Migration Scope

• Information Excluded from the Migration Scope

Information Included Into the Migration Scope

The following objects are migrated, including their curation information (or logical metadata).

• BI Folders—Sites, projects, and workbooks

• BI Reports—Sheets and dashboards

• BI DataSources—Published datasources

The following configuration information is migrated:

• Server Connection and Additional Settings on the settings page, such as the host, username, password, domain names, extraction batch sizes, and other information.

Information Excluded from the Migration Scope

The following objects are not migrated seamlessly and will need an additional action to be transferred.

• The curation information for the following objects is not migrated:

  • BI Report Columns—Sheet and Dashboard fields

  • BI DataSource Columns—Datasource fields of published and unpublished datasources

  • BI DataSources—Embedded and unpublished datasources

• The SSL certificate of the Tableau host if SSL was in use and a certificate was uploaded. The certificate will need to be re-uploaded.

  Note

  Ensure you have the certificate file ready for re-uploading after the migration.

Additional Action

After you perform the migration via the user interface, you will also be required to perform some additional steps on the backend of the Alation server to complete the process. The reason behind this requirement is explained below.

There is a known issue with the native Tableau connector where it generates a new BI object every time the name changes in Tableau instead of updating the existing object. It might also lead to hidden curation if those objects were curated. The native connector uses the name attribute as a unique identifier for the following BI objects:

• BI Report Columns

• BI DataSource Columns

• BI DataSources—Unpublished datasources

The OCF connector addresses this issue, as it relies on the Tableau’s Metadata API, which returns unique identifiers for the object types listed above.

Post migration, you will be required to run a curation migration script on the Alation server to ensure that the curation data (logcal metadata) for these three BI object types is transferred to the newly created OCF objects. You will need to contact Alation Support to get access to the curation migration script.

Note

If you don’t have any curation information for the three affected object types (Report Columns, DataSource Columns, and DataSources of the unpublished (embedded) datasource type, you can skip the additional manual step.

Overview of Migration Steps

To migrate your native Tableau source to OCF, follow this workflow:

1. Fulfill the Prerequisites.

2. Perform the migration and run an extraction—Migrate your Tableau Native Source to OCF.

3. Run the curation migration script—Run the Curation Migration Script.

4. Review the curation script log—Reading the Curation Migration Report.

Prerequisites

You need the Server Admin privileges to perform the steps below.

Important

Ensure that your native Tableau BI server source does not have any logical curationdata that’s not migrate-able, currently. Everything that we can migrate is listed in the Supported curation types for Migration. If you found curation items that cannot be migrated, do not perform the migration, contact Support instead.

1. Log in to Alation.

2. Click the gear icon in the top right corner to open the Admin Settings page.

3. In the Server Admin section, click Feature Configuration.

4. Enable the Enable Native Connector Migration to OCF Connector toggle.

   

5. Click Save Changes.

6. Refresh the page.

   

7. Open the settings page of your native Tableau source.

8. Ensure that the Remove Projects that are not captured by the list above checkbox is selected. This is required for all extraction jobs you will run during the migration process.

   • If the checkbox is not visible, toggle the Select Projects to Include or Exclude from Extraction button to reveal it.

   • This must be checked for MDE runs as part of migration. You can uncheck it after the migration process is completed if necessary.

     

9. Ensure that you have a recent successful MDE run and that the Remove Projects that are not captured by the list above checkbox was checked.

   • If you haven’t recently run MDE, then ensure that the Remove Projects checkbox is selected and run MDE to retrieve the latest metadata into Alation.

10. Install the latest versions of the Tableau OCF connector.

    • See Set Up OCF Connector for Tableau.

11. Back up your SSL certificate.

    • If you use SSL for connections between Tableau and Alation, you must download your certificate file from the Tableau Server before the migration process as it will not be migrated.

Curation Type	Description
Post (anchor tags)	Links in an Article where a BI Object has been referenced using report fields or datasource fields or unpublished fields.
PostRevision (anchor tags)	Links in older versions of an Article (revisions).
Flag (anchor tags)	Links in WARN or DEPRECATED flags’ text fields.
TextLog (anchor tags)	Links in RDBMS object descriptions and titles e.g. tables, schemas, columns, etc.
GenericFieldValue (anchor tags)	Links in generic fields like Steward, RTFs (rich text fields), and Description.
CustomFieldValue (anchor tags)	Links in CustomFieldValue text fields, RTFs, etc. This curation type is deprecated but must be updated for backwards compatibility
ValueHistory (anchor tags)	Links to Field Edit History for descriptions, stewards, RTFs, and other objects where you see a small clock icon.
Threads (conversations)	Conversations feature in Alation.
PostMention	These tables DO NOT contain customer data. These are the tables that power the Alation instance that collects customer data.
Mark (star/watch)	These are tied to a specific user and can only be seen by the user that created the Mark in Alation’s Catalog.
Flag (Endorse, Warn, Deprecate)	Data quality flags used in Alation.
Tag	Alation tags to categorize an object.
GenericFieldValue (oid)	These are field values like People, Sets (Experts, Stewards), Object Sets, RichTextFields, DatePickers, and DropDowns.
CustomFieldValue (oid)	This is a deprecated curation type that was replaced by GenericFieldValue. However, we do need to migrate it too for backwards compatibility.
ValueHistory (oid)	Field Edit History on objects like description and history.
Domain (ExcludeMembers)	This is an exclude membership rule and cannot be seen in the UI
Domains (MembershipRule)	This is a membership rule and cannot be seen in the UI
Domains (Member)	This Domain value can be seen on the top right hand side of the Alation catalog

Migrate your Tableau Native Source to OCF

Warning

Migrating from the Tableau native connector to OCF is irreversible. Ensure that all Prerequisites have been met.

To migrate the Tableau source:

1. Open the catalog page of the Tableau source you want to migrate.

2. Select the Migrate button in the top right corner.

   

3. From the drop-down list, select the latest Tableau OCF connector version that you installed.

   

4. Enter the connector name to confirm the action.

   

5. Click Migrate.

6. Verify that the settings page has changed and all previous settings were preserved (except SSL).

Settings Page Differences

Native	OCF

7. If using SSL, re-upload the certificate file.

8. Run the connection test to ensure it’s connecting successfully.

9. Ensure that item 8 of the Prerequisites is done.

10. Run the first MDE on the OCF connector.

11. Once completed, take note of the first_ocf_mde_date in this format: 'dd-mm-yyyy'. E.g. '25-12-2023'.

    • This date represents when the OCF MDE was first triggered and not when it was completed. This is important for accuracy in the curation data migration.

    • Without this value, the connector will consider stale data for migration.

12. The OCF MDE should run without any errors that might indicate missing metadata.

    • Please refer to the connector logs for errors.

Run the Curation Migration Script

Note

Contact Alation Support to get access to the curation migration script.

Script Arguments

• -s—BI Sever rosemeta ID from the source URL.

  • Example: if the BI source URL is https://aaa.alationcloud.com/bi/v2/server/13/, the ID is 13.

• first_ocf_mde_date—Date on which OCF MDE was triggered

  • The format must be 'dd-mm-yyyy' e.g. '16-11-2023'

Running the Script

To run the script:

1. Use SSH to connect to the Alation server.

2. Enter the Alation shell.

   sudo /etc/init.d/alation shell
   

3. Navigate to the one_off_scripts directory and place the curation migration script provided by Alation support.

   cd /opt/alation/django/rosemeta/one_off_scripts/
   # assuming that the script was uploaded into the /tmp folder on the Alation machine
   cp /tmp/copy_tableau_logical_data_from_deleted_objects_to_new_objects.py .
   

4. Change the user to alation.

   sudo su alation
   

5. The curation migration script has two modes:

   • Dry-run—Executes the script without actually making any database writes, only reads.

   • Confirm—Makes permanent changes to the database and is not reversible. It should only be run once.

   Both commands are shown in the example below.

   Note

   The script execution is a database-intensive as well as time-consuming operation based on the volume of data on the BI Server.

   DRY RUN:

   nohup python -m copy_tableau_logical_data_from_deleted_objects_to_new_objects -s <bi_server_id> -first_ocf_mde_date '<date>' &

   CONFIRM:

   nohup python -m copy_tableau_logical_data_from_deleted_objects_to_new_objects -s <bi_server_id> -first_ocf_mde_date '<date>'  --confirm &

   EXAMPLE:

   nohup python -m copy_tableau_logical_data_from_deleted_objects_to_new_objects -s 2 -first_ocf_mde_date '16-11-2023'  --confirm &

6. You can track the progress of the script using its PID (process id). The command above will run the script in the background and output a PID. We can use this PID to check the status.

   ps -p 14496 -o %cpu,%mem,cmd
   
   %CPU %MEM CMD
   98.9  0.1 python -m copy_tableau_logical_data_from_deleted_objects_to_new_objects -s 3 -first_ocf_mde_date 25-10-2023 --confirm
   

7. The curation script migration logs are located in the directory /tmp/tableau_native_curation_to_ocf_migration.log. After the migration completes, review this log file for any URLs of the objects that could not be migrated. An object may not be migrated:

   • Most commonly because Alation could not find its match in the new OCF source, for example when it was renamed or its parent was deleted.

   • If any URLs are printed, they will show the URL and curation that was missed:

     http://localhost:8000//bi/v2/datasource_column/5/ | curation type: {Mark (star/watch)
     

   • If no URLs are found and/or the error keyword is not found in the logs, then you should see the message Completed the curation migration without any lost curation in the log file.

   • The native BI objects that aren’t in the migration scope (see: Information Excluded from the Migration Scope) will get soft-deleted during the Tableau Native connector to OCF migration process. This means they will remain accessible with the URL, but their catalog page will be marked as deleted.

   • When you run the migration script after the native to ocf migration, it will list all objects that were not matched and their URLs. Use the URLs to view the objects and their logical metadata if you have to manually transfer the curation information for these objects to the corresponding pages of the objects under the now migrated OCF BI source

Reading the Curation Migration Report

Example

**** Searching for Native to OCF matches for Data Source Columns

-----------------------------------------------------------------

Data Source Columns - Found a total of: 53 instances of curation

The following are the URLs (count: 1) to the Data Source Columns which we could not automatically match to an OCF object. They need to be reviewed (for validity) and to migrate their curation manually.

http://localhost:8000//bi/v2/datasource_column/5/ | curation type: {'Mark (star/watch)', 'ValueHistory (oid)', 'GenericFieldValue (anchor tags)', 'Tag'}

2024-01-29 10:00:25.025069 —— ========================== Data Source Columns Curation Migration End ==========================

Explanation

• Line 2: Indicates we’re searching for matches between native and OCF BI objects

• Line 5: Indicates that we found 53 instances of curation on native-based datasource_columns objects

• Line 7: Indicates the number of BI objects (and how many BI objects) for which we couldn’t find an OCF match and so cannot copy curation to it (OCF object)

• Line 9: Indicates the URL for the object we could not migrate curation and the type of curation in brackets. Review the URLs and migrate the curation data on them manually.

Troubleshooting

• For Connector settings migration errors, please refer to the migration log file: /opt/alation/site/logs/ypireti.log

• Permissions error: Ensure that you are running the script as the user alation

• Django imports errors or issues: Ensure that you run the script from the directory /opt/alation/django/rosemeta/one_off_scripts/

• For any other issues, please reach out to Alation Support with the following log file attached to the Support ticket: /tmp/tableau_native_curation_to_ocf_migration-debug.log