OCF Connector for Impala on CDP: Overview

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Customer Managed Applies to customer-managed instances of Alation

Compatible with Alation version 2022.4 or later

The OCF connector for Impala on Cloudera Data Platform (CDP) was developed by Alation and earlier certified only for on-premise CDP.

From version 2.0.0, this connector also supports Public Cloud CDP Distribution and Private Cloud CDP Distribution.

To download the Impala on CDP OCF connector package, go to the Alation Connector Hub available from the Customer Portal. Go to Customer Portal > Connectors > Alation Connector Hub. Only Alation users with access to the Customer Portal can access the Alation Connector Hub. If you don’t have access to the Customer Portal, contact Alation Support.

This connector should be used to catalog Impala on CDP as a data source in Alation. The connector catalogs Impala objects such as schemas, tables, views, and columns. It enables end-users to search and find the Impala metadata from the Alation user interface. After metadata is extracted, it is represented in the Alation catalog as a hierarchy of catalog pages under the parent data source. Alation users can leverage the full catalog functionality to search for and find the extracted metadata, curate the corresponding catalog pages, create documentation about the data source, and exchange information about it.

Team

The following administrators are required to install this connector:

  • Alation administrator

    • Installs the connector

    • Creates and configures an Impala data source in the catalog

    • Performs initial extraction and prepares the data source for Alation users.

  • Impala administrator

    • Creates a service account and grants it the required privileges

    • Provides the JDBC URI

    • Provides access to the metastore server to extract metadata

    • Provides access to the query log history directory on HDFS or Amazon S3.

Scope

The table below describes which metadata objects are extracted by the connector and which catalog functionality is supported.

Feature

Scope

Availability

Authentication

Basic authentication

Authentication with a service account created on the database that uses a username and password

Yes

LDAP

Authentication with a database service account that is an LDAP account in an organization’s network

Yes

SSL

Database connection over SSL

Yes

Kerberos

Support for Kerberos authentication

Yes

Keytab

Support for Kerberos with keytabs

Yes

SSO

SSO authentication with an identity provider application

No

Metadata Extraction (MDE)

Default MDE

Extraction of metadata based on Metastore API

Yes

Custom query-based MDE

Extraction of metadata based on extraction queries provided by a user

No

Extracted metadata objects

Data source

Data source object in Alation that is parent to extracted metadata

Yes

Schemas

List of schemas

Yes

Tables

List of tables

Yes

Columns

List of columns

Yes

Column data types

Column data types

Yes

Views

List of views

Yes

Source comments

Source comments

Yes

Primary keys

Primary key information for extracted tables

No

Foreign keys

Foreign key information for extracted tables

No

Functions

Function metadata

No

Function definitions

Function definition metadata

No

Sampling and Profiling

(Sampling and profiling is not available for complex data types)

Table sampling

Retrieval of data samples from extracted tables

Yes

Column sampling

Retrieval of data samples from extracted columns

Yes

Deep column profiling

Profiling of columns with the calculation of value distribution stats

Yes

Dynamic profiling

Ability for individual users to connect with their own database accounts to retrieve table and column samples and profiles

Yes

Custom query-based table sampling

Ability to use custom queries for sampling specific tables

Yes

Custom query-based column sampling

Ability to use custom queries for profiling specific columns

Yes

Query Log Ingestion (QLI)

API-based QLI

Ingestion of query history based on Impala audit file that contains query history data

Yes

Query-based QLI

Ingestion of query history based on a custom query history extraction query

No

JOINs and filters

Calculation of JOIN and filter information based on ingested query history

Yes

Predicates

Ability to parse predicates in ingested queries

Yes

Lineage

Automatic lineage generation

Auto-calculation of lineage based on query history ingested from QLI, MDE, and Compose queries

Yes

Column-level lineage

Calculation of lineage at column level

No

Compose

Customer-managed (on-prem) Alation instances

Compose on on-prem Alation instances

Yes

Alation Cloud Service instances, connection without Agent

Compose for data sources connected without Alation Agent

Yes

Alation Cloud Service instances, connection via Agent

Compose for data sources connected through Alation Agent

No

Kerberos authentication with Compose

Authentication in Compose through Kerberos

No

SSO authentication in Compose

Authentication in Compose with SSO credentials

No

Limitations

  • For Impala OCF Connector version 2.0.0, Compose is supported only with Alation version 2023.3.3.2 or later.

  • Compose doesn’t work with Kerberos authentication.

  • Sampling and profiling is not available for complex data types.

  • Data upload into columns with complex data types is not supported.