OCF Connector for Google BigQuery: Overview¶
Alation Cloud Service Applies to Alation Cloud Service instances of Alation
Customer Managed Applies to customer-managed instances of Alation
The OCF connector for Google BigQuery was developed by Alation and is available as a Zip file that can be uploaded and installed in the Alation application. The connector is compiled together with the required database driver, so no additional effort is needed to procure and install the driver.
To download the Google BigQuery OCF connector package, go to the Alation Connector Hub available from the Customer Portal. Go to Customer Portal > Connectors > Alation Connector Hub. Only Alation users with access to the Customer Portal can access the Alation Connector Hub. If you don’t have access to the Customer Portal, contact Alation Support.
This connector should be used to catalog Google BigQuery as a data source on Alation customer-managed (on-prem) and Cloud Service instances. It extracts and catalogs such database objects as schemas, tables, views, and columns. The extracted metadata is represented in the data catalog as a hierarchy of catalog pages under a parent data source. Alation users can leverage the full catalog functionality to search and find the metadata, curate the corresponding catalog pages, create documentation about the data source, exchange information about it, and follow data transformations using lineage diagrams.
Team¶
Alation administrator:
Ensures that Alation Connector Manager is available or installs it.
Installs the OCF connector.
Creates and configures a Google BigQuery data source in the data catalog.
Performs initial extraction and prepares the data source for Alation users.
Google Cloud Platform administrator:
Assists in configuring authentication for extraction and Compose.
Provides the JDBC URI.
Scope¶
The table below describes which metadata objects are extracted by the connector and which catalog functionality is supported.
Feature |
Scope |
Availability |
|---|---|---|
Authentication |
||
Authentication with a service account |
Authentication with a service account |
Yes |
Authentication with OAuth and a user account |
Authentication with a user account and an OAuth client |
Yes |
Metadata extraction (MDE) |
||
Default MDE |
Extraction of metadata from Google BigQuery projects. This connector uses Google BigQuery API to retrieve metadata into the Alation catalog |
Yes |
Custom query-based MDE |
Extraction of metadata based on extraction queries provided by a user |
No |
Extracted metadata objects |
||
Data Source |
Data source object in Alation that is parent to extracted metadata |
Yes |
Schemas |
List of schemas |
Yes |
Tables |
List of tables |
Yes |
Columns |
List of columns |
Yes |
Column data types |
Column data types |
Yes |
Views |
List of views Extraction of materialized views is supported |
Yes |
Source comments |
Source comments are extracted for schema, table, view, materialized view, and column objects |
Yes |
Primary keys |
Primary key information for extracted tables |
No |
Foreign keys |
Foreign key information for extracted tables |
No |
Functions |
Function metadata |
No |
Function definitions |
Function definition metadata |
No |
Sampling and profiling |
||
Table sampling |
Retrieval of data samples from extracted tables |
Yes |
Column sampling |
Retrieval of data samples from extracted columns |
Yes |
Deep column profiling |
Profiling of columns with the calculation of value distribution stats |
Yes |
Dynamic profiling |
Ability for individual users to connect with their own database accounts to retrieve table and column samples and profiles |
Yes |
Custom query-based table sampling |
Ability to use custom queries for sampling specific tables |
Yes |
Custom query-based column sampling |
Ability to use custom queries for profiling specific columns |
Yes |
Query log ingestion (QLI) |
||
Table-based QLI |
Ingestion of query history based on a table or view that contains query history data |
Yes |
Query-based QLI |
Ingestion of query history based on a custom query history extraction query |
Yes |
JOINs and filters |
Calculation of JOIN and filter information based on ingested query history |
Yes |
Predicates |
Ability to parse predicates in ingested queries |
Yes |
Lineage |
||
Automatic lineage generation |
Auto-calculation of lineage based on query history ingested from QLI, MDE, and Compose queries |
Yes |
Column-level lineage |
Calculation of lineage data at the column level. Requires the column-level lineage parser add-on for Google BigQuery |
Yes |
Compose |
||
Customer-managed (on-prem) Alation instances |
Compose on on-prem Alation instances |
Yes |
Alation Cloud Service instances |
Depending on your network configuration, you may be using Alation Agent to connect to your data source. Compose with Agent is supported from connector version 1.1.1.4664. |
Yes |
OAuth |
Authentication in Compose using the OAuth protocol |
Yes |