Set stewardship.curation_progress.ingestion.batch_size Parameter

Customer Managed Applies to customer-managed instances of Alation

The Curation Progress report on the Stewardship Dashboard is computed for all data objects in batches.

Note

In releases before V R6 (5.10.x) the computation job is triggered when the user turns on the Stewardship Dashboard feature (using the alation.feature_flags.enable_stewardship_dashboard flag in alation_conf). Starting from V R6, all Analytical Stewardship features are enabled by default.

The curation progress computation batch size can be regulated with the  stewardship.curation_progress.ingestion.batch_size parameter. The default is set to 1,000.

Users can try decreasing  the value of the stewardship.curation_progress.ingestion.batch_size, parameter if their instance is low on resources and they observe the following:

  • The Analytical Stewardship Dashboard-related tasks are stuck and the system performance deteriorates

  • Analytical Stewardship Dashboard tasks block or delay the progress of other tasks.

On larger instances without a limitation on resources, if you observe the slowness  for Stewardship Dashboard-related Celery tasks, you can increase the value of stewardship.curation_progress.ingestion.batch_size  parameter to 10,000.

To configure stewardship.curation_progress.ingestion.batch_size using the alation_conf file:

  1. Enter the Alation shell:

    sudo /etc/init.d/alation shell

  2. To view the current value of the stewardship.curation_progress.ingestion.batch_size, do:

    alation_conf stewardship.curation_progress.ingestion.batch_size

  3. To set the value of stewardship.curation_progress.ingestion.batch_size, run the following command substituting <value in number of tasks> with the actual value:

    alation_conf stewardship.curation_progress.ingestion.batch_size -s <value in number of tasks>

  4. Restart the Celery component:

    alation_supervisor restart celery:*