kfp_toolbox.pipelines module#

kfp_toolbox.pipelines.load_pipeline_from_file(filepath: str | PathLike) → Pipeline[source]#

Load a pipeline object from the pipeline file.

Load a Pipeline object from a pre-compiled file that represents the pipeline.

Parameters:

filepath (Union[str, os.PathLike]) – The path of the pre-compiled file that represents the pipeline.

Raises:

ValueError – If the filepath file has an invalid schema.

Returns:

An object that represents the pipeline.

Return type:

Pipeline

Deprecated since version 0.6.0: Use pipeline_parser.parse_pipeline_package() instead.

kfp_toolbox.pipelines.submit_pipeline_job(pipeline_file: str | PathLike, endpoint: str | None = None, iap_client_id: str | None = None, api_namespace: str = 'kubeflow', other_client_id: str | None = None, other_client_secret: str | None = None, arguments: Mapping[str, Any] | None = None, run_name: str | None = None, experiment_name: str | None = None, namespace: str | None = None, pipeline_root: str | None = None, enable_caching: bool | None = None, service_account: str | None = None, encryption_spec_key_name: str | None = None, labels: Mapping[str, str] | None = None, project: str | None = None, location: str | None = None, network: str | None = None)[source]#

Submit a pipeline job.

Submit a pipeline job to the appropriate environment. If an endpoint is specified, it is considered an instance of Kubeflow Pipelines. Otherwise, attempt to submit to Vertex AI Pipelines.

Parameters:

  • pipeline_file (Union[str, os.PathLike]) – Path of the pipeline package file.

  • endpoint (Optional[str], optional) – Endpoint of the KFP API service to connect. Used only for Kubeflow Pipelines. Defaults to None.

  • iap_client_id (Optional[str], optional) – The client ID used by Identity-Aware Proxy. Used only for Kubeflow Pipelines. Defaults to None.

  • api_namespace (str, optional) – Kubernetes namespace to connect to the KFP API. Used only for Kubeflow Pipelines. Defaults to “kubeflow”.

  • other_client_id (Optional[str], optional) – The client ID used to obtain the auth codes and refresh tokens. Used only for Kubeflow Pipelines. Defaults to None.

  • other_client_secret (Optional[str], optional) – The client secret used to obtain the auth codes and refresh tokens. Used only for Kubeflow Pipelines. Defaults to None.

  • arguments (Optional[Mapping[str, Any]], optional) – Arguments to the pipeline function provided as a dict. Defaults to None.

  • run_name (Optional[str], optional) – Name of the run to be shown in the UI. Defaults to None.

  • experiment_name (Optional[str], optional) – Name of the experiment to add the run to. Defaults to None.

  • namespace (Optional[str], optional) – Kubernetes namespace where the pipeline runs are created. Used only for Kubeflow Pipelines. Defaults to None.

  • pipeline_root (Optional[str], optional) – The root path of the pipeline outputs. Defaults to None.

  • enable_caching (Optional[bool], optional) – Whether or not to enable caching for the run. Defaults to None.

  • service_account (Optional[str], optional) – Specifies which Kubernetes service account this run uses. Defaults to None.

  • encryption_spec_key_name (Optional[str], optional) – The Cloud KMS resource identifier of the customer managed encryption key used to protect the job. Used only for Vertex AI Pipelines. Defaults to None.

  • labels (Optional[Mapping[str, str]], optional) – The user defined metadata to organize PipelineJob. Used only for Vertex AI Pipelines. Defaults to None.

  • project (Optional[str], optional) – The project that you want to run this PipelineJob in. Used only for Vertex AI Pipelines. Defaults to None.

  • location (Optional[str], optional) – Location to create PipelineJob. Used only for Vertex AI Pipelines. Defaults to None.

  • network (Optional[str], optional) – The full name of the Compute Engine network to which the job should be peered. Used only for Vertex AI Pipelines. Defaults to None.

Deprecated since version 0.6.0: Use pipeline_jobs.submit_pipeline_job() instead.

kfp_toolbox.pipelines.timestamp(format: str = '%Y%m%d%H%M%S', prefix: str = None, suffix: str = None, postfix: str = None, separator: str = '-', tz_offset: int = '0')#

Generate a time string in a specified format.

It adds a prefix or suffix to the time string generated from the format, along with a separator. tz_offset can be specified to represent a time zone other than UTC.

This function is used as a component. Since caching is disabled by default, it is executed each time at the current time unless caching is enabled in the pipeline.

Parameters:

  • format (str, optional) – Time format string. Strings that cannot be interpreted as a format string are output as is. Defaults to “%Y%m%d%H%M%S”.

  • prefix (Optional[str], optional) – A prefix string to be prepended to the format string. Defaults to None.

  • suffix (Optional[str], optional) – A suffix string to be added after the format string. Defaults to None.

  • postfix (Optional[str], optional) – DEPRECATED: Use suffix instead. A postfix string to be added after the format string. Defaults to None.

  • separator (str, optional) – Separator to be added between the format string, prefix and suffix. Defaults to “-“.

  • tz_offset (int, optional) – Time zone offset specified in hours. Defaults to 0.

Returns:

The current time string represented by a format string. A prefix and suffix string separated by a separator is appended.

Return type:

str

Deprecated since version 0.5.0: postfix argument is deprecated. Use suffix instead.