Skip to main content
Version: Next

SageMaker

Certified

Important Capabilities

CapabilityStatusNotes
Detect Deleted EntitiesOptionally enabled via stateful_ingestion.remove_stale_metadata
Table-Level LineageEnabled by default

This plugin extracts the following:

  • Feature groups
  • Models, jobs, and lineage between the two (e.g. when jobs output a model or a model is used by a job)

CLI based Ingestion

Install the Plugin

The sagemaker source works out of the box with acryl-datahub.

Starter Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
type: sagemaker
config:
# Coordinates
aws_region: "my-aws-region"

sink:
# sink configs

Config Details

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
aws_access_key_id
string
AWS access key ID. Can be auto-detected, see the AWS boto3 docs for details.
aws_advanced_config
object
Advanced AWS configuration options. These are passed directly to botocore.config.Config.
aws_endpoint_url
string
The AWS service endpoint. This is normally constructed automatically, but can be overridden here.
aws_profile
string
Named AWS profile to use. Only used if access key / secret are unset. If not set the default will be used
aws_proxy
map(str,string)
aws_region
string
AWS region code.
aws_secret_access_key
string
AWS secret access key. Can be auto-detected, see the AWS boto3 docs for details.
aws_session_token
string
AWS session token. Can be auto-detected, see the AWS boto3 docs for details.
extract_feature_groups
boolean
Whether to extract feature groups.
Default: True
extract_jobs
One of string, boolean
Whether to extract AutoML jobs.
Default: True
extract_models
boolean
Whether to extract models.
Default: True
read_timeout
number
The timeout for reading from the connection (in seconds).
Default: 60
env
string
The environment that all assets produced by this connector belong to
Default: PROD
aws_role
One of string, array
AWS roles to assume. If using the string format, the role ARN can be specified directly. If using the object format, the role can be specified in the RoleArn field and additional available arguments are documented at https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html?highlight=assume_role#STS.Client.assume_role
aws_role.union
One of string, AwsAssumeRoleConfig
aws_role.union.RoleArn 
string
ARN of the role to assume.
aws_role.union.ExternalId
string
External ID to use when assuming the role.
database_pattern
AllowDenyPattern
regex patterns for databases to filter in ingestion.
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
database_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
database_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
database_pattern.allow.string
string
database_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
database_pattern.deny.string
string
table_pattern
AllowDenyPattern
regex patterns for tables to filter in ingestion.
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
table_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
table_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
table_pattern.allow.string
string
table_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
table_pattern.deny.string
string
stateful_ingestion
StatefulStaleMetadataRemovalConfig
Base specialized config for Stateful Ingestion with stale metadata removal capability.
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Code Coordinates

  • Class Name: datahub.ingestion.source.aws.sagemaker.SagemakerSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for SageMaker, feel free to ping us on our Slack.