适用于:
”的基于 API 的管道创作
”,SaaS 连接器
”数据库连接器
默认情况下,Lakeflow Connect 托管引入管道创建的目标表的名称与源表相同。 但是,可以选择重命名引入的表。 例如,如果将对象引入同一架构中的两个目标表中,则必须重命名其中一个目标表来区分它们(不支持重复项)。 如果重命名管道中的表,它将变为仅限 API 的管道,并且不能再在 UI 中编辑管道。
示例:Google Analytics
Databricks 资产捆绑包
resources:
pipelines:
pipeline_ga4:
name: <pipeline-name>
catalog: <target-catalog> # Location of the pipeline event log
schema: <target-schema> # Location of the pipeline event log
ingestion_definition:
connection_name: <connection>
objects:
- table:
source_url: <project-id>
source_schema: <property-name>
destination_catalog: <target-catalog>
destination_schema: <target-schema>
destination_table: <custom-target-table-name> # Table rename
Databricks 笔记本
Google Analytics:
pipeline_spec = """
{
"name": "<pipeline>",
"ingestion_definition": {
"connection_name": "<connection>",
"objects": [
{
"table": {
"source_catalog": "<project-id>",
"source_schema": "<property-name>",
"source_table": "<source-table>",
"destination_catalog": "<target-catalog>",
"destination_schema": "<target-schema>",
"destination_table": "<custom-target-table-name>",
}
}
]
}
}
"""
Databricks 命令行界面 (CLI)
{
"resources": {
"pipelines": {
"pipeline_ga4": {
"name": "<pipeline>",
"catalog": "<target-catalog>",
"schema": "<target-schema>",
"ingestion_definition": {
"connection_name": "<connection>",
"objects": [
{
"table": {
"source_url": "<project-id>",
"source_schema": "<property-name>",
"destination_catalog": "<destination-catalog>",
"destination_schema": "<destination-schema>",
"destination_table": "<custom-destination-table-name>"
}
}
]
}
}
}
}
}
示例:Salesforce
Databricks 资产捆绑包
resources:
pipelines:
pipeline_sfdc:
name: <pipeline-name>
catalog: <target-catalog> # Location of the pipeline event log
schema: <target-schema> # Location of the pipeline event log
ingestion_definition:
connection_name: <connection>
objects:
- table:
source_schema: <source-schema>
source_table: <source-table>
destination_catalog: <target-catalog>
destination_schema: <target-schema>
destination_table: <custom-target-table-name> # Table rename
Databricks 笔记本
pipeline_spec = """
{
"name": "<pipeline>",
"ingestion_definition": {
"connection_name": "<connection>",
"objects": [
{
"table": {
"source_catalog": "<source-catalog>",
"source_schema": "<source-schema>",
"source_table": "<source-table>",
"destination_catalog": "<target-catalog>",
"destination_schema": "<target-schema>",
"destination_table": "<custom-target-table-name>",
}
}
]
}
}
"""
Databricks 命令行界面 (CLI)
{
"resources": {
"pipelines": {
"pipeline_sfdc": {
"name": "<pipeline>",
"catalog": "<target-catalog>",
"schema": "<target-schema>",
"ingestion_definition": {
"connection_name": "<connection>",
"objects": [
{
"table": {
"source_schema": "<source-schema>",
"source_table": "<source-table>",
"destination_catalog": "<destination-catalog>",
"destination_schema": "<destination-schema>",
"destination_table": "<custom-destination-table-name>"
}
}
]
}
}
}
}
}
示例:Workday
Databricks 资产捆绑包
resources:
pipelines:
pipeline_workday:
name: <pipeline>
catalog: <target-catalog> # Location of the pipeline event log
schema: <target-schema> # Location of the pipeline event log
ingestion_definition:
connection_name: <connection>
objects:
- report:
source_url: <report-url>
destination_catalog: <target-catalog>
destination_schema: <target-schema>
destination_table: <custom-target-table-name> # Table rename
Databricks 笔记本
pipeline_spec = """
{
"name": "<pipeline>",
"ingestion_definition": {
"connection_name": "<connection>",
"objects": [
{
"report": {
"source_url": "<report-url>",
"destination_catalog": "<target-catalog>",
"destination_schema": "<target-schema>",
"destination_table": "<custom-target-table-name>",
}
}
]
}
}
"""
Databricks 命令行界面 (CLI)
{
"resources": {
"pipelines": {
"pipeline_workday": {
"name": "<pipeline>",
"catalog": "<target-catalog>",
"schema": "<target-schema>",
"ingestion_definition": {
"connection_name": "<connection>",
"objects": [
{
"report": {
"source_url": "<report-url>",
"destination_catalog": "<destination-catalog>",
"destination_schema": "<destination-schema>",
"destination_table": "<custom-destination-table-name>"
}
}
]
}
}
}
}
}