Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/lakehouse/catalogs/maxcompute-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
}
---

[MaxCompute](https://help.aliyun.com/zh/maxcompute/) is an enterprise-level SaaS (Software as a Service) cloud data warehouse on Alibaba Cloud. Through the open storage SDK provided by MaxCompute, Doris can retrieve MaxCompute table information and perform queries and writes.

Check notice on line 9 in docs/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//help.aliyun.com/zh/maxcompute/. Owner%3A @apache/doris-website-maintainers

## Applicable Scenarios

Expand All @@ -17,9 +17,9 @@

## Usage Notes

1. Starting from version 2.1.7, MaxCompute Catalog is developed based on the [Open Storage SDK](https://help.aliyun.com/zh/maxcompute/user-guide/overview-1). Prior to this version, it was developed based on the Tunnel API.

Check notice on line 20 in docs/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//help.aliyun.com/zh/maxcompute/user-guide/overview-1. Owner%3A @apache/doris-website-maintainers

2. There are certain limitations when using the Open Storage SDK. Please refer to the `Usage Limitations` section in this [document](https://help.aliyun.com/zh/maxcompute/user-guide/overview-1).

Check notice on line 22 in docs/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//help.aliyun.com/zh/maxcompute/user-guide/overview-1. Owner%3A @apache/doris-website-maintainers

3. Before Doris version 3.1.3, a Project in MaxCompute corresponds to a Database in Doris. In version 3.1.3, you can introduce the MaxCompute schema hierarchy through the `mc.enable.namespace.schema` parameter.

Expand Down Expand Up @@ -72,7 +72,7 @@

| Property Name | Description | Supported Doris Version |
| --- | --- | --- |
| `mc.default.project` | The name of the MaxCompute project to access. You can create and manage projects in the [MaxCompute Project List](https://maxcompute.console.aliyun.com/cn-beijing/project-list). | |

Check notice on line 75 in docs/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//maxcompute.console.aliyun.com/cn-beijing/project-list. Owner%3A @apache/doris-website-maintainers
| `mc.region` | The region where MaxCompute is activated. You can find the corresponding Region from the Endpoint. | Before 2.1.7 (exclusive) |
| `mc.endpoint` | The region where MaxCompute is activated. Please refer to the "How to Obtain Endpoint and Quota" section below for configuration. | 2.1.7 (inclusive) and later |

Expand Down Expand Up @@ -396,6 +396,8 @@
- Does not support creating clustered tables, transactional tables, Delta Tables, or external tables.
:::

When creating or dropping databases or tables, Doris invokes the corresponding MaxCompute API to perform the operation. After the operation returns, metadata changes on the MaxCompute side need a short time before they are fully visible to subsequent operations.

This feature is only available when the `mc.enable.namespace.schema` property is set to `true`.

### Creating and Dropping Databases
Expand Down Expand Up @@ -474,13 +476,13 @@

1. If using exclusive resource groups for data transfer service

Please refer to the "2. Authorization" section in the "Using Exclusive Data Service Resource Groups" chapter in this [document](https://help.aliyun.com/zh/maxcompute/user-guide/purchase-and-use-exclusive-resource-groups-for-dts) to enable the corresponding permissions. In the "Quota Management" list, view and copy the corresponding `QuotaName`, and specify `"mc.quota" = "QuotaName"`. At this point, you can choose either VPC or public network to access MaxCompute, but VPC has guaranteed bandwidth while public network bandwidth resources are limited.

Check notice on line 479 in docs/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//help.aliyun.com/zh/maxcompute/user-guide/purchase-and-use-exclusive-resource-groups-for-dts. Owner%3A @apache/doris-website-maintainers

2. If using pay-as-you-go

Please refer to the "Using Open Storage (Pay-as-you-go)" section in this [document](https://help.aliyun.com/zh/maxcompute/user-guide/overview-1) to enable the Open Storage (Storage API) switch and grant permissions to the user corresponding to the AK and SK. In this case, `mc.quota` is the default value `pay-as-you-go`, and you do not need to specify this value additionally. With pay-as-you-go, you can only use VPC to access MaxCompute and cannot access via public network. Only prepaid users can access MaxCompute via public network.

Check notice on line 483 in docs/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//help.aliyun.com/zh/maxcompute/user-guide/overview-1. Owner%3A @apache/doris-website-maintainers

3. Configure `mc.endpoint` according to the "Region Endpoint Reference Table" in the [Alibaba Cloud Endpoints documentation](https://help.aliyun.com/zh/maxcompute/user-guide/endpoints)

Check notice on line 485 in docs/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//help.aliyun.com/zh/maxcompute/user-guide/endpoints. Owner%3A @apache/doris-website-maintainers

Users accessing via VPC need to configure `mc.endpoint` according to the "VPC Network Endpoint" column in the "Regional Endpoint Reference Table (Alibaba Cloud VPC Network Connection Method)" table. Users accessing via public network can choose from the "Classic Network Endpoint" column in the "Regional Endpoint Reference Table (Alibaba Cloud Classic Network Connection Method)" table, or the "External Network Endpoint" column in the "Regional Endpoint Reference Table (External Network Connection Method)" table to configure `mc.endpoint`.

Expand All @@ -503,11 +505,11 @@

Users can also specify `mc.odps_endpoint` and `mc.tunnel_endpoint` individually to customize the service address, which is suitable for privately deployed MaxCompute environments.

For configuring MaxCompute Endpoint and Tunnel Endpoint, please refer to [Endpoints for Different Regions and Network Connection Methods](https://help.aliyun.com/zh/maxcompute/user-guide/endpoints).

Check notice on line 508 in docs/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//help.aliyun.com/zh/maxcompute/user-guide/endpoints. Owner%3A @apache/doris-website-maintainers

### Resource Usage Control

Users can adjust the [table-level request concurrency](https://help.aliyun.com/zh/maxcompute/user-guide/data-transfer-service-quota-manage?spm=a2c4g.11186623.help-menu-search-27797.d_2) by adjusting the two Session Variables `parallel_pipeline_task_num` and `num_scanner_threads` to control resource consumption in the data transfer service. The corresponding concurrency equals `max(parallel_pipeline_task_num * be num * num_scanner_threads)`.

Check notice on line 512 in docs/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//help.aliyun.com/zh/maxcompute/user-guide/data-transfer-service-quota-manage?spm=a2c4g.11186623.help-menu-search-27797.d_2. Owner%3A @apache/doris-website-maintainers

Note:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
}
---

[MaxCompute](https://help.aliyun.com/zh/maxcompute/) 是阿里云上的企业级 SaaS(Software as a Service)模式云数据仓库。通过 MaxCompute 提供的开放存储 SDK,Doris 可以获取 MaxCompute 的表信息,并进行查询和写入操作。

Check notice on line 9 in i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//help.aliyun.com/zh/maxcompute/. Owner%3A @apache/doris-website-maintainers

## 适用场景

Expand Down Expand Up @@ -399,6 +399,8 @@
- 不支持创建聚簇表、事务表、Delta Table 和外部表。
:::

创建或删除库表时,Doris 会调用 MaxCompute API 执行对应操作。操作返回后,MaxCompute 侧元数据变更需要短暂时间才能对后续操作完全可见。

该功能仅在 `mc.enable.namespace.schema` 属性为 `true` 时可用。

### 创建和删除库
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -399,6 +399,8 @@ CREATE TABLE mc_tbl AS SELECT * FROM other_table;
- 不支持创建聚簇表、事务表、Delta Table 和外部表。
:::

创建或删除库表时,Doris 会调用 MaxCompute API 执行对应操作。操作返回后,MaxCompute 侧元数据变更需要短暂时间才能对后续操作完全可见。

该功能仅在 `mc.enable.namespace.schema` 属性为 `true` 时可用。

### 创建和删除库
Expand Down
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
---

Check warning on line 1 in ja-source/docusaurus-plugin-content-docs/current/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-cjk-spacing

Chinese text should contain spaces around adjacent English words or numbers. Owner%3A @apache/doris-website-maintainers
{
"title": "MaxCompute カタログ",
"language": "ja",
"description": "Apache Doris MaxCompute カタログは、Alibaba Cloud MaxComputeデータに対する連合クエリをサポートし、データ移行なしにクロスソース分析のためのデータ統合と書き戻しを可能にします。"
}
---

Check warning on line 7 in ja-source/docusaurus-plugin-content-docs/current/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-cjk-spacing

Chinese text should contain spaces around adjacent English words or numbers. Owner%3A @apache/doris-website-maintainers
[MaxCompute](https://help.aliyun.com/zh/maxcompute/)は、Alibaba Cloud上のエンタープライズレベルSaaS(Software as a Service)クラウドデータウェアハウスです。MaxComputeが提供するオープンストレージSDKを通じて、DorisはMaxComputeテーブル情報を取得し、クエリや書き込みを実行できます。

Check warning on line 8 in ja-source/docusaurus-plugin-content-docs/current/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-cjk-spacing

Chinese text should contain spaces around adjacent English words or numbers. Owner%3A @apache/doris-website-maintainers

## 適用シナリオ

| シナリオ | 説明 |

Check warning on line 12 in ja-source/docusaurus-plugin-content-docs/current/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-cjk-spacing

Chinese text should contain spaces around adjacent English words or numbers. Owner%3A @apache/doris-website-maintainers
| ---- | ------------------------------------------------------ |
| データ統合 | MaxComputeデータを読み取り、Doris内部テーブルに書き込みます。 |
| データライトバック | INSERTコマンドを使用してMaxCompute tableにデータを書き込みます。(バージョン4.1.0以降でサポート) |

Check warning on line 16 in ja-source/docusaurus-plugin-content-docs/current/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-cjk-spacing

Chinese text should contain spaces around adjacent English words or numbers. Owner%3A @apache/doris-website-maintainers
## 使用上の注意

1. バージョン2.1.7以降、MaxCompute カタログは[Open Storage SDK](https://help.aliyun.com/zh/maxcompute/user-guide/overview-1)に基づいて開発されています。このバージョン以前は、Tunnel APIに基づいて開発されていました。
Expand Down Expand Up @@ -43,15 +43,15 @@
| `mc.secret_key` | SecretKey。[Alibaba Cloud Console](https://ram.console.aliyun.com/manage/ak)で作成と管理ができます。 | |
| `mc.region` | MaxComputeが有効化されているリージョン。EndpointからそれぞれのRegionを見つけることができます。 | Before 2.1.7 (exclusive) |
| `mc.endpoint` | MaxComputeが有効化されているリージョン。設定については以下の「EndpointとQuotaを取得する方法」のセクションを参照してください。 | 2.1.7 (inclusive) and later |

Check warning on line 46 in ja-source/docusaurus-plugin-content-docs/current/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-cjk-spacing

Chinese text should contain spaces around adjacent English words or numbers. Owner%3A @apache/doris-website-maintainers
* `{McOptionalProperties}`

Check warning on line 48 in ja-source/docusaurus-plugin-content-docs/current/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-cjk-spacing

Chinese text should contain spaces around adjacent English words or numbers. Owner%3A @apache/doris-website-maintainers
| Property Name | Default Value | Description | Supported Doris Version |

Check warning on line 49 in ja-source/docusaurus-plugin-content-docs/current/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-cjk-spacing

Chinese text should contain spaces around adjacent English words or numbers. Owner%3A @apache/doris-website-maintainers
| -------------------------- | ------------- | -------------------------------------------------------------------------- | ------------ |
| `mc.tunnel_endpoint` | None | 付録の「カスタムサービスアドレス」を参照。 | Before 2.1.7 (exclusive) |
| `mc.odps_endpoint` | None | 付録の「カスタムサービスアドレス」を参照。 | Before 2.1.7 (exclusive) |
| `mc.quota` | `pay-as-you-go` | Quota名。設定については以下の「EndpointとQuotaを取得する方法」のセクションを参照してください。 | 2.1.7 (inclusive) and later |
| `mc.split_strategy` | `byte_size` | splitのパーティショニング方法を設定。バイトサイズによるパーティション`byte_size`または行数による`row_count`に設定可能。 | 2.1.7 (inclusive) and later |

Check warning on line 54 in ja-source/docusaurus-plugin-content-docs/current/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-cjk-spacing

Chinese text should contain spaces around adjacent English words or numbers. Owner%3A @apache/doris-website-maintainers
| `mc.split_byte_size` | `268435456` | 各splitが読み取るファイルサイズ(バイト単位)。デフォルトは256MB。`"mc.split_strategy" = "byte_size"`の場合のみ有効。 | 2.1.7 (inclusive) and later |
| `mc.split_row_count` | `1048576` | 各splitが読み取る行数。`"mc.split_strategy" = "row_count"`の場合のみ有効。 | 2.1.7 (inclusive) and later |
| `mc.split_cross_partition` | `false` | 生成されたsplitがパーティションをまたぐかどうか。 | 2.1.8 (inclusive) and later |
Expand All @@ -67,7 +67,7 @@
CommonPropertiesセクションは共通プロパティを記入するために使用されます。[Catalog Overview](../catalog-overview.md)の「Common Properties」セクションを参照してください。

### サポートされるMaxComputeバージョン

Check warning on line 70 in ja-source/docusaurus-plugin-content-docs/current/lakehouse/catalogs/maxcompute-catalog.md

View workflow job for this annotation

GitHub Actions / Build Check

markdown-cjk-spacing

Chinese text should contain spaces around adjacent English words or numbers. Owner%3A @apache/doris-website-maintainers
パブリッククラウド版のMaxComputeのみサポートしています。プライベートクラウド版のサポートについては、Dorisコミュニティサポートにお問い合わせください。

### サポートされるMaxComputeテーブル
Expand Down Expand Up @@ -229,6 +229,8 @@
- クラスター化されたテーブル、トランザクショナルテーブル、Delta Tables、外部テーブルの作成はサポートしていません。
:::

データベースやテーブルを作成または削除する際、Dorisは対応するMaxCompute APIを呼び出して操作を実行します。操作が返された後、MaxCompute側のメタデータ変更が後続の操作から完全に見えるようになるまで、短い時間が必要です。

> この機能は`mc.enable.namespace.schema`プロパティが`true`に設定されている場合のみ利用可能です。

### データベースの作成と削除
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,8 @@ CREATE TABLE mc_tbl AS SELECT * FROM other_table;
- クラスター化テーブル、トランザクションテーブル、Delta Tables、または外部テーブルの作成はサポートしていません。
:::

データベースやテーブルを作成または削除する際、Dorisは対応するMaxCompute APIを呼び出して操作を実行します。操作が返された後、MaxCompute側のメタデータ変更が後続の操作から完全に見えるようになるまで、短い時間が必要です。

> この機能は`mc.enable.namespace.schema`プロパティが`true`に設定されている場合のみ利用可能です。

### データベースの作成と削除
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,8 @@ Starting from version 4.1.0, Doris supports creating and dropping MaxCompute dat
- Does not support creating clustered tables, transactional tables, Delta Tables, or external tables.
:::

When creating or dropping databases or tables, Doris invokes the corresponding MaxCompute API to perform the operation. After the operation returns, metadata changes on the MaxCompute side need a short time before they are fully visible to subsequent operations.

This feature is only available when the `mc.enable.namespace.schema` property is set to `true`.

### Creating and Dropping Databases
Expand Down
Loading