Third-Party Dataset Development

There are many document libraries available online, such as Lark, Yuque, and others. Different FastGPT users may use different document libraries. FastGPT has built-in support for Lark and Yuque, but if you need to integrate other document libraries, follow this guide.

Unified API Specification

To provide a unified interface for different document libraries, FastGPT defines a standard API specification with 4 endpoints. See the API File Library endpoints.

All built-in document libraries are extensions of the standard API File Library. Refer to the code in FastGPT/packages/service/core/dataset/apiDataset/yuqueDataset/api.ts to build extensions for other document libraries. You need to implement 4 endpoints:

1. Get file list
2. Get file content / file link
3. Get original file preview URL
4. Get file detail information

Building a Third-Party File Library

This walkthrough uses a Lark Knowledge Dataset (FeishuKnowledgeDataset) as the example integration. The screenshots show earlier source layouts and authentication variants. Match the server type, form fields, and request adapter to the version you are extending; the steps below identify the code responsibilities to update.

1. Add Third-Party Document Library Parameters

First, go to FastGPT/packages/global/core/dataset/apiDataset/type.ts in the FastGPT project and add the third-party document library server type. Design the fields based on your needs. For example, the Yuque Dataset requires userId and token for authentication.

export type YuqueServer = {
  userId: string;
  token?: string;
  basePath?: string;
};

Add the new server configuration as an optional property on ApiDatasetServerType, alongside entries such as yuqueServer?: YuqueServer. The Lark example uses feishuKnowledgeServer?: FeishuKnowledgeServer.

2. Create the Hook File

Each third-party document library uses a Hook pattern to maintain a set of API endpoints. The Hook contains 5 functions to implement.

• Create a folder for your document library under FastGPT\packages\service\core\dataset\apiDataset\, then create an api.ts file inside it
• In api.ts, define the following 5 functions:
  • listFiles: Get the file list
  • getFileContent: Get file content / file link
  • getFileDetail: Get file detail information
  • getFilePreviewUrl: Get the original file preview URL
  • getFileId: Get the original file's real ID

3. Add the Dataset Type

In FastGPT/packages/global/core/dataset/type.ts, import your new Dataset type.

4. Add Dataset Data Retrieval

In FastGPT\packages\global\core\dataset\apiDataset\utils.ts, extend filterApiDatasetServerPublicData: read your new server property from apiDatasetServer and add its public representation to the returned object. Follow the existing handling of credentials when preparing data for the client.

5. Add Dataset Invocation Method

In FastGPT\packages\service\core\dataset\apiDataset\index.ts, import the new request adapter and select it inside getApiDatasetRequest when the corresponding server configuration is present. In this example, the feishuKnowledgeServer branch returns useFeishuKnowledgeDatasetRequest({ feishuKnowledgeServer }).

Adding the Frontend

Add your i18n translations in FastGPT\packages\web\i18n\zh-CN\dataset.json, FastGPT\packages\web\i18n\en\dataset.json, and FastGPT\packages\web\i18n\zh-Hant\dataset.json. Add a Dataset name, a configuration-form title, and a description. The example keys are feishu_knowledge_dataset, feishu_knowledge_dataset_config, and feishu_knowledge_dataset_desc; provide values for each locale.

In FastGPT\packages\service\support\user/audit\util.ts, add a branch to getI18nDatasetType for the new DatasetTypeEnum value. Return its translated Dataset name so audit records display the correct label.

Add your Dataset icons under FastGPT\packages\web\components\common\Icon\icons\core\dataset\. You need two icons: Outline (monochrome) and Color (colored), as shown below.

In FastGPT\packages\web\components\common\Icon\constants.ts, register your icons. The import path points to where the icons are stored.

In FastGPT\packages\global\core\dataset\constants.ts, add your Dataset type to both DatasetTypeEnum and ApiDatasetTypeMap.

In FastGPT\projects\app\src\pages\dataset\list\index.tsx, add the following. This file handles the menu that appears when clicking the "New" button on the Dataset list page. Add a menu item with its icon, translated label and description, and an onClick handler that calls onSelectDatasetType with the new DatasetTypeEnum value.

In FastGPT\projects\app\src\pageComponents\dataset\detail\Info\index.tsx, render a configuration section for the new Dataset type. Show its translated configuration title and account or app identifier. Wire the edit icon to setEditedAPIDataset, passing the Dataset ID and its server configuration.

Adding the Configuration Form

In FastGPT\projects\app\src\pageComponents\dataset\ApiDatasetForm.tsx, add the following. This file handles the field input form when creating a Dataset. Import the new server type, add it to the form data type, and render its credential fields when the corresponding Dataset type is selected. Register each required field with validation. The pictured form uses App ID and App Secret; token-based adapters should expose the token fields defined by their server type.

For adapters with root directory support, include renderBaseUrlSelector() and renderDirectoryModal() to display the current path and folder picker. Directory details come from the adapter's getFileDetail implementation. The user clicks Select, chooses a folder, and clicks Confirm; the selected path then appears in the form.

{renderBaseUrlSelector()} // Renders the `Base URL` field
{renderDirectoryModal()} // The `Select Root Directory` modal that appears when clicking `Select` (see image)

If the Dataset needs root directory support, also add the following in the ApiDatasetForm file.

1. Parse the Dataset Type

Watch the form's apiDatasetServer value, then read your adapter configuration from it. For this example, use apiDatasetServer?.feishuKnowledgeServer.

2. Add Root Directory Selection Logic and parentId Assignment

Include your adapter's basePath in the parentId calculation. Add a case to canSelectBaseUrl that enables folder selection once the required credentials are present, and include those fields in the memoization dependencies. The token-based Lark example checks feishuKnowledgeServer?.user_access_token.

3. Add Field Validation and Assignment Logic

Before loading a path for a new Dataset, validate the required credentials again. Pass datasetId, parentId, and apiDatasetServer to getApiDatasetPaths. In onSelectBaseUrl, normalize the root selection to an empty string and save the selected value to the adapter's basePath field. The example uses setValue('apiDatasetServer.feishuKnowledgeServer.basePath', value).

Tips

After creating the Dataset, we recommend running a full test of all Dataset features to check for issues. If you encounter problems that aren't covered in this documentation, it's likely that some configuration was missed. Do a global search for YuqueServer and yuqueServer to verify that your type has been added everywhere it's needed.