Web Site Sync

This feature is currently only available to commercial edition users.

What is Web Site Sync

Web Site Sync uses crawler technology to automatically discover all pages under the same domain from an entry URL, supporting up to 200 sub-pages. For compliance and security reasons, FastGPT only supports crawling static sites, primarily intended for quickly building datasets from documentation sites.

Tip: Most China-based media sites are not supported, including WeChat Official Accounts, CSDN, Zhihu, etc. You can verify whether a site is static by sending a curl request from the terminal:

curl https://doc.fastgpt.cn/guide/getting-started

How to Use

1. Create a New Dataset and Select Web Site Sync

Open the Dataset list, create a new Dataset, and choose Web Site Sync as its type.

2. Click to Configure Site Information

Open the new Dataset and choose the option to configure its site information.

3. Enter the URL and Selector

Enter the site's entry URL and a CSS selector for the content to import. The selector examples below show how to identify the article body.

Click Start Sync and wait for the system to automatically crawl the site content.

Create an App and Bind the Dataset

Create an app, select this Dataset in the app's Dataset configuration, and save the configuration to use the synced content in conversations.

How to Use Selectors

Selectors are based on HTML/CSS/JS. You can use selectors to target specific content to crawl rather than the entire site. Here's how:

Open the Browser DevTools (usually F12, or Right-click > Inspect)

Use the element picker to select the page content you want to import, then inspect its HTML tag, attributes, and classes.

Enter the Element Selector

For a CSS selectors reference, see the MDN CSS Selectors guide.

In the image above, we selected an area corresponding to a div tag with three attributes: data-prismjs-copy, data-prismjs-copy-success, and data-prismjs-copy-error. We only need one, so the selector is: div[data-prismjs-copy]

Besides attribute selectors, class and ID selectors are also common. For example:

The class in the image contains class names (there may be multiple separated by spaces — just pick one). The selector would be: .docs-content

Using Multiple Selectors

In the earlier demo, we used multiple selectors for the FastGPT documentation site, separated by commas.

We want to select content from the two tags shown above, which requires two selectors. The first is: .docs-content .mb-0.d-flex, meaning child elements under the docs-content class that have both the mb-0 and d-flex classes.

The second is .docs-content div[data-prismjs-copy], meaning div elements under the docs-content class that have the data-prismjs-copy attribute.

Separate the two selectors with a comma: .docs-content .mb-0.d-flex, .docs-content div[data-prismjs-copy]