Skip to main content
POST
Python

Authorizations

x-api-key
string
header
required

API key for authentication

Body

application/json
name
string
required

Name of the web crawl data source

Maximum string length: 200
start_url
string
required

Start URL of the web crawl

Maximum string length: 2000
max_depth
integer
required

Maximum crawl depth

Required range: 1 <= x <= 10
max_num_files
integer
required

Maximum number of files to crawl

Required range: 1 <= x <= 100000
path_filters
string[]

Path filters for crawling. The total number of characters across all elements in the array must be 2000 or fewer.

Maximum array length: 2000
content_pattern
string[]

Content patterns for filtering. The total number of characters across all elements in the array must be 2000 or fewer.

Maximum array length: 2000
html_only
boolean
default:false

When true, only HTML files will be downloaded

use_browser
boolean
default:false

Whether to use a headless browser for crawling

no_canonical_check
boolean
default:false

Whether to disable duplicate filtering by canonical URL

file_extensions
string[]
Maximum array length: 2000
Maximum string length: 10
rrule
string

Recurrence rule (RFC 5545 RRULE)

metadata
object

(reserved for future use) Additional metadata for the web crawl data source

load_more_enabled
boolean

Whether to click load-more controls. Defaults to use_browser. Enabling requires use_browser=true.

load_more_selectors
string[]

Ordered CSS selectors, validated before saving. Empty or omitted uses built-in controls.

load_more_max_clicks
integer

Maximum click attempts per page. Defaults to 10.

Required range: 1 <= x <= 100

Response

Successfully created web crawl data source

id
string
required

Web crawl data source ID

name
string
required

Name of the web crawl ingestion setting

start_url
string
required

Start URL of the web crawl

status
enum<string>
required

Job status

Available options:
unknown,
queued,
not_started,
managed,
starting,
started,
success,
failure,
canceling,
canceled,
deleting,
delete_job_failure
ingestion_setting_id
string

Web crawl ingestion setting ID

creation_time
integer<int64>

Creation time (Unix timestamp in seconds)

start_time
integer<int64>

Job start time (Unix timestamp in seconds)

end_time
integer<int64>

Job end time (Unix timestamp in seconds)

error
object | null
metadata
object | null

(reserved for future use) Additional metadata for the web crawl data source