# Apify Website Content Crawler

[Apify](https://apify.com/) is a web scraping and data extraction platform that provides an app store with more than a thousand ready-made cloud tools called Actors.

The [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor can deeply crawl websites, clean their HTML by removing a cookies modals, footers, or navigation, and then transform the HTML into Markdown. This Markdown can then be stored in a vector database for semantic search or Retrieval-Augmented Generation (RAG).

<figure><img src="/files/ydRD4BOcHDfp20T8ZJbK" alt="" width="266"><figcaption><p>Apify Website Content Crawler Node</p></figcaption></figure>

## Crawl Entire Website

1. *(Optional)* Connect [**Text Splitter**](/readme/chatflows/langchain/text-splitters.md).
2. Connect Apify API (create a new credential with your [Apify API token](https://my.apify.com/account#/integrations)).
3. Input one or more URLs (separated by commas) where the crawler will start, e.g `https://innovativesol.com/`.
4. Select the crawler type. Refer to [Website Content Crawler documentation for more information](https://apify.com/apify/website-content-crawler/input-schema#crawlerType).
5. *(Optional)* Specify additional parameters such as maximum crawling depth and the maximum number of pages to crawl.

## Output

Loads website content as a Document.

## Resources

* [Website Content Crawler](https://apify.com/apify/website-content-crawler)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://tailwindsdocs.innovativesol.com/readme/chatflows/langchain/document-loaders/apify-website-content-crawler.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.