Better alignment with previous behavior

Oct 13, 2025 by pynkocean.com 20 views

crawl_rules.policy needs additional option(s)

Problem Description

The current behavior of crawl_rules.policy in v0.22 has been observed to have the following behaviors:

Allow: The page is crawled and indexed.
Deny: The page is not crawled and not indexed.

However, for sites that have index pages or site listings, this forces you to index those pages or at least send them to Elasticsearch and remove them using ingestion pipelines, which is not very elegant. This is particularly evident in cases where you want to crawl a site's directory pages but not index them.

Proposed Solution

To address this issue, a change to the crawl_rules.policy behavior is proposed as follows:

Index: The page is crawled and indexed. This would be a re-enumeration of the current allow behavior.
Discard: The page is not crawled and not indexed. This would be a re-enumeration of the current deny behavior.
Crawl: The page is crawled but not indexed. This would be a new behavior that would allow the page to be crawled but not force its indexing.

The addition of the crawl option would enable the crawler to bounce through pages and use their links for crawling, but not force the indexing of the page. This seems to be more in line with the deny behavior of the previous Elastic Crawler.

Alternatives

Other alternatives to consider are:

Deny behaving as per the previous crawler: This would mean that the page is crawled but not indexed, which could be a suitable solution for some use cases.
Post-crawl filtering: This would involve filtering out pages that are not intended to be indexed after they have been crawled. This could be a more flexible solution, but it may also introduce additional complexity.

Additional Context

This issue came to light when crawling a site with a large, multi-page directory. The intention was to filter out the directory pages, but a significant drop in the number of crawled pages was observed. This highlighted the need for a more nuanced approach to crawling and indexing.

Benefits of the Proposed Solution

The proposed solution offers several benefits, including:

Improved flexibility: The addition of the crawl option would provide more flexibility in terms of how pages are crawled and indexed.
Better alignment with previous behavior: The crawl option would be more in line with the behavior of the previous Elastic Crawler, which would make it easier to migrate to the new behavior.
Reduced complexity: The proposed solution would reduce the complexity of the crawling and indexing process, making it easier to manage and maintain.

Implementation Plan

To implement the proposed solution, the following steps would be taken:

Update the crawl_rules.policy behavior: The crawl_rules.policy behavior would be updated to include the new crawl option.
Test the new behavior: The new behavior would be thoroughly tested to ensure that it works as expected.
Deploy the updated behavior: The updated behavior would be deployed to production, and the crawling and indexing process would be monitored to ensure that it is working correctly.

Conclusion

The proposed solution would provide a more nuanced approach to crawling and indexing, allowing for more flexibility and better alignment with the previous behavior. The benefits of the proposed solution include improved flexibility, better alignment with previous behavior, and reduced complexity. The implementation plan would involve updating the crawl_rules.policy behavior, testing the new behavior, and deploying the updated behavior to production.
crawl_rules.policy needs additional option(s) - Q&A

Q: What is the current behavior of `crawl_rules.policy` in v0.22?

A: The current behavior of crawl_rules.policy in v0.22 has the following behaviors:

Allow: The page is crawled and indexed.
Deny: The page is not crawled and not indexed.

Q: Why is the current behavior not suitable for all use cases?

A: The current behavior is not suitable for all use cases because it forces you to index pages that you may not want to index. For example, if you have a site with a large, multi-page directory, you may want to crawl the directory pages but not index them. The current behavior does not allow for this level of control.

Q: What is the proposed solution to address this issue?

A: The proposed solution is to add a new option to the crawl_rules.policy behavior called crawl. This option would allow the page to be crawled but not indexed.

Q: How would the `crawl` option work?

A: The crawl option would enable the crawler to bounce through pages and use their links for crawling, but not force the indexing of the page. This would be more in line with the deny behavior of the previous Elastic Crawler.

Q: What are the benefits of the proposed solution?

A: The benefits of the proposed solution include:

Improved flexibility: The addition of the crawl option would provide more flexibility in terms of how pages are crawled and indexed.
Better alignment with previous behavior: The crawl option would be more in line with the behavior of the previous Elastic Crawler, which would make it easier to migrate to the new behavior.
Reduced complexity: The proposed solution would reduce the complexity of the crawling and indexing process, making it easier to manage and maintain.

Q: How would the implementation plan work?

A: The implementation plan would involve:

Updating the crawl_rules.policy behavior: The crawl_rules.policy behavior would be updated to include the new crawl option.
Testing the new behavior: The new behavior would be thoroughly tested to ensure that it works as expected.
Deploying the updated behavior: The updated behavior would be deployed to production, and the crawling and indexing process would be monitored to ensure that it is working correctly.

Q: What are the alternatives to the proposed solution?

A: The alternatives to the proposed solution are:

Deny behaving as per the previous crawler: This would mean that the page is crawled but not indexed, which could be a suitable solution for some use cases.
Post-crawl filtering: This would involve filtering out pages that are not intended to be indexed after they have been crawled. This could be a more flexible solution, but it may also introduce additional complexity.

Q: Why is the proposed solution better than the alternatives?

A: The proposed solution is better than the alternatives because it provides a more nuanced approach to crawling and indexing. The crawl option would allow for more flexibility in terms of how pages are crawled and indexed, and it would be more in line with the behavior of the previous Elastic Crawler.

Q: What are the next steps in implementing the proposed solution?

A: The next steps in implementing the proposed solution would be to update the crawl_rules.policy behavior, test the new behavior, and deploy the updated behavior to production.