zgy 8d861fabfe fix(web-crawler): remove Playwright rendering andoptimize the robots.txt entry-failure message (#3040) ...

* refactor(web-crawler): remove Playwright rendering, keep static SPA shells

Drop the Playwright-based fallback rendering path so SPA pages are stored
as their static shell HTML instead of being rendered headlessly. SPA
shells now surface the <noscript> notice (e.g. "You need to enable
JavaScript to run this app.") rather than producing an empty document,
and robots.txt-blocked entry pages get a human-readable error message.

- Delete playwright_renderer.py, render_heuristics.py and their tests
- Strip fallback_playwright/playwright_timeout config, fallback_rendered
  counter, and render-hint plumbing from the crawler and web importer
- HTMLParser: drop the SPA-empty-pattern stripping and fall back to
  <noscript> text when trafilatura extracts nothing
- Humanize the robots.txt entry-failure message
- Migrate the orphaned _convert_to_raw_url tests to HTTPAccessor (the
  method moved there in an earlier reorg) and remove the stale file

* refactor(web-importer): simplify robots.txt-blocked import message

Replace the verbose robots.txt explanation with a short compliance hint
that omits the URL and points users to local-file import instead.

2026-07-06 18:48:39 +08:00

..

__init__.py

feat: add recursive web crawler based on Scrapy (#2836)

2026-07-03 19:25:10 +08:00

test_html.py

fix(web-crawler): remove Playwright rendering andoptimize the robots.txt entry-failure message (#3040)

2026-07-06 18:48:39 +08:00