Yosoi Repository Agent Guide

Context & Philosophy

Yosoi is an AI-powered tool that discovers resilient selectors for web scraping. The core philosophy is "Discover once, scrape forever." We use LLMs to analyze HTML structure and find selectors that are robust to layout changes, then validate them to ensure accuracy.

Fail Fast: We do not use fallback heuristics. If AI discovery fails, we fail the process. This ensures we don't return garbage data from unreliable selectors.

Technology Stack & Standards

• Language: Python 3.10+
• Package Manager: uv (Strict requirement. DO NOT use pip/poetry directly).
• Linting/Formatting: ruff
• Testing: pytest
• Type Checking: mypy
• Retry Logic: tenacity (Mandatory for all flaky/network operations)

Retry Logic & Durability

We use tenacity to handle retries.

• DO NOT use time.sleep() in loops.
• DO use granular Retrying context managers or decorators.
• DO use wait_exponential to avoid thundering herds.

Example

from tenacity import Retrying, stop_after_attempt, wait_exponential, retry_if_exception_type

# Preferred pattern: Context Manager
for attempt in Retrying(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, max=10),
    retry=retry_if_exception_type(NetworkError),
    reraise=True
):
    with attempt:
        make_network_call()

Critical Rules

1. Dependency Management: ALWAYS use uv add or uv sync. never install with pip.
2. Running Code: ALWAYS use uv run <command>.
   • Example: uv run yosoi --url ...
   • Example: uv run pytest
3. Code Style: Run uv run ruff check . and uv run ruff format . before finishing a task.
4. Type Safety: Maintain strong typing. Use mypy to verify.
5. Retry Logic: Use tenacity for all retry patterns. Never implement custom retry loops with for or while.

Repository Structure

• yosoi/: The core python package.
• tests/: Integration and unit tests.
• examples/: Usage examples.
• .yosoi/: Local storage for selectors, debug HTML, and logs (gitignored).
  • logs/: Contains run logs in run_YYYYMMDD_HHMMSS.log format.
  • debug_html/: Extracted HTML for debugging.

Logging & Observability

• Local Logs: Every run generates a log file in .yosoi/logs/. These logs contain detailed debug information and full tracebacks.
• Logfire: Used for cloud-based observability if LOGFIRE_TOKEN is set.
• Console: Keeping it minimal and stylish for human eyes.

Interaction Guidelines

When working on this repo, generic python solutions often fail. Always check pyproject.toml for available scripts and configuration.