spb/ultra-sharp-agent-skills Public

1<!--2Author: Simon-Pierre Boucher3Contact: contact@spboucher.ai4-->56# HTML Recipes78## Contents9- Selecting elements (CSS selectors vs find_all)10- Extract: tables → CSV, links, clean text11- Modify: rewrite links, insert/remove/replace elements12- Sanitize: strip scripts and inline handlers13- Convert HTML → Markdown14- Gotchas (parser differences, encoding, whitespace)1516## Selecting elements (CSS selectors vs find_all)1718```python19from bs4 import BeautifulSoup20soup = BeautifulSoup(open("page.html", encoding="utf-8"), "html.parser")2122soup.select("div.card > h2")            # CSS — best for structural paths23soup.select_one("#main table")          # first match or None24soup.find_all("a", href=True)           # find_all — best for attribute filters25soup.find_all("img", alt=False)         # images missing alt26soup.find("h2", string="Pricing")       # exact text match27```2829`select()` covers most CSS: descendants, `>`, `[attr=val]`, `:nth-of-type()`. It does not run JavaScript-era pseudo-classes like `:visible`.3031## Extract: tables → CSV, links, clean text3233Table to CSV, with span detection:3435```python36import csv3738def table_to_csv(table, out_path):39    if table.find(attrs={"rowspan": True}) or table.find(attrs={"colspan": True}):40        print("warning: table uses rowspan/colspan; columns may misalign")41    rows = [[c.get_text(strip=True) for c in tr.find_all(["td", "th"])]42            for tr in table.find_all("tr")]43    with open(out_path, "w", newline="", encoding="utf-8") as f:44        csv.writer(f).writerows(rows)4546for i, table in enumerate(soup.find_all("table")):47    table_to_csv(table, f"table_{i}.csv")48```4950Links with absolute resolution against a known base:5152```python53from urllib.parse import urljoin54base = "https://example.com/docs/"55links = [(a.get_text(strip=True), urljoin(base, a["href"]))56         for a in soup.find_all("a", href=True)]57```5859Readable text without script/style noise:6061```python62for tag in soup(["script", "style", "noscript"]):63    tag.decompose()64text = soup.get_text(separator="\n", strip=True)65```6667## Modify: rewrite links, insert/remove/replace elements6869```python70# Rewrite links (http → https, or path migration)71for a in soup.find_all("a", href=True):72    if a["href"].startswith("http://example.com"):73        a["href"] = a["href"].replace("http://", "https://", 1)7475# Insert: add a class, append a child, insert a sibling76div = soup.select_one("div.content")77div["class"] = div.get("class", []) + ["highlight"]78new_p = soup.new_tag("p")79new_p.string = "Appended paragraph."80div.append(new_p)81div.insert_after(soup.new_tag("hr"))8283# Remove vs unwrap84soup.select_one("aside.ad").decompose()      # delete element and children85for span in soup.find_all("span", class_="tracking"):86    span.unwrap()                            # keep children, drop the tag8788# Replace89old = soup.select_one("center")90new = soup.new_tag("div", attrs={"style": "text-align:center"})91new.extend(list(old.contents))92old.replace_with(new)9394open("page.html", "w", encoding="utf-8").write(str(soup))95```9697## Sanitize: strip scripts and inline handlers9899For untrusted HTML that will be displayed, remove active content:100101```python102for tag in soup(["script", "iframe", "object", "embed"]):103    tag.decompose()104for tag in soup.find_all(True):105    for attr in list(tag.attrs):106        if attr.lower().startswith("on"):        # onclick, onload, ...107            del tag[attr]108    if tag.get("href", "").lstrip().lower().startswith("javascript:"):109        del tag["href"]110```111112(For production-grade sanitizing use the `bleach` library — this covers the common cases.)113114## Convert HTML → Markdown115116```bash117pandoc page.html -t gfm -o page.md            # brew install pandoc118```119120Or in Python, `pip install markdownify`:121122```python123from markdownify import markdownify124md = markdownify(open("page.html", encoding="utf-8").read(), heading_style="ATX")125```126127## Gotchas (parser differences, encoding, whitespace)128129- **Parsers repair differently.** `html.parser` leaves fragments bare; `lxml` wraps them in `<html><body>`; `html5lib` (`pip install html5lib`) repairs exactly like a browser but is slow. If output gained wrapper tags the input lacked, you switched parsers mid-task.130- **Misnested tags relocate.** A `<table>` with stray `</div>` inside may have its rows moved outside the table in the repaired tree — "the selector finds nothing" often means the repair changed the structure, not that the data is missing. Inspect `soup.prettify()` on a slice.131- **`str(soup)` vs `soup.prettify()`.** `str()` preserves the whitespace as parsed; `prettify()` re-indents every node and inserts newlines *inside text*, which can change rendering (whitespace matters around inline elements). Never prettify an existing file.132- **Encoding detection.** BeautifulSoup guesses from `<meta charset>` and byte patterns when given bytes; given a mis-decoded *string*, it can't fix it. When in doubt, pass `open(path, "rb")`.133- **`.string` vs `.get_text()`.** `.string` is `None` when a tag has multiple children; `.get_text()` always returns the concatenated text. Use `.get_text(strip=True)` for extraction.134- **Attribute multi-values.** `class` comes back as a *list* (`["btn", "large"]`); `id` as a string. Appending a class means list-append, not string concat.135- **Entities are decoded on parse** (`&amp;` → `&`) and re-encoded minimally on output; byte-identical round-trips of untouched regions are not guaranteed, only semantically identical ones — diff renders, not bytes.136- **Comments** are `Comment` nodes, invisible to `get_text()`; find them with `soup.find_all(string=lambda s: isinstance(s, Comment))` (`from bs4 import Comment`).137