# HTML Recipes ## Contents - Selecting elements (CSS selectors vs find_all) - Extract: tables → CSV, links, clean text - Modify: rewrite links, insert/remove/replace elements - Sanitize: strip scripts and inline handlers - Convert HTML → Markdown - Gotchas (parser differences, encoding, whitespace) ## Selecting elements (CSS selectors vs find_all) ```python from bs4 import BeautifulSoup soup = BeautifulSoup(open("page.html", encoding="utf-8"), "html.parser") soup.select("div.card > h2") # CSS — best for structural paths soup.select_one("#main table") # first match or None soup.find_all("a", href=True) # find_all — best for attribute filters soup.find_all("img", alt=False) # images missing alt soup.find("h2", string="Pricing") # exact text match ``` `select()` covers most CSS: descendants, `>`, `[attr=val]`, `:nth-of-type()`. It does not run JavaScript-era pseudo-classes like `:visible`. ## Extract: tables → CSV, links, clean text Table to CSV, with span detection: ```python import csv def table_to_csv(table, out_path): if table.find(attrs={"rowspan": True}) or table.find(attrs={"colspan": True}): print("warning: table uses rowspan/colspan; columns may misalign") rows = [[c.get_text(strip=True) for c in tr.find_all(["td", "th"])] for tr in table.find_all("tr")] with open(out_path, "w", newline="", encoding="utf-8") as f: csv.writer(f).writerows(rows) for i, table in enumerate(soup.find_all("table")): table_to_csv(table, f"table_{i}.csv") ``` Links with absolute resolution against a known base: ```python from urllib.parse import urljoin base = "https://example.com/docs/" links = [(a.get_text(strip=True), urljoin(base, a["href"])) for a in soup.find_all("a", href=True)] ``` Readable text without script/style noise: ```python for tag in soup(["script", "style", "noscript"]): tag.decompose() text = soup.get_text(separator="\n", strip=True) ``` ## Modify: rewrite links, insert/remove/replace elements ```python # Rewrite links (http → https, or path migration) for a in soup.find_all("a", href=True): if a["href"].startswith("http://example.com"): a["href"] = a["href"].replace("http://", "https://", 1) # Insert: add a class, append a child, insert a sibling div = soup.select_one("div.content") div["class"] = div.get("class", []) + ["highlight"] new_p = soup.new_tag("p") new_p.string = "Appended paragraph." div.append(new_p) div.insert_after(soup.new_tag("hr")) # Remove vs unwrap soup.select_one("aside.ad").decompose() # delete element and children for span in soup.find_all("span", class_="tracking"): span.unwrap() # keep children, drop the tag # Replace old = soup.select_one("center") new = soup.new_tag("div", attrs={"style": "text-align:center"}) new.extend(list(old.contents)) old.replace_with(new) open("page.html", "w", encoding="utf-8").write(str(soup)) ``` ## Sanitize: strip scripts and inline handlers For untrusted HTML that will be displayed, remove active content: ```python for tag in soup(["script", "iframe", "object", "embed"]): tag.decompose() for tag in soup.find_all(True): for attr in list(tag.attrs): if attr.lower().startswith("on"): # onclick, onload, ... del tag[attr] if tag.get("href", "").lstrip().lower().startswith("javascript:"): del tag["href"] ``` (For production-grade sanitizing use the `bleach` library — this covers the common cases.) ## Convert HTML → Markdown ```bash pandoc page.html -t gfm -o page.md # brew install pandoc ``` Or in Python, `pip install markdownify`: ```python from markdownify import markdownify md = markdownify(open("page.html", encoding="utf-8").read(), heading_style="ATX") ``` ## Gotchas (parser differences, encoding, whitespace) - **Parsers repair differently.** `html.parser` leaves fragments bare; `lxml` wraps them in ``; `html5lib` (`pip install html5lib`) repairs exactly like a browser but is slow. If output gained wrapper tags the input lacked, you switched parsers mid-task. - **Misnested tags relocate.** A `` with stray `` inside may have its rows moved outside the table in the repaired tree — "the selector finds nothing" often means the repair changed the structure, not that the data is missing. Inspect `soup.prettify()` on a slice. - **`str(soup)` vs `soup.prettify()`.** `str()` preserves the whitespace as parsed; `prettify()` re-indents every node and inserts newlines *inside text*, which can change rendering (whitespace matters around inline elements). Never prettify an existing file. - **Encoding detection.** BeautifulSoup guesses from `` and byte patterns when given bytes; given a mis-decoded *string*, it can't fix it. When in doubt, pass `open(path, "rb")`. - **`.string` vs `.get_text()`.** `.string` is `None` when a tag has multiple children; `.get_text()` always returns the concatenated text. Use `.get_text(strip=True)` for extraction. - **Attribute multi-values.** `class` comes back as a *list* (`["btn", "large"]`); `id` as a string. Appending a class means list-append, not string concat. - **Entities are decoded on parse** (`&` → `&`) and re-encoded minimally on output; byte-identical round-trips of untouched regions are not guaranteed, only semantically identical ones — diff renders, not bytes. - **Comments** are `Comment` nodes, invisible to `get_text()`; find them with `soup.find_all(string=lambda s: isinstance(s, Comment))` (`from bs4 import Comment`).