# HTML Recipes

# Contents

Selecting elements (CSS selectors vs find_all)
Extract: tables → CSV, links, clean text
Modify: rewrite links, insert/remove/replace elements
Sanitize: strip scripts and inline handlers
Convert HTML → Markdown
Gotchas (parser differences, encoding, whitespace)

# Selecting elements (CSS selectors vs find_all)

python

from bs4 import BeautifulSoup
soup = BeautifulSoup(open("page.html", encoding="utf-8"), "html.parser")

soup.select("div.card > h2")            # CSS — best for structural paths
soup.select_one("#main table")          # first match or None
soup.find_all("a", href=True)           # find_all — best for attribute filters
soup.find_all("img", alt=False)         # images missing alt
soup.find("h2", string="Pricing")       # exact text match

select() covers most CSS: descendants, >, [attr=val], :nth-of-type(). It does not run JavaScript-era pseudo-classes like :visible.

# Extract: tables → CSV, links, clean text

Table to CSV, with span detection:

python

import csv

def table_to_csv(table, out_path):
    if table.find(attrs={"rowspan": True}) or table.find(attrs={"colspan": True}):
        print("warning: table uses rowspan/colspan; columns may misalign")
    rows = [[c.get_text(strip=True) for c in tr.find_all(["td", "th"])]
            for tr in table.find_all("tr")]
    with open(out_path, "w", newline="", encoding="utf-8") as f:
        csv.writer(f).writerows(rows)

for i, table in enumerate(soup.find_all("table")):
    table_to_csv(table, f"table_{i}.csv")

Links with absolute resolution against a known base:

python

from urllib.parse import urljoin
base = "https://example.com/docs/"
links = [(a.get_text(strip=True), urljoin(base, a["href"]))
         for a in soup.find_all("a", href=True)]

Readable text without script/style noise:

python

for tag in soup(["script", "style", "noscript"]):
    tag.decompose()
text = soup.get_text(separator="\n", strip=True)

# Modify: rewrite links, insert/remove/replace elements

python

# Rewrite links (http → https, or path migration)
for a in soup.find_all("a", href=True):
    if a["href"].startswith("http://example.com"):
        a["href"] = a["href"].replace("http://", "https://", 1)

# Insert: add a class, append a child, insert a sibling
div = soup.select_one("div.content")
div["class"] = div.get("class", []) + ["highlight"]
new_p = soup.new_tag("p")
new_p.string = "Appended paragraph."
div.append(new_p)
div.insert_after(soup.new_tag("hr"))

# Remove vs unwrap
soup.select_one("aside.ad").decompose()      # delete element and children
for span in soup.find_all("span", class_="tracking"):
    span.unwrap()                            # keep children, drop the tag

# Replace
old = soup.select_one("center")
new = soup.new_tag("div", attrs={"style": "text-align:center"})
new.extend(list(old.contents))
old.replace_with(new)

open("page.html", "w", encoding="utf-8").write(str(soup))

# Sanitize: strip scripts and inline handlers

For untrusted HTML that will be displayed, remove active content:

python

for tag in soup(["script", "iframe", "object", "embed"]):
    tag.decompose()
for tag in soup.find_all(True):
    for attr in list(tag.attrs):
        if attr.lower().startswith("on"):        # onclick, onload, ...
            del tag[attr]
    if tag.get("href", "").lstrip().lower().startswith("javascript:"):
        del tag["href"]

(For production-grade sanitizing use the bleach library — this covers the common cases.)

# Convert HTML → Markdown

bash

pandoc page.html -t gfm -o page.md            # brew install pandoc

Or in Python, pip install markdownify:

python

from markdownify import markdownify
md = markdownify(open("page.html", encoding="utf-8").read(), heading_style="ATX")

# Gotchas (parser differences, encoding, whitespace)

Parsers repair differently. html.parser leaves fragments bare; lxml wraps them in <html><body>; html5lib (pip install html5lib) repairs exactly like a browser but is slow. If output gained wrapper tags the input lacked, you switched parsers mid-task.
Misnested tags relocate. A <table> with stray </div> inside may have its rows moved outside the table in the repaired tree — "the selector finds nothing" often means the repair changed the structure, not that the data is missing. Inspect soup.prettify() on a slice.
str(soup) vs soup.prettify(). str() preserves the whitespace as parsed; prettify() re-indents every node and inserts newlines inside text, which can change rendering (whitespace matters around inline elements). Never prettify an existing file.
Encoding detection. BeautifulSoup guesses from <meta charset> and byte patterns when given bytes; given a mis-decoded string, it can't fix it. When in doubt, pass open(path, "rb").
.string vs .get_text(). .string is None when a tag has multiple children; .get_text() always returns the concatenated text. Use .get_text(strip=True) for extraction.
Attribute multi-values. class comes back as a list (["btn", "large"]); id as a string. Appending a class means list-append, not string concat.
Entities are decoded on parse (& → &) and re-encoded minimally on output; byte-identical round-trips of untouched regions are not guaranteed, only semantically identical ones — diff renders, not bytes.
Comments are Comment nodes, invisible to get_text(); find them with soup.find_all(string=lambda s: isinstance(s, Comment)) (from bs4 import Comment).