# name: processing-html description: Creates, reads, and modifies local HTML files — extracting text, tables, and links, or editing markup with BeautifulSoup. Use when the user asks to parse, scrape data out of, edit, clean up, or generate an .html or .htm file, extract a table or links from saved HTML, or build a static HTML page. Do not use for fetching live web pages (that is web browsing/scraping) or for XML data files.

# Processing HTML

# When to use / when NOT to use

Use for: local .html/.htm files — extracting text, tables, or links; targeted markup edits; generating static pages from scratch.
Do NOT use for: fetching live web pages (that is web browsing/scraping — get the file first, then this skill applies) or XML data files (use the XML skill — XML parsers are strict, HTML parsers are tolerant).

# Quick reference

Default: BeautifulSoup4 with the built-in html.parser (pip install beautifulsoup4). Escape hatch: the lxml parser (pip install lxml) for large or badly malformed files — faster and more lenient.

python

from bs4 import BeautifulSoup

soup = BeautifulSoup(open("page.html", encoding="utf-8"), "html.parser")

# Read / extract
title = soup.title.get_text(strip=True) if soup.title else ""
links = [(a.get_text(strip=True), a["href"]) for a in soup.find_all("a", href=True)]
rows = [[c.get_text(strip=True) for c in tr.find_all(["td", "th"])]
        for tr in soup.select("table tr")]           # lists-of-lists

# Modify — targeted, leave everything else untouched
for img in soup.find_all("img", src=True):
    if not img.get("alt"):
        img["alt"] = ""
open("page.html", "w", encoding="utf-8").write(str(soup))

Create — write HTML5 directly, no library:

html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Page title</title>
</head>
<body>
  <main>…</main>
</body>
</html>

# Rules

Never regex-parse HTML. Use the soup — selectors and tree navigation, always.
Edit surgically: modify the matched tags only; do not re-indent or prettify() an existing file (it rewrites every text node's whitespace).
New pages are HTML5: doctype, <meta charset="utf-8">, semantic tags (main, nav, article), lang attribute.
Extract tables as lists-of-lists; flag rowspan/colspan cells instead of silently mis-aligning columns.

# Workflow

Identify the operation: extract / modify / create.
Parse with html.parser; if the file is huge (>5 MB) or the tree looks wrong (missing siblings, truncated body), reparse with "lxml".
Perform the operation with the narrowest selector that matches (recipes in references/recipes.md).
Write back with the file's original encoding.
Validate: re-parse the written file and confirm the edit landed (query the changed element) and the element count of untouched regions is unchanged. Fix and repeat until clean.

# Edge cases & failure modes

bs4/lxml missing → pip install beautifulsoup4 / pip install lxml.
Malformed HTML → parsers auto-repair rather than error; if extraction returns nothing, the repaired tree may differ from the source — try the "lxml" parser and inspect soup.prettify()[:2000] to see the actual tree before concluding data is absent.
Encoding → check <meta charset> before assuming UTF-8; on UnicodeDecodeError, reopen with that charset or pass raw bytes to BeautifulSoup and let it detect.
JavaScript-rendered content → if the data isn't in the file, it never was: say so — no parser will find DOM built at runtime.
Fragments (no <html>/<body>) → parse and write back as-is; do not let output gain wrapper tags the input lacked (html.parser doesn't add them; lxml does).

# References

Deeper copy-paste recipes (selectors, table→CSV, rewriting links, sanitizing): see references/recipes.md.