How to Parse XML with Lxml

Installing and Importing lxml

The XML tools live in the etree module. Install the package and import it before you start.

pip install lxml

from lxml import etree

The etree module gives you both parsing functions and the element classes you will use to traverse the result.

Parsing from a String or a File

If your XML is already in memory as a string, use fromstring(). If it lives on disk, use parse(), which returns an ElementTree wrapper rather than the root element directly.

xml_data = """
<catalog>
  <proxy type="residential">
    <country>Germany</country>
    <rotating>true</rotating>
  </proxy>
  <proxy type="datacenter">
    <country>Netherlands</country>
    <rotating>false</rotating>
  </proxy>
</catalog>
"""

root = etree.fromstring(xml_data)

# From a file instead:
# tree = etree.parse('catalog.xml')
# root = tree.getroot()

Either way you end up with a root Element that you can iterate and query.

Navigating the Element Tree

Each element is iterable over its direct children, and you can read tags and text easily.

for proxy in root:
    country = proxy.find('country').text
    rotating = proxy.find('rotating').text
    print(proxy.tag, country, rotating)

find() returns the first matching child, while findall() returns every match. Use iter() to walk descendants at any depth.

Reading Attributes

Attributes are exposed through the .attrib dictionary or the .get() method.

for proxy in root.findall('proxy'):
    print(proxy.get('type'))     # 'residential', 'datacenter'

Using .get() is safer than indexing .attrib directly because it returns None when the attribute is missing instead of raising an error.

Querying with XPath

XPath is where lxml shines. Instead of looping manually, describe what you want and let the engine find it.

# All country names
countries = root.xpath('//country/text()')

# Only residential proxy countries
res = root.xpath('//proxy[@type="residential"]/country/text()')

Predicates like [@type="residential"] filter by attribute, and functions such as contains() and starts-with() let you match partial values.

Handling Namespaces

Real-world XML, especially feeds and standards-based formats, often declares namespaces. You must register a prefix map to query namespaced elements.

ns = {'atom': 'http://www.w3.org/2005/Atom'}
links = root.xpath('//atom:link/@href', namespaces=ns)

Skipping the namespace map is the most common reason an XPath query silently returns nothing on otherwise valid XML.

Error Handling and Recovery

Malformed documents raise an XMLSyntaxError. For messy or partially broken sources, a forgiving parser can help.

parser = etree.XMLParser(recover=True)
root = etree.fromstring(broken_xml, parser=parser)

• Use recover=True to skip past minor syntax problems.
• Wrap parsing in try / except etree.XMLSyntaxError to log and continue on bad input.
• Validate against a schema with etree.XMLSchema when correctness is critical.

Putting It Together for Scraping

When you fetch XML endpoints at scale, such as multilingual sitemaps or regional feeds, pairing lxml parsing with rotating proxies keeps requests flowing without tripping rate limits, while the parsing logic above turns each response into clean, structured records.

Parsing Untrusted XML Safely

XML parsing has a well-known security pitfall: external entity expansion, known as XXE, which can read local files or trigger network calls when you parse a hostile document. The base guide shows how to recover from malformed input, but recovery is not the same as safety. For any XML you did not author, harden the parser explicitly.

parser = etree.XMLParser(
    resolve_entities=False,
    no_network=True,
    huge_tree=False,
)
root = etree.fromstring(untrusted_bytes, parser=parser)

Disabling entity resolution stops the classic billion-laughs and file-disclosure attacks, while no_network=True prevents the parser from fetching remote DTDs. Leaving huge_tree off keeps lxml's built-in limits in place, which guard against deliberately oversized documents designed to exhaust memory.

Streaming Large Documents Without the Memory Hit

A multi-gigabyte sitemap or export will not fit comfortably in a single in-memory tree. The event-driven iterparse approach reads one element at a time and lets you discard it once processed.

for event, elem in etree.iterparse('export.xml', tag='record'):
    handle(elem.findtext('id'), elem.findtext('name'))
    elem.clear()
    while elem.getprevious() is not None:
        del elem.getparent()[0]

The clear() call empties the element you just used, and the inner loop deletes already-processed siblings so the root does not slowly fill with emptied shells. This is the difference between a job that completes and one that is killed by the operating system for using too much memory.

Building and Modifying Trees, Not Just Reading

Parsing is half the story; lxml is equally good at constructing XML. You can assemble a document programmatically, set attributes and text, then serialise it.

root = etree.Element('catalog')
proxy = etree.SubElement(root, 'proxy', type='residential')
etree.SubElement(proxy, 'country').text = 'Germany'

out = etree.tostring(root, pretty_print=True,
                     xml_declaration=True, encoding='UTF-8')

Always pass an explicit encoding when serialising. Without it lxml returns a Python str and omits the declaration, which can break consumers that rely on the charset line. pretty_print=True is fine for human-readable output but adds whitespace, so avoid it when byte-for-byte fidelity or signature validation matters.

Transforming XML with XSLT

When you need to reshape one XML vocabulary into another, native XSLT often beats hand-written traversal. lxml compiles a stylesheet once and applies it to many documents.

xslt = etree.XSLT(etree.parse('transform.xsl'))
result = xslt(root)

This is ideal for converting vendor feeds into your own schema, or flattening nested records into a tabular form, and it keeps transformation logic declarative and in one file rather than scattered across Python loops.

Keep comparing with these closely related pages.