How to Get Text Using Lxml

Setting Up lxml

If you have not installed the library yet, add it to your environment first. lxml is a fast, C-backed parser that handles both HTML and XML, which is why it is so popular for web data work.

pip install lxml

from lxml import html, etree

source = """
<div class="card">
  <h2>Proxy Basics</h2>
  <p>Rotating <b>residential</b> IPs help with blocks.</p>
</div>
"""

tree = html.fromstring(source)

Here html.fromstring() returns the root element of the parsed fragment. From this point on, every text operation works against that tree.

Getting Text from a Single Element

The simplest property is .text. It returns only the text that appears immediately after the element's opening tag and before its first child, not the text inside nested children.

p = tree.xpath('//p')[0]
print(p.text)   # 'Rotating '

Notice that .text stops at the <b> child. It does not include "residential" because that word belongs to the child element, not the paragraph itself. This surprises a lot of beginners.

Understanding .tail

Every element also has a .tail attribute: the text that follows the element's closing tag but still sits inside the parent. The "residential" word's container has a tail of " IPs help with blocks." Together, .text and .tail let lxml represent mixed content faithfully.

Getting All Text in a Subtree

Most of the time you want every readable word inside an element, regardless of nesting. The cleanest method is text_content(), available on HTML elements.

p = tree.xpath('//p')[0]
print(p.text_content())   # 'Rotating residential IPs help with blocks.'

This concatenates the element's own text, all descendant text, and the tail text in document order. It is the go-to call when you simply want the visible string of a block.

Using itertext()

If you need the pieces rather than one joined string, iterate over them:

for chunk in p.itertext():
    print(repr(chunk))

This yields each text node separately, which is handy when you want to filter, strip, or rejoin them with custom separators.

Extracting Text with XPath

XPath gives you a declarative way to target text directly. Two patterns matter most.

• //h2/text() selects the direct text nodes of matching elements, returning a list of strings.
• string(//div) or the per-node .xpath('string(.)') returns the full collapsed text of a node, similar to text_content().

heading = tree.xpath('//h2/text()')
print(heading)            # ['Proxy Basics']

full = tree.xpath('string(//div)')
print(full.strip())       # 'Proxy Basics Rotating residential IPs help with blocks.'

Use text() when you want individual nodes and string() when you want one merged result.

Cleaning the Output

Raw extracted text often carries leftover whitespace and line breaks from the source markup. A small normalisation step keeps your data tidy.

import re

raw = p.text_content()
clean = re.sub(r'\s+', ' ', raw).strip()

For XML rather than HTML, parse with etree.fromstring() instead of the html module. The text methods behave the same, though XML is stricter about well-formed input and namespaces.

Why This Matters for Scraping

Reliable text extraction is the foundation of clean datasets. When you combine lxml with rotating proxies, you reduce blocks while you collect pages at scale, and the parsing logic above turns each fetched response into structured, usable text.

Stripping Noise Before You Extract

The cleanest way to avoid menu text, scripts and inline styles leaking into your output is to remove those subtrees before you ever call a text method. lxml ships a dedicated cleaner and a simple strip helper for exactly this.

from lxml import html, etree

tree = html.fromstring(source)
etree.strip_elements(tree, 'script', 'style', with_tail=False)

text = tree.text_content()

The with_tail=False flag is important: without it, lxml keeps the tail text that followed each removed tag, which can leave stray fragments behind. For heavier sanitisation, lxml.html.clean.Cleaner can drop comments, embedded objects and forms in one pass, leaving you with readable prose only.

Filtering itertext() to Real Content

People assume itertext() yields only words, but it also surfaces comment nodes and processing instructions when they carry text. You can pass tag names to restrict it, or test the parent tag of each chunk to skip the ones you do not want.

wanted = []
for node in tree.iter():
    if node.tag in ('script', 'style'):
        continue
    if node.text and node.text.strip():
        wanted.append(node.text.strip())

Iterating elements yourself, rather than calling a blanket joiner, gives you a hook to attach the source tag, class or XPath to each piece of text, which is invaluable when you later need to know where a string came from.

Whitespace and Unicode the Engine Way

Instead of post-processing with a regex, you can push normalisation into XPath. normalize-space() collapses internal runs of whitespace and trims the ends, all inside the C engine.

clean = tree.xpath('normalize-space(string(//div[@class="card"]))')

Be aware that normalize-space treats non-breaking spaces and other Unicode whitespace differently from Python's str.strip(). If your source uses   heavily, you may still want a follow-up pass that replaces   before storing the result, otherwise downstream comparisons and de-duplication can quietly fail.

Streaming Text Out of Huge Documents

When a file is too big to hold in memory, build the tree incrementally and clear elements as you finish with them. This keeps memory flat even across enormous catalogues or exports.

for _, el in etree.iterparse('big.html', html=True, tag='p'):
    print(el.text_content().strip())
    el.clear()

The el.clear() call frees the children you have already read. For maximum thrift, also delete preceding siblings so the root does not accumulate emptied shells. This pattern turns a script that would crash on a multi-gigabyte file into one that runs in constant memory.

Keep comparing with these closely related pages.