How to Extract Text with Formatting Using Beautifulsoup

The problem with a plain get_text() call

By default, soup.get_text() concatenates all the text nodes it finds. Block-level elements like <p>, <div>, <li> and <br> carry visual meaning in the browser, but that meaning is lost once the markup is stripped. The result is text where the last word of one paragraph runs straight into the first word of the next.

The fix is to tell BeautifulSoup how to join the pieces, or to process elements individually so you can decide where breaks belong.

Quick win: the separator argument

The simplest improvement is passing a separator to get_text(). This inserts a chosen string between text fragments, which restores at least some spacing.

from bs4 import BeautifulSoup

html = """
<div>
  <p>First paragraph.</p>
  <p>Second paragraph.</p>
  <ul><li>Item one</li><li>Item two</li></ul>
</div>
"""

soup = BeautifulSoup(html, "html.parser")
text = soup.get_text(separator="\n", strip=True)
print(text)

Using separator="\n" with strip=True places each text node on its own line and trims surrounding whitespace. This is a big step up from the default and is enough for many tasks where you just want paragraphs and list items separated.

Walking specific block elements

When you need finer control, iterate over the elements you care about and build the output yourself. This lets you treat headings, paragraphs and list items differently.

lines = []
for el in soup.find_all(["h1", "h2", "h3", "p", "li"]):
    txt = el.get_text(strip=True)
    if not txt:
        continue
    if el.name == "li":
        lines.append("- " + txt)
    else:
        lines.append(txt)

formatted = "\n\n".join(lines)
print(formatted)

Here list items get a leading dash so they read as bullets, and other blocks are separated by blank lines. You can extend the logic to add a # prefix for headings or to indent nested lists, producing output close to Markdown.

Handling line breaks

Inline <br> tags do not appear as elements you can easily loop over, so they tend to vanish. One robust trick is to replace them with newline text nodes before extracting:

for br in soup.find_all("br"):
    br.replace_with("\n")

Run this before your get_text() call and single-line breaks inside paragraphs will survive into the output.

Recursively descending the tree

For complex pages, a recursive walk gives you full authority over spacing. You visit each node, emit its text, and add newlines when you leave a block-level tag. This is more code, but it produces the most faithful results when structure really matters.

BLOCK = {"p", "div", "li", "h1", "h2", "h3", "h4", "tr", "section"}

def render(node, out):
    for child in node.children:
        if child.name is None:
            out.append(child.strip())
        else:
            render(child, out)
            if child.name in BLOCK:
                out.append("\n")

out = []
render(soup, out)
print(" ".join(p for p in out if p))

Adjust the BLOCK set and joining logic to suit the kind of pages you scrape. Tables, for example, often need tab separators between cells and newlines between rows.

Tips for cleaner, more reliable output

• Strip out <script> and <style> tags first with .decompose() so their contents never reach your text.
• Normalise repeated blank lines with a quick regular expression so multiple breaks collapse into one.
• Decode HTML entities by relying on BeautifulSoup itself rather than parsing raw bytes by hand.
• Test against several real pages, because layouts vary far more than a single sample suggests.

Where proxies fit in

Extracting clean, formatted text is only useful if you can fetch the pages in the first place. At scale, repeated requests from one IP often get throttled or blocked, so many scrapers route traffic through proxies to spread load and reach region-specific content. If you are weighing options, Cheapest Proxies (cheapest-proxies.com) is a strong value-focused choice worth considering, and it is always wise to compare proxy plans on coverage and price before committing.

Preserve inline emphasis, not just block breaks

Most tutorials stop at separating paragraphs and list items, but real readable output also needs inline semantics. A sentence where one word was bold or linked loses meaning when flattened. Instead of stripping everything, walk inline elements and wrap their text with lightweight markers as you go: surround <strong> and <b> content with double asterisks, <em> and <i> with single ones, and turn an <a> into a text (url) or Markdown link pair. Because these tags nest inside block elements, the cleanest approach is to handle them in a child-visiting pass before you decide where block breaks go, so the markers travel with the text into your final string.

Tables, definition lists and other awkward structures

Generic separator tricks fall apart on tabular data because every cell becomes an undifferentiated line. To keep a table legible, iterate rows with find_all("tr"), pull each row's cells with find_all(["td", "th"]), join cells with a tab or pipe, and join rows with newlines. The same row-then-cell thinking applies to definition lists (<dl> with <dt> and <dd> pairs) and to nested ordered lists, where you should track depth and indent accordingly. The general rule is that any element whose meaning depends on two-dimensional layout needs explicit serialisation logic rather than a flat separator.

Quick serialisation checklist for tables

• Treat header rows separately so you can underline or bold them.
• Replace empty cells with a placeholder so columns stay aligned.
• Escape any separator character that already appears inside cell text.

When to reach for a dedicated converter instead

Hand-rolled extraction is great for learning and for narrow, predictable pages, but it becomes a maintenance burden across many layouts. Mature HTML-to-Markdown and HTML-to-text libraries already encode the edge cases for headings, blockquotes, code blocks, ordered lists and links. A practical pattern is to use BeautifulSoup as the cleaning layer, removing navigation, scripts and styles, then pass the reduced HTML fragment to the converter for serialisation. You get BeautifulSoup's flexible selection together with a converter's battle-tested formatting rules, which is usually faster than perfecting a recursive renderer yourself.

Encoding, whitespace and entity pitfalls

Faithful output also depends on getting characters right. Source HTML is full of indentation whitespace that is invisible in a browser but shows up as ragged spacing in extracted text, so collapse runs of whitespace with a regular expression after extraction. Non-breaking spaces and other entities should be left to BeautifulSoup to decode rather than handled manually, and you should confirm the response encoding before parsing so accented characters and symbols survive. These small details are what separate output that merely looks structured from output you can safely feed into search, diffing or a language model downstream.

Keep comparing with these closely related pages.