Recent chats
ASKGEEZ Q&A GUIDETechnology & devices

How can I split text on commas while keeping commas inside quoted phrases?

Quick answer

A plain comma split treats every comma as a separator, including commas inside quoted fields. For a tightly controlled string with double-quoted fields and no escaped quotes or embedded newlines, a regex can split only at commas followed by an even number of quotes. For actual CSV or input with varied quoting rules, use a CSV-aware parser configured for that format.

LET’S WORK THROUGH IT

The question, the details, the next step.

Question

Why does splitting on commas give me the wrong fields? For example, I have Ada,"London, UK",editor and want three values.

Geez

A basic split uses each comma as a separator, without checking whether it appears inside quotes. Python’s re.split likewise splits at occurrences of its separator pattern; it does not automatically interpret CSV quoting. [S2]

For your example, a plain comma split would break London, UK into two pieces. The intended fields are Ada, London, UK, and editor. The key distinction is whether you have a deliberately simple string or a CSV record whose quoting rules need to be interpreted. A delimiter is just a character to a plain split; quote-aware parsing requires additional rules about where quoted fields begin and end.

Follow-up question

Can you show a regex for the simple case, and spell out what assumptions it makes?

Geez

If the input is one line, uses double quotes only to surround fields, and never has escaped quotes inside a field, you can split at commas that have an even number of double quotes to their right:

import re

text = 'Ada,"London, UK",editor'
fields = re.split(r',(?=(?:[^"]*"[^"]*")*[^"]*$)', text)
print(fields)

For that constrained example, the result is ['Ada', '"London, UK"', 'editor']; the quotes remain in the field. The lookahead checks the rest of the line for pairs of quotes, so the comma inside the quoted phrase is not treated as a separator.

This pattern is an illustration, not a general CSV parser. It relies on balanced, unescaped quotes and a simple one-line format. It does not establish that arbitrary CSV—or malformed input—will be parsed correctly. If your input can break any of those assumptions, don’t try to patch the pattern without first defining the format rules.

Follow-up question

What should I use when the text is real CSV, or when fields might contain escaped quotes or line breaks?

Geez

Use a CSV-aware reader rather than extending the regex. Python’s csv.reader reads records according to a dialect, including settings such as the delimiter and quote character; quoted fields can contain delimiters or newlines. Configure the reader for the actual CSV conventions you receive, since dialects and parser settings can vary. [S1]

For example, if text contains CSV data and the assumed dialect uses commas and double quotes:

import csv
from io import StringIO

text = 'Ada,"London, UK",editor\n'
rows = list(csv.reader(StringIO(text)))
print(rows[0])

This gives ['Ada', 'London, UK', 'editor'] for that example. If reading a file, pass the file object to csv.reader instead. Check the expected delimiter, quote character, and escaping conventions before relying on the result; this example does not determine the format of an unknown input. In particular, make sure the parser settings match the data producer’s dialect rather than assuming every CSV source uses identical conventions.

A practical checklist

  • Decide whether the input is a simple, controlled string or CSV data with a defined dialect.

  • Use the regex only if quotes are balanced, unescaped, and the input is a single line.

  • For general CSV, choose a CSV-aware reader and configure its delimiter and quote character for the input.

  • Test representative cases, including a quoted comma and any escaping or line breaks the input may contain.

  • Inspect the parsed fields before using them if the input format is uncertain.

A few more useful answers

Will the regex remove the quotation marks?

No. The shown split leaves the quotation marks in the field. Removing them safely depends on the input rules; a CSV reader returns parsed field values for its configured dialect.

Does the regex handle doubled quotes inside a CSV field?

Not reliably. Its assumptions exclude escaped quotes, so use a CSV-aware parser for fields that may contain them.

Can a CSV field contain a newline?

Quoted fields can contain newlines under CSV conventions, and Python’s CSV reader supports records with quoted newlines. A one-line regex split is not a suitable general solution for that case. [S1]

Sources

References used for this guide.

  1. [S1] Python 3.12 documentation: CSV reader, delimiters, and quoting
    docs.python.org · Checked
  2. [S2] Python 3.16 documentation: regular-expression splitting
    docs.python.org · Checked

YOUR QUESTION. YOUR DETAILS.

Let’s work through your situation.

Get a tailored answer or help with your next task. Starts at $1, with a quote before you pay.

Ask your own question
← Back to Geez at Work