How to remove invisible characters from text
· 7 min read
Removal is not one operation. Unusual spaces should become an ordinary space so words do not merge; true invisibles should be deleted; script-meaningful characters and private-use glyphs should be reported rather than removed. Getting these distinctions wrong is how cleaners corrupt legitimate multilingual text.
The four actions
| Action | Applies to | Why |
|---|---|---|
| Replace with a space | U+00A0, U+202F, U+2007, U+3000, U+2800 and the rest of the space family | Deleting them merges the words they separate |
| Delete | Zero-width characters, Unicode Tags, reserved default-ignorables, noncharacters | Nothing legitimate renders from them |
| Convert to a newline | U+2028, U+2029, and the VT, FF and NEL controls | They separate lines or records; deleting them joins unrelated content |
| Report, do not remove | Private-use characters, script format characters, homoglyphs | They are usually visible and often correct; the decision belongs to the reader |
Context rules that prevent damage
- Keep a zero-width joiner between two emoji, or between letters of Arabic, Persian or an Indic script.
- Keep a zero-width non-joiner between letters of a joining script — Persian orthography depends on it.
- Keep a joiner that follows a virama, which is how Malayalam chillu and the Marathi eyelash-ra are written.
- Keep a variation selector after a base that legitimately takes one: an emoji, a CJK ideograph, or a Mongolian letter.
- Keep tag characters that form one of the three real subdivision flag emoji.
- Never treat a run of two or more chained invisibles as legitimate — no correct usage produces one.
Typography is a separate decision
Smart quotes, em dashes and ellipses are visible characters, not hidden ones. Converting them to ASCII is useful when text is heading for a plain-text field, a config file or a diff, and destructive when it is heading for publication.
That makes it a separate switch from invisible-character removal. It is still worth reporting what is there even when the switch is off, so nobody is told their document is clean when it still carries characters they cared about.
Doing it at scale
For a document, paste it into a tool and read the report. For a pipeline, sanitize at the boundary where untrusted text enters your system and keep the findings — the fact that a payload was present is worth more than the fact that it was removed.
curl -X POST https://api.zerotraceai.net/v1/clean \
-H "Authorization: Bearer $ZEROTRACE_KEY" \
-H "Content-Type: application/json" \
-d '{"text": "...", "options": {"profile": "safe"}}'Frequently asked
- Will cleaning break Turkish characters?
- It should not. Turkish letters such as ı, İ, ğ, ş, ç, ö and ü are ordinary letters with no invisible component, and a correct cleaner never touches them. The same applies to accented Latin, Greek and Cyrillic text.
- Does removing invisible characters change how the text looks?
- Almost never — that is what makes them invisible. The exceptions are unusual spaces, which become ordinary spaces of a slightly different width, and emoji sequences if you clean aggressively rather than safely.
Keep reading
- Invisible Unicode characters: a complete reference
Every category of invisible, zero-width and blank-rendering Unicode character, what each one is legitimately for, and when its presence is a signal that something is hidden in your text.
- Do AI watermarks use invisible characters? No — and here is the evidence
SynthID-Text and Anthropic's Claude watermark are statistical, living in the model's word choices rather than in any added character. Stripping invisible characters has no effect on them whatsoever.
- Invisible prompt injection with Unicode Tags
How the U+E0000 block hides instructions that people cannot see but language models read, why it works, and how to strip it before untrusted text reaches a model.