Import CoNLL File

Please select existing CoNLL/CoNLL-U file to import for visual inspection

Info

Name: Corpus Navigator
Version: 0.1.0
Date: 31 January 2026
Publisher: The Balagha Corpus
Author: Mandar Marathe

Use: This tool implements standardized rules for numbering words in any text, supporting multiple languages and writing systems. The tool allows for the determination of word position offsets consistently and reliably.

Requirements: The tool is a fully stand-alone HTML/Javascript file which can be run on any web browser, and does not need internet access.

Supports: RTL languages (Arabic, Hebrew, Persian, Urdu), LTR languages (English, Spanish, etc.), and mixed-direction text. Import / export of CoNLL-style files.

Applicable to: Social media posts, tweets, comments, messages, and any text corpus requiring consistent word position verification.

Implementation Notes

For Manual Counting:

  1. Read through these rules carefully before beginning
  2. Count from 1, not 0
  3. Write down each word with its position to avoid errors
  4. Use the Word Position Finder to verify your counts
  5. When in doubt, consult the examples above

For Programmatic Counting:

  1. The Word Position Finder implements all rules above
  2. The tool produces identical output to careful manual counting
  3. Use the tool for verification of annotations

Verification Process

To verify a word position annotation in your dataset:

  1. Retrieve the text from your corpus
  2. Copy the full text content
  3. Paste into the Word Position Finder (switch to "Word Position Finder" tab)
  4. Locate the word position referenced in the annotation (e.g., W7-8)
  5. Verify the annotation matches the identified span

Changelog

Version 0.1.0 (31 January 2026)

  • Initial release (previously known as Word Position Finder (v0.1.0))
  • 21 core rules defined
  • Language-agnostic implementation
  • Support for RTL and LTR scripts
  • Import / export of CoNLL-style files
  • Complete examples provided

Citation

Mandar Marathe. "Corpus Navigator (v0.1.0)" Github, 2026. https://github.com/Balagha-Corpus/Corpus-Navigator/tree/v0.1.0.

Contact

For questions about this tool or to report issues, please email [email protected].

License

This documentation is released under CC-BY 4.0 license.
You are free to share and adapt with attribution.

Word Counting Rules

This document defines the rules used to determine word positions in text corpus annotations. These rules apply whether counting is performed manually or programmatically.

Purpose: To ensure consistent and reproducible verification of annotations across any text dataset, regardless of language or content type.

Core Rules

Rule 1: Word Position Indexing

Word positions start from 1 (not 0)

Example (English): "Hello world"

  • W1: Hello
  • W2: world

Example (Arabic): "مرحبا بك"

  • W1: مرحبا
  • W2: بك

Rule 2: Hashtag Tokenization

Hashtags are split on separators: dash -, underscore _, or emdash —

The # symbol remains attached to the first segment only.

Example 1: #climate_change

  • W1: #climate
  • W2: change

Example 2: #breaking-news

  • W1: #breaking
  • W2: news

Example 3: Single-word hashtag #important

  • W1: #important (no splitting)

Example 4 (Arabic): #آخر_الأخبار

  • W1: #آخر
  • W2: الأخبار

Rule 3: Emoji Handling

All emoji are stripped and do not count as words

This includes: flag emoji, symbols, face emoji, and any Unicode emoji character.

Example (English): "Great news 🎉 today ❤️"

  • W1: Great
  • W2: news
  • W3: today

Example (Arabic): "خبر جيد 🎉 اليوم ❤️"

  • W1: خبر
  • W2: جيد
  • W3: اليوم

Rule 4: Punctuation Handling

Punctuation attached to words is kept; free-standing punctuation is stripped

Attached Punctuation (kept as part of word):

Example 1 (English): world, → one word (comma attached)

Example 2 (English): really? → one word (question mark attached)

Example 3 (English): "hello" → one word (quotes attached)

Example 4 (Arabic): والكلمة، → one word (comma attached)

Free-standing Punctuation (stripped):

Example 1: . . . or ... (ellipsis alone) → zero words

Example 2: , (comma alone) → zero words

Example 3 (English): "Hello , world"

  • W1: Hello
  • W2: world
  • (Free-standing comma is stripped)

Example 4 (Arabic): "كلمة ، أخرى"

  • W1: كلمة
  • W2: أخرى
  • (Free-standing comma is stripped)

Rule 5: URL Handling

URLs count as single words regardless of length

Example (English): "Check https://example.com/article for details"

  • W1: Check
  • W2: https://example.com/article
  • W3: for
  • W4: details

Example (Arabic): "انظر https://example.com/article للتفاصيل"

  • W1: انظر
  • W2: https://example.com/article
  • W3: للتفاصيل

Language-Specific Rules

Rule 6: Non-Latin Punctuation Marks

Handle language-specific punctuation the same as Western punctuation

Examples include:

  • Arabic comma ، - attached: keep; free-standing: strip
  • Arabic question mark ؟ - attached: keep; free-standing: strip
  • Arabic semicolon ؛ - attached: keep; free-standing: strip
  • Chinese/Japanese punctuation - attached: keep; free-standing: strip
  • Ellipsis … (single character) - free-standing: strip

Example (Arabic): "الكلمة؛ كلمة"

  • W1: الكلمة؛
  • W2: كلمة

Rule 7: Tatweel (Kashida) Character

Tatweel ـ is kept as part of the word (Arabic-specific)

Tatweel is used for emphasis or visual elongation in Arabic social media.

Example (Arabic): "كلـــمة" (with tatweel)

  • W1: كلـــمة (one word)

Rule 8: Diacritics

All diacritics are preserved and do not affect word boundaries

This applies to any language with diacritics: Arabic harakat, Hebrew niqqud, Vietnamese tone marks, etc.

Example (Arabic): "الكَلِمَةُ"

  • W1: الكَلِمَةُ (one word with all diacritics)

Example (Vietnamese): "tiếng"

  • W1: tiếng (one word with tone marks)

Rule 9: Numerals

Numerals in any script count as single words

This includes Western numerals (0-9), Arabic-Indic numerals (٠-٩), and numerals from other writing systems.

Example 1 (English): "In 2025 we started"

  • W1: In
  • W2: 2025
  • W3: we
  • W4: started

Example 2 (Arabic): "في ٢٠٢٥ بدأنا"

  • W1: في
  • W2: ٢٠٢٥
  • W3: بدأنا

Example 3: Date format "15/1/2025"

  • W1: 15/1/2025 (one word)

Rule 10: Mentions and Tags

Social media mentions (@ symbol) count as single words

Example (English): "Thanks @username for help"

  • W1: Thanks
  • W2: @username
  • W3: for
  • W4: help

Example (Arabic): "شكرا @username على"

  • W1: شكرا
  • W2: @username
  • W3: على

Rule 11: Quote Marks

Quote marks attached to text are kept; free-standing are stripped

Includes Western quotes "", Arabic quotes «», and other quotation marks.

Example 1 (English): "word" → one word

Example 2 (Arabic): «الكلمة» → one word

Example 3: Free-standing " " or « » → zero words (stripped)

Rule 12: Repeated Letters for Emphasis

Letter repetition counts as single word

Common in social media across many languages for emphasis or excitement.

Example (English): "yessss"

  • W1: yessss (one word)

Example (Arabic): "نعممممم"

  • W1: نعممممم (one word)

Rule 13: Mixed-Script Text

Each segment in a different script separated by space is a separate word

Example (Arabic-English): "كلمة word أخرى"

  • W1: كلمة
  • W2: word
  • W3: أخرى

Example (Hebrew-English): "מילה word אחרת"

  • W1: מילה
  • W2: word
  • W3: אחרת

Note: No space between scripts (rare): "wordكلمة" → W1: wordكلمة (one word)

Rule 14: Line Breaks and Whitespace

Line breaks and paragraph marks are treated as whitespace

  • \n (newline) → treated as space
  • \r (carriage return) → treated as space
  • \t (tab) → treated as space
  • Multiple consecutive spaces → collapsed to single space

Example: "word\nanother" → same as "word another"

  • W1: word
  • W2: another

Rule 15: Zero-Width Characters

Zero-width characters are stripped and ignored

This includes:

  • Zero-width joiner (ZWJ) \u200D
  • Zero-width non-joiner (ZWNJ) \u200C
  • Zero-width space \u200B

These characters do not create word boundaries and are completely removed.

Rule 16: Retweet Markers

RT markers are counted as separate words if present

Example 1 (English): "RT @user: Important message"

  • W1: RT
  • W2: @user:
  • W3: Important
  • W4: message

Example 2 (Arabic): "RT: رسالة مهمة"

  • W1: RT:
  • W2: رسالة
  • W3: مهمة

Rule 17: Leading and Trailing Whitespace

Strip all leading and trailing whitespace before tokenization

Example: " word another " → "word another"

  • W1: word
  • W2: another

Edge Cases

Rule 18: Parentheses and Brackets

Content within parentheses/brackets is treated as part of the word

Example 1: (word) → W1: (word) (one word)

Example 2: ( word ) (spaces inside) → W1: (word) (spaces normalized)

Example 3: Free-standing () or [] → zero words (stripped)

Rule 19: Special Symbols

Ampersands and standalone symbols are stripped

  • & alone → stripped (zero words)
  • @ alone → stripped (zero words)
  • # alone → stripped (zero words)
  • These only count if attached to text

Example (English): "word & another"

  • W1: word
  • W2: another
  • (The & is stripped)

Rule 20: Mixed Punctuation from Different Scripts

Punctuation from any script follows the same attachment rules

Example: "word، another. كلمة"

  • W1: word،
  • W2: another.
  • W3: كلمة

Rule 21: Special Ligatures

Special ligatures count as single words

Examples:

  • Arabic: ﷲ (Allah ligature) → W1: ﷲ (one word)
  • Arabic: ﷺ (PBUH ligature) → W1: ﷺ (one word)
  • Other scripts with ligatures follow the same pattern

Complete Examples

Example 1: Simple Text (English)

Text: "The weather today is beautiful"

Tokenization:

  • W1: The
  • W2: weather
  • W3: today
  • W4: is
  • W5: beautiful

Example 2: Social Media Post with Hashtag (English)

Text: "Breaking news #climate_action is trending today"

Tokenization:

  • W1: Breaking
  • W2: news
  • W3: #climate
  • W4: action
  • W5: is
  • W6: trending
  • W7: today

Example 3: Post with Emoji and Punctuation (English)

Text: "Great news 🎉 everyone, check it out! ❤️"

Tokenization:

  • W1: Great
  • W2: news
  • W3: everyone,
  • W4: check
  • W5: it
  • W6: out!

(Emoji 🎉 and ❤️ are stripped)

Example 4: Complex Post with URL (English)

Text: "RT @user: Amazing story... #breaking-news #must_read https://example.com 👍"

Tokenization:

  • W1: RT
  • W2: @user:
  • W3: Amazing
  • W4: story...
  • W5: #breaking
  • W6: news
  • W7: #must
  • W8: read
  • W9: https://example.com

(Emoji 👍 is stripped; ellipsis attached to word is kept)

Example 5: Simple Text (Arabic)

Text: "الطقس اليوم جميل جدا"

Tokenization:

  • W1: الطقس
  • W2: اليوم
  • W3: جميل
  • W4: جدا

Example 6: Social Media Post (Arabic)

Text: "خبر عاجل #آخر_الأخبار يتصدر الآن"

Tokenization:

  • W1: خبر
  • W2: عاجل
  • W3: #آخر
  • W4: الأخبار
  • W5: يتصدر
  • W6: الآن

Example 7: Mixed Language Post

Text: "Check this article المقال مهم very important"

Tokenization:

  • W1: Check
  • W2: this
  • W3: article
  • W4: المقال
  • W5: مهم
  • W6: very
  • W7: important
Corpus Navigator - Free download