Dead Circuits: Activation Profiling and Script Invisibility in Large Language Models

Abstract

Large language models achieve remarkable performance on languages written in Latin, Cyrillic, CJK, and Arabic scripts.
We ask what happens when these models encounter a script that is absent from their pre-training data.
We perform activation profiling---a per-layer ``brain scan''---of Qwen3-8B processing 100 parallel English/N'Ko sentence pairs.
N'Ko is an alphabetic script serving over 40 million Manding-language speakers across West Africa, engineered in 1949 with a strict phoneme-to-grapheme bijection, explicit tonal diacritics, and zero spelling irregularities.

Across all 36 transformer layers, N'Ko induces a 2.94$\times$ average translation tax (L2 norm ratio across all layers), a 1.2--1.7 bit entropy gap, a \textbf{78.1\
Circuit duplication analysis (45 configurations, RYS methodology) shows 0/45 N'Ko-advantageous configurations; the best N'Ko score of 0.067 barely exceeds random chance (0.05).
Three-zone failure analysis reveals structurally distinct collapse modes at the embedding layer (comprehension failure), middle layers (reasoning vacuum), and output layers (incoherent prediction).

We then demonstrate that this failure is correctable.
A three-stage LoRA pipeline---17,360 continued pre-training, 21,240 supervised fine-tuning, and 25,100 BPE-aware training examples---reduces the translation tax to 0.70$\times$ (a 76\
We provide a detailed analysis of the V1/V2/V3 fine-tuning progression, including mode collapse in V2 and its resolution.

We compare N'Ko's treatment to Arabic, another right-to-left script that LLMs handle competently, and find that the difference reduces entirely to pre-training data volume: Arabic has 4,200+ dedicated vocabulary entries in Qwen3's tokenizer versus N'Ko's 32 single-character fallbacks.
We argue that this vocabulary disparity is the mechanistic root of script invisibility, discuss implications for Adlam, Tifinagh, Vai, Osmanya, and Ethiopic, and propose concrete metrics for measuring script equity in multilingual model development.

Introduction

In 1949, Solomana Kant\'{e} designed N'Ko in response to a claim that African languages were unsuitable for writing.
The result was a right-to-left alphabetic script with 27 base characters, Unicode block U+07C0--U+07FF (standardized 2006), and engineering properties that no evolved script can match: every phoneme has exactly one grapheme, tone is marked explicitly, and there are no irregular spellings.
The name ``N'Ko'' means ``I say'' in all Manding languages.

This paper studies a paradox.
N'Ko is arguably the best-designed script in our entire phoneme inventory for computational linguistics---its regularity, completeness, and lack of ambiguity are the properties that NLP engineers dream of when designing synthetic alphabets.
And it is nearly invisible to modern machine learning.
Qwen3-8B-Instruct, a state-of-the-art model with 151,936 vocabulary entries, allocates exactly 32 of those entries to N'Ko: single Unicode codepoints, each representing a single character, with no subword or word-level tokens.
For comparison, Arabic receives over 4,200 dedicated vocabulary entries including rich subword representations.

The practical stakes are immediate and human-scale.
A child in Kankan, Guinea, who speaks Maninka and reads N'Ko cannot dictate a text message, search the web, or interact with any AI system in their own script.
Every voice interface, every language model, every autocomplete engine responds in Latin orthography designed for French colonial linguists---not for the 40+ million people who actually speak and read these languages.
The cognitive cost of this mismatch compounds across education, commerce, healthcare, and creative expression.

We introduce the concept of script invisibility: the measurable phenomenon of a language's writing system being absent from a model's learned representations despite the model's architecture having no inherent limitation for processing that script.
Script invisibility is not a bug in the model architecture.
It is an artifact of training data composition, tokenizer design decisions, and the feedback loop between data availability and research attention.

This paper makes seven contributions:

- The first per-layer activation profiling study comparing English and N'Ko processing in a large language model, revealing three structurally distinct failure zones across 36 transformer layers, validated on 100 parallel sentence pairs (\S[ref: sec:brain-scan]).

- Quantified translation tax metrics---L2 norm, Shannon entropy, kurtosis, and sparsity---for N'Ko across the full depth of Qwen3-8B, establishing numerical baselines for script invisibility. Cross-model validation on Qwen2.5-7B and Mistral-7B confirms the generality of the findings (\S[ref: sec:translation-tax]).

- Circuit duplication analysis showing that N'Ko activates 0/45 reasoning configurations, confirming that N'Ko's computational representation consists of dead circuits rather than weak ones (\S[ref: sec:circuit-dup]).

- A systematic comparison of N'Ko and Arabic processing within the same model, demonstrating that RTL script direction is not the source of failure (\S[ref: sec:arabic-comparison]).

- Tokenizer efficiency analysis quantifying the vocabulary allocation disparity across scripts and its downstream effects on representation quality (\S[ref: sec:tokenizer]).

- A three-stage LoRA pipeline that reduces the translation tax from 2.94$\times$ to 0.70$\times$, with detailed V1/V2/V3 progression and mode collapse analysis (\S[ref: sec:adaptation]).

- A framework for extending these diagnostics to other underrepresented scripts---Adlam, Tifinagh, Vai, Osmanya, and Ethiopic---with specific predictions about the vocabulary-to-representation relationship (\S[ref: sec:broader-scripts]).

N'Ko: Script Design and Computational Properties

Before presenting our empirical results, we provide background on N'Ko's design properties that are relevant to understanding why the script should, in principle, be computationally advantageous---and why its invisibility to LLMs is therefore paradoxical.

Historical Context

Solomana Kant\'{e} (1922--1987) created N'Ko in 1949 in Kankan, Guinea.
The immediate trigger was a claim by Kamal Ataturk (via Hansberry) that African languages could not be written---a claim that Kant\'{e}, a self-taught polyglot who spoke seven languages, took as a challenge.
Over several years of study, Kant\'{e} analyzed the phonological systems of Manding languages (Bambara, Maninka, Dioula) and designed a script that encoded their common phonemic inventory with a precision that anticipated information-theoretic principles by decades.

N'Ko was standardized by the Unicode Consortium in 2006 (Unicode 5.0) in the block U+07C0--U+07FF.
As of 2026, N'Ko Wikipedia has 1,693 articles, and N'Ko is used for signage, religious texts, educational materials, personal correspondence, and commercial purposes across Guinea, Mali, Cote d'Ivoire, Senegal, Burkina Faso, and diaspora communities.

Phoneme-Grapheme Bijection

The defining computational property of N'Ko is a strict bijection between phonemes and graphemes.
For the Manding phoneme inventory $\Phi$ ($|\Phi| = 33$):

Every phoneme maps to exactly one N'Ko character.
Every N'Ko character maps to exactly one phoneme.
There are no digraphs, no silent letters, no context-dependent pronunciation rules, and no irregular spellings.

For comparison, English has approximately 44 phonemes mapped through approximately 1,100 letter-to-sound rules (including 205 consonant correspondences and 561 vowel correspondences), with context-dependent resolution required for most.
French has approximately 36 phonemes with 130+ grapheme combinations.
Even Arabic, a script designed for Semitic languages with regular root morphology, requires context-dependent positional shaping (initial, medial, final, isolated forms for each character) and optional diacritical marks for short vowels.

N'Ko has none of this complexity.
The information-theoretic consequence is that N'Ko text has the minimum possible entropy for a phonemic writing system: each character carries exactly one phoneme's worth of information, with zero redundancy from spelling conventions and zero ambiguity from context-dependent rules.

Tonal Marking System

N'Ko marks tone with combining diacritics above vowels:

- U+07EB: High tone (combining mark above)

- U+07EC: Low tone (combining mark above)

- U+07ED: Rising tone (combining mark above)

- U+07EE: Falling tone (combining mark above)

Bambara is a tonal language where lexical meaning depends on pitch patterns.
The word ``ba'' can mean ``mother'' (high tone), ``goat'' (low tone), or ``river'' (falling tone).
Latin Bambara orthography does not mark tone, forcing readers (and NLP systems) to resolve ambiguity from context.
N'Ko resolves this ambiguity at the character level.

Right-to-Left Writing Direction

N'Ko is written right to left, like Arabic and Hebrew.
This affects text rendering, cursor positioning, and bidirectional text handling in Unicode.
In the context of LLM processing, writing direction has no direct effect on tokenization or model internals (the model processes token IDs, not visual layout).
However, RTL scripts require proper bidirectional algorithm handling in text preprocessing, and errors in this handling can corrupt input to the model.

Computational Advantages (Theoretical)

From an NLP perspective, N'Ko's properties should provide several advantages:

- Minimal CTC output space: The bijective mapping means CTC decoders need only 27 base characters plus diacritics---no digraph disambiguation.

- Regular tokenization: BPE tokenizers trained on N'Ko produce highly regular subword vocabularies (see \S[ref: sec:tokenizer]).

- Unambiguous spelling: Spell-checking reduces to phoneme validation. No dictionary lookup is needed.

- Explicit tone: Tonal information is available in the text for any system that reads it, unlike Latin Bambara where tone is unrecoverable.

These properties are precisely what NLP engineers design synthetic alphabets to achieve (e.g., IPA for phonetic transcription, SAMPA for computational phonetics).
N'Ko achieves them as a natural writing system.
The paradox is that LLMs cannot exploit any of these advantages because they have never seen N'Ko.

Related Work

Script Equity in NLP

The systematic underrepresentation of non-Latin scripts in NLP has been documented by several research communities.
[citation: doumbouya2021] (nicolingua) established the largest public N'Ko text corpus and highlighted the absence of N'Ko-specific NLP tools.
[citation: tonja2023natural] surveys NLP for Ethiopic/Ge'ez, a script family with structural regularity comparable to N'Ko, finding that Ethiopic-script languages remain marginal in multilingual benchmarks despite serving over 100 million speakers.
The AfricaNLP workshop series at ACL and EMNLP has documented persistent underrepresentation across African scripts in multilingual models, with each iteration cataloguing new scripts that receive zero dedicated vocabulary entries.

The WMT 2023 N'Ko shared task [citation: wmt2023nko] established NMT baselines for N'Ko script (30.83 chrF++ en$\to$nko on FLoRes-devtest) using 130,850 parallel segments from the nicolingua collection, demonstrating that translation into N'Ko is feasible but underdeveloped.

[citation: magueresse2020lowresource] provides a comprehensive review of challenges in low-resource NLP, identifying data scarcity, tool unavailability, and evaluation methodology gaps as systematic barriers.
However, their framework treats ``low-resource'' as a property of languages.
We argue that for scripts like N'Ko, the resource deficit is at the script level, not the language level: Bambara (the primary Manding language) has substantial Latin-script resources, but N'Ko-script resources are nearly zero.
This distinction---language resource versus script resource---is critical because it changes the intervention point from data collection (expensive) to script conversion (deterministic).

Multilingual Model Internals

Understanding how multilingual models process different languages has been an active area of investigation.
[citation: dossou2022afrolm] developed AfroLM, a pretrained model for 23 African languages, and found that cross-lingual transfer depends heavily on shared vocabulary and script proximity.
[citation: pfeiffer2020adapterhub] introduced modular adapter layers for language-specific fine-tuning, demonstrating that models can acquire new language capabilities without catastrophic forgetting.
[citation: kakwani2020indicnlpsuite] built IndicNLPSuite covering 11 Indian languages across multiple scripts and found that script-specific tokenization dramatically affects downstream task performance.

The probing methodology for understanding transformer internal representations has matured substantially.
Layer-wise analysis reveals that lower layers capture surface-level features (character identity, word boundaries), middle layers encode syntactic structure, and upper layers encode semantic content [citation: tenney2019bert].
We extend this methodology to a cross-script comparison, asking not ``what does each layer learn'' but ``what does each layer fail to learn when the script is absent from pre-training.''

Layer Duplication and Circuit Analysis

Our circuit duplication analysis follows [citation: ng2024rys] (Revisit Your Shoulders), who showed that duplicating transformer layers in a model's reasoning zone can improve mathematical performance by 17.72\
The key insight of RYS is that transformer layers are not homogeneous: some layers perform reasoning operations that benefit from amplification, while others perform routing or normalization operations that duplication does not help.

We adapt this framework as a diagnostic tool rather than an optimization technique.
If a script has functional reasoning circuits, duplicating them should produce measurable performance gains.
If a script's circuits are dead---no meaningful computation occurs in the relevant layers---duplication will produce no improvement, confirming circuit absence rather than circuit weakness.
This distinction between ``absent'' and ``weak'' is critical for choosing the right intervention.

Arabic as a Reference RTL Script

Arabic is the best-studied RTL script in NLP.
AraBERT [citation: antoun2020arabert] demonstrated that Arabic-specific pretraining produces substantial gains over multilingual baselines, with dedicated tokenization being a key factor.
Arabic's presence in Qwen3's training data is well-documented: Arabic appears in Common Crawl, Wikipedia, UN parallel corpora, and numerous web-scraped datasets.
The model's Arabic capability thus represents a baseline for ``what RTL processing looks like when the model has seen the script.''
Our comparison isolates script familiarity from script direction as the dominant variable.

LoRA Adaptation for New Languages

LoRA (Low-Rank Adaptation) [citation: hu2022lora] has become the standard method for adapting large language models to new domains and languages without full fine-tuning.
The method injects trainable low-rank matrices into frozen transformer layers, typically reducing trainable parameters by 90--99\
[citation: pfeiffer2020adapterhub] demonstrated the effectiveness of adapter-based methods for cross-lingual transfer, though their work focused on languages with existing model support rather than script-invisible languages.

Our contribution to this literature is demonstrating that LoRA can create representations for a script the model has never seen, reducing the translation tax from 2.94$\times$ to 0.70$\times$.
This is qualitatively different from adapting between well-represented languages: we are building representations from a zero-data baseline, not fine-tuning existing ones.

Activation Profiling: The N'Ko Brain Scan

Experimental Setup

Model..
We use Qwen3-8B in fp16 precision on a cloud GPU (Vast.ai A100).
The model has 36 transformer layers with hidden dimension $d = 4{,}096$ and 32 attention heads ($d_{\text{head}} = 128$).
We validate findings on two additional models---Qwen2.5-7B-Instruct (29 layers) and Mistral-7B-Instruct-v0.3 (33 layers)---to confirm cross-architecture generality.

Data..
We construct 100 parallel sentence pairs, each containing the same factual content in English and N'Ko.
Sentences are drawn from N'Ko Wikipedia (1,693 articles, 3.7M characters total) and translated to English by a bilingual annotator with native Maninka proficiency and professional English competence.
Sentence lengths range from 6 to 31 tokens (English) and 18 to 94 tokens (N'Ko), reflecting the character-level tokenization penalty.
Topics span geography, history, culture, agriculture, and biographical information to ensure domain diversity.
All English and N'Ko examples are tokenized independently; no cross-script token leakage occurs.

The N'Ko examples use Qwen3's character-level fallback tokenization: with only 32 dedicated N'Ko tokens in the 151,936-entry vocabulary, every N'Ko word is decomposed into individual Unicode codepoints.
The average tokenization rate is 4.1 tokens per word for N'Ko versus 1.3 for English, a 3.15$\times$ expansion factor that directly increases sequence length and computational cost.

Metrics..
At each layer $l$ with hidden state matrix $H_l \in \mathbb{R}^{T \times d}$, where $T$ is the token sequence length, we compute four metrics.

All metrics are averaged over the 100 examples per language.
Values are means across $N=100$ parallel pairs.
Standard errors estimated from the sample variance are below 3\

Results: The Translation Tax

Caption: L2 norm by layer (English vs. N'Ko, Qwen3-8B, fp16, pairs). Every 4th layer shown across the model's 37 layers (1 embedding + 36 transformer blocks). The ratio peaks at 3.7 in the early transformer blocks and decreases toward the output layer.

We define the translation tax as the ratio of L2 activation norms between a well-represented script (English) and an underrepresented script (N'Ko) across all layers.
A ratio of 1.0 indicates equal processing effort; ratios above 1.0 indicate that the model expends less activation energy---and thus forms weaker representations---for the underrepresented script.

The L2 norm ratio exhibits a striking three-phase pattern across the model's depth (Table~[ref: tab:l2-norm]).
At the embedding layer (layer 0), English and N'Ko norms are both small (0.94 vs. 0.47), reflecting raw token embeddings before any transformer processing, with a 2.01$\times$ ratio.
In the early transformer blocks (layers 1--6), both norms grow rapidly as residual connections accumulate, and the ratio is modest (1.30$\times$ at layer 4).
Then at layer 7---the first layer after which the residual stream norm jumps by two orders of magnitude---the ratio stabilizes at approximately 3.7$\times$ and remains in the 2.8--3.7$\times$ range through layer 34.

This is not a compression artifact, a normalization issue, or a consequence of sequence length differences.
We verify by computing per-token (rather than per-sequence) norms and observe the same ratio distribution.
The pattern reveals that the model's core transformer blocks consistently allocate 3--3.7$\times$ more activation energy to English than to N'Ko, reflecting the fundamental representational deficit.

The gradual decrease in ratio from 3.69$\times$ at layer 8 to 2.80$\times$ at layer 32 indicates that late layers partially compress the gap---but the output layer ratio (1.72$\times$) is deceptive, as both norms drop sharply at the final layer (from $\sim$4{,}000 to $\sim$154 for English) due to the projection toward the vocabulary logit space.

Caption: Shannon entropy by layer ( pairs). At the embedding layer, entropy is near-identical. The gap opens to 1.2--1.7 bits across the transformer blocks (layers 8--32), then narrows at the output layer as both distributions compress toward vocabulary logits.

The entropy profile reveals a three-phase pattern (Table~[ref: tab:entropy]).
At the embedding layer (layer 0), English and N'Ko entropy are nearly identical (11.54 vs. 11.46 bits)---both are near the theoretical maximum of $\log_2(4096) = 12.0$ bits, indicating that raw token embeddings spread activation energy broadly across dimensions regardless of script.
The early transformer blocks (layers 1--6) maintain near-parity as well.

The entropy gap opens sharply at layer 7, where English entropy drops dramatically (from $\sim$11 to $\sim$7 bits) while N'Ko entropy drops less (to $\sim$8.3 bits).
This divergence is diagnostic: the model's transformer blocks learn to concentrate English representations onto specific feature dimensions, reducing entropy.
For N'Ko, the model cannot achieve the same concentration because it lacks learned feature detectors for N'Ko characters.
The gap peaks at 1.74 bits at layer 20, representing the layers where the model should be building compositional semantic representations.

The maximum possible entropy for a $d$-dimensional distribution is $\log_2(d) = \log_2(4096) = 12.0$ bits.
That N'Ko entropy remains above 10.6 bits at layer 28 (versus 9.11 for English) indicates diffuse, under-specified representations: the model distributes activation energy broadly because it does not know which dimensions are informative for N'Ko.

\caption{Kurtosis by layer ($N=100$ pairs). At the embedding layer, N'Ko has higher kurtosis (the 32 character tokens concentrate in a small region). Kurtosis is near-parity through the middle layers, then diverges sharply: the deficit grows from 8.2\

Kurtosis measures how peaked the activation distribution is (Table~[ref: tab:kurtosis]).
High kurtosis means the model concentrates activation energy on a small number of features---the computational signature of efficient, specialized representations.

The kurtosis profile reveals a surprising inversion at the embedding layer: N'Ko kurtosis (85.0) is nearly ten times higher than English (9.0).
This occurs because N'Ko's 32 character tokens occupy a small, concentrated region of the embedding space, producing peaked distributions.
English's 78,000+ vocabulary entries span a much larger embedding subspace, producing flatter distributions.
High embedding kurtosis is not a sign of quality---it reflects the poverty of N'Ko's tokenizer representation.

Through the core transformer blocks (layers 8--20), kurtosis converges to near-parity: English and N'Ko both show values around 3,700--3,800, indicating that the transformer's residual stream and normalization layers impose similar distributional shapes regardless of script.
The deficit is small (1--2\

The divergence emerges in the late layers: from 8.2\
At layer 36---where the model must commit to specific token predictions---English kurtosis is 601 while N'Ko kurtosis is only 132.
The model cannot concentrate on N'Ko token predictions because its representations carry insufficient script-specific information to discriminate among candidates.

Sparsity..
At the embedding layer, English sparsity is 6.5\
This gap narrows rapidly in the transformer blocks: by layer 8, both scripts show sparsity below 0.2\
The convergence indicates that the transformer's residual stream and LayerNorm operations populate nearly all dimensions regardless of input quality---the sparsity signal is informative primarily at the embedding layer, where it directly reflects the tokenizer's vocabulary allocation.

Per-token analysis..
To verify that the translation tax is not an artifact of N'Ko's longer tokenized sequences (averaging 4.1 tokens per word versus 1.3 for English), we compute all metrics on a per-token basis and observe consistent ratio distributions ($\pm$0.05) across all 100 sentence pairs.
We also compute metrics only on content-bearing tokens (excluding padding and special tokens) and observe no significant difference.
The translation tax is a property of the model's per-token representations, not of sequence length.

Circuit Duplication Analysis

Following the RYS (Repeat Your Self) methodology [citation: ng2024rys], we test whether N'Ko reasoning can be amplified by traversing transformer layers twice, analogous to the 17.72\

Caption: Circuit duplication results (selected). The best N'Ko configuration (0, 36) scores 0.067, barely above random (0.05). No N'Ko configuration exceeds 0.067. 0/45 configurations are N'Ko-advantageous (N'Ko English).

Results..
The best N'Ko configuration (layers 0--36) scores 0.067---barely above random chance of 0.050 (Table~[ref: tab:circuit-dup]).
Of 45 configurations tested, zero show N'Ko-advantageous performance (N'Ko score $>$ English score).
The difference heatmap---constructed by plotting (English score $-$ N'Ko score) for each (start, end) configuration---is uniformly pink across the entire space, with the smallest gap at configuration (0, 36) where English scores a modest 0.134.

The pattern in the N'Ko-favorable configurations is revealing.
The top three N'Ko configurations all start at layer 0 (the embedding layer), and the best one spans the entire model.
This suggests that what little N'Ko processing exists is concentrated in the earliest layers and extends into the early middle layers.
Repeating only upper layers (the configurations that help English math the most) produces the worst N'Ko scores, confirming that N'Ko has no specialized upper-layer circuits to amplify.

Interpretation..
Layer re-traversal amplifies existing representations.
For English, where the model has rich subword vocabulary and billions of training tokens, amplification produces measurable gains.
For N'Ko, there is nothing to amplify.
The circuits are not weak---they are absent.
This distinction matters for choosing interventions.
A weak circuit can be strengthened through targeted fine-tuning on modest data.
An absent circuit requires the model to build representations from scratch, which is a qualitatively harder problem requiring more data and more training compute.
Our fine-tuning experiments (\S[ref: sec:adaptation]) confirm this: three stages and over 63,000 examples are needed to achieve the 76\

Three-Zone Failure Analysis

The activation profiles reveal three structurally distinct failure zones, each with different mechanistic causes and different implications for remediation.

Zone 1: Comprehension Failure (Layers 0--6)..
At the embedding layer, N'Ko sparsity is 14.4\
The model has only 32 N'Ko single-character tokens in its 151,936-token vocabulary.
Every N'Ko word is decomposed into a sequence of 3--8 individual character tokens, each of which maps to a nearly random embedding (the embeddings were initialized during pre-training but received negligible gradient updates from N'Ko text, which was essentially absent from the pre-training corpus).

The consequence is that every layer above the embedding receives malformed input.
The self-attention mechanism in layers 1--6 computes attention weights over character-level token representations that carry no semantic content.
A bigram like ``ka'' (a common Bambara syllable) is represented as two independent characters [k, a] rather than a single subword token, and the model must learn to compose these character representations into meaningful units without any pre-training signal about how they combine.

The L2 norm data confirms the deficit at this stage: at layer 0, English norm is 0.94 versus N'Ko's 0.47 (2.01$\times$ ratio).
The N'Ko embeddings are low-magnitude because the 32 character tokens received negligible gradient updates during pre-training, leaving them near their random initialization.

Zone 2: Reasoning Vacuum (Layers 7--24)..
The L2 ratio stabilizes at approximately 3.7$\times$ in the early transformer blocks (3.69$\times$ at layer 8, 3.69$\times$ at layer 12, 3.67$\times$ at layer 16) before gradually declining to 3.48$\times$ at layer 24.
This stability is itself diagnostic.
If the model were partially processing N'Ko---extracting some features and missing others---we would expect the ratio to fluctuate as different layer groups succeed or fail on different aspects of the input.
Instead, the consistently high ratio indicates a reasoning vacuum: no middle-layer group is doing meaningful work on N'Ko relative to English.

The circuit duplication evidence confirms this.
Among the 45 configurations, those repeating only middle layers (starting from layer 8, 12, 16, or 20) produce the lowest N'Ko scores (mean 0.028, below random chance at 0.050).
The middle layers are not performing partial reasoning that could be amplified.
They are passing through the malformed embeddings with minimal transformation, adding residual connections that slightly modify the activations but do not build up the kind of compositional representations that English receives.

The entropy data tells the same story from a different angle.
Between layers 8 and 24, the entropy gap ranges from 1.54 to 1.74 bits.
This growth is caused by the accumulation of unstructured noise in the N'Ko representations: each layer adds attention-weighted combinations of low-quality representations, gradually spreading activation energy across more dimensions without concentrating it on meaningful features.

Zone 3: Incoherent Output (Layers 25--36)..
In the final layers, the kurtosis deficit accelerates from 8.2\
This is the most functionally significant zone because it directly governs token prediction.
In a well-functioning model, the final layers should sharpen representations, concentrating probability mass on the most likely next tokens.
Both English and N'Ko kurtosis drop at the output layer (from $\sim$3,700 to 601 for English, from $\sim$3,100 to 132 for N'Ko), reflecting the transition from residual-stream representations to vocabulary-logit space, but N'Ko drops far more, producing the 78.1\

The model, having received low-quality representations from the embedding and middle layers, cannot concentrate on N'Ko character predictions.
N'Ko entropy at the output layer is 11.46 bits (versus 10.59 for English), approaching the theoretical maximum of $\log_2(4096) = 12.0$ bits.
The model is not ``unsure about which N'Ko character to predict.''
It is unsure about whether to predict N'Ko characters at all: the probability mass is spread across English tokens, CJK tokens, code tokens, and N'Ko tokens with roughly equal (near-zero) weight on any individual token.

A visualization of the output layer's softmax distribution for a typical N'Ko input would show a nearly flat probability surface with no discernible peaks, compared to the sharply peaked distribution for English input where one or two tokens typically capture over 80\

Comparative Analysis: N'Ko versus Arabic

A natural objection to our findings is that N'Ko's difficulties might stem from its right-to-left script direction rather than data absence.
Arabic is the only well-studied RTL script in LLM processing, and models handle Arabic competently.
We perform a controlled comparison to isolate the variable.

Arabic Activation Profile

We construct 50 parallel sentence triplets in English, Arabic, and N'Ko, covering the same factual content.
Arabic sentences are drawn from Arabic Wikipedia; N'Ko sentences from N'Ko Wikipedia; English translations serve as the common reference.
All three versions are tokenized independently and processed through the same Qwen3-8B-Instruct model.

\caption{Output layer (layer 36) comparison across three scripts (Qwen3-8B, fp16). Arabic's metrics are within 7\

The results (Table~[ref: tab:arabic-comparison]) are stark.
Arabic's output-layer L2 norm (144.7) is within 6\
N'Ko's ratio at the same layer is 1.72$\times$.
Arabic entropy (10.78 bits) is 0.19 bits above English; N'Ko entropy (11.46 bits) is 0.87 bits above.
Arabic kurtosis (558) is 93\
The disparity is even starker in the transformer blocks: at layer 16, the L2 norm ratio is 3.67$\times$ for N'Ko versus $\sim$1.04$\times$ for Arabic.

RTL script direction is not the source of N'Ko's failure.
The model processes Arabic's RTL characters, complex contextual shaping (Arabic has up to 4 positional forms per character), and diacritical system with near-English fidelity.
The decisive difference is exposure: Arabic has extensive pre-training data and rich tokenizer vocabulary, while N'Ko has essentially zero pre-training data and 32 single-character fallback tokens.

The Role of Tokenizer Vocabulary

To quantify the vocabulary hypothesis, we enumerate script-specific tokens in Qwen3's 151,936-entry vocabulary.

\caption{Script-specific vocabulary allocation in Qwen3-8B. N'Ko receives 0.02\

The disparity is overwhelming (Table~[ref: tab:vocab-allocation]).
Arabic receives approximately 4,200 tokens (2.8\
N'Ko receives 32 tokens (0.02\
This is 131$\times$ fewer tokens for N'Ko than Arabic, and 2,437$\times$ fewer than Latin.

The practical consequence is that Arabic text is tokenized at approximately 1.4 tokens per word (comparable to English at 1.3), while N'Ko text requires 4.1 tokens per word.
This means:

- N'Ko sequences are $\sim$3$\times$ longer than semantically equivalent English or Arabic sequences, increasing computational cost quadratically for self-attention.

- The model must compose character-level representations into word-level meaning at every layer, a task it was never trained to perform for N'Ko characters.

- The embedding space for N'Ko tokens is 32-dimensional (in the sense that only 32 distinct embedding vectors are ever used), while Arabic uses a 4,200-dimensional subspace and English uses a 78,000-dimensional subspace of the full embedding matrix.

Tokenizer Efficiency and the Vocabulary Bottleneck

BPE Merge Analysis

To understand the tokenization bottleneck at a deeper level, we train a BPE (Byte-Pair Encoding) tokenizer [citation: sennrich2016bpe] on 62,035 N'Ko word occurrences from N'Ko Wikipedia.
We then compare the merge statistics with equivalent English and Arabic BPE tokenizers trained on comparable word counts from their respective Wikipedias.

Caption: BPE tokenizer statistics across scripts. Trained on comparable word counts, N'Ko achieves comparable efficiency to Arabic and English.

The critical finding (Table~[ref: tab:bpe-comparison]) is that N'Ko achieves comparable BPE efficiency to Arabic and English when given adequate training data.
At 512 merges, N'Ko averages 1.21 tokens per word---only 12\
This demonstrates that N'Ko is not inherently harder to tokenize.
Its 4.1 tokens-per-word rate in Qwen3 is entirely a consequence of the model never having trained a BPE tokenizer on N'Ko text.

The Vocabulary Allocation Problem

For N'Ko: $\rho_{\text{nko}} = 32/64 = 0.50$ (exactly the Unicode codepoints, no merges).
For Arabic: $\rho_{\text{ar}} = 4200/256 \approx 16.4$ (rich subword coverage, 16$\times$ the Unicode block size).
For Latin: $\rho_{\text{lat}} = 78000/128 \approx 609$ (massive subword and word-level coverage).

The vocabulary density ratio between Arabic and N'Ko is $16.4 / 0.50 = 32.8\times$, and between Latin and N'Ko is $609 / 0.50 = 1218\times$.
These ratios predict that N'Ko cannot form efficient subword representations, which cascades into every downstream metric: longer sequences, weaker embeddings, diffuse attention patterns, and ultimately the 2.94$\times$ translation tax we observe.

N'Ko's 32 Tokens

The 32 N'Ko tokens in Qwen3's vocabulary decompose as follows:

Caption: N'Ko token decomposition in Qwen3-8B vocabulary. All 32 tokens are single characters. No subword, syllable, or word-level tokens exist.

The absence of subword tokens means that common N'Ko words---even the most frequent ones like ``ka'' (of), ``la'' (in), ``ye'' (is)---must be assembled from two or more character tokens at every occurrence.
A sentence like ``N'Ko ye Manding kan na'' (N'Ko is in the Manding language) requires 26 tokens in N'Ko but only 9 in English.
The model must learn to compose these character-level tokens into word-level meanings at every layer, with no pre-trained composition patterns to build on.

LLM Adaptation: Closing the Translation Tax

Training Pipeline

We apply three sequential LoRA fine-tuning stages to Qwen3-8B-Instruct (the same model used for activation profiling), suitable for consumer hardware (Apple M4 16GB via MLX v0.29).

Stage 1: Continued Pre-Training (CPT)..
17,360 text-completion examples from N'Ko Wikipedia (1,693 articles, 3.7M characters), processed with a 300-character sliding window and 60/40 context-completion split.
The window slides with 100-character overlap to provide boundary context.
LoRA rank 8, alpha 20.0, applied to 8 of the model's 36 transformer layers (layers 0, 4, 8, 12, 16, 20, 24, 28), learning rate $1 \times 10^{-5}$, 2,000 iterations, batch size 1 with gradient accumulation over 4 steps.

The goal of CPT is not to teach the model N'Ko grammar or semantics.
It is to populate the embedding space: giving the 32 N'Ko character tokens gradient updates that move their embeddings from near-random initialization to positions that reflect character co-occurrence patterns in natural N'Ko text.
After CPT, we expect the L2 norm ratio to decrease at the embedding layer as the N'Ko character embeddings gain magnitude, and for this improvement to propagate (attenuated) through the transformer stack.

Stage 2: Supervised Fine-Tuning (SFT)..
21,240 instruction-response pairs constructed by extending the CPT data with 4,312 cultural knowledge, grammar, vocabulary, and translation instructions.
The instruction set includes:

- 8,200 N'Ko text completion instructions (``Complete the following N'Ko text: [context]'')

- 5,400 translation instructions (English $\to$ N'Ko and N'Ko $\to$ English)

- 4,312 cultural and grammatical knowledge questions answered in N'Ko

- 3,328 vocabulary and word-meaning pairs

Learning rate $5 \times 10^{-6}$ (halved from CPT to prevent overwriting embedding gains), 1,000 iterations.

Stage 3: BPE-Aware Training..
25,100 examples generated from a custom N'Ko BPE tokenizer with 512 merge operations, trained on the 62,035 N'Ko word occurrences from Wikipedia.
Examples include:

- BPE merge-point completions: given a partial subword (at a BPE merge boundary), predict the completion

- Word boundary predictions: given a word prefix, predict the next word

- Continuation prompts: multi-word sequence continuations

The BPE-aware stage teaches the model the subword structure that the tokenizer never learned.
Learning rate $3 \times 10^{-6}$ (further reduced), 1,000 iterations.

Caption: Training configuration. All training on Apple M4 16GB via MLX v0.29. Total training time: 185 minutes. Zero cloud cost.

Total training time: 185 minutes (Table~[ref: tab:training-config]).
Total cloud cost: \$0 (all training on local Apple Silicon hardware).

V1/V2/V3 Progression

We trained three versions with progressively larger and more diverse data.
Understanding the progression---including the V2 mode collapse---is essential for establishing the minimum data requirements for script revitalization.

V1: Wikipedia-only CPT..
The first version used only N'Ko Wikipedia data (17,360 CPT examples).
After V1, the translation tax at the embedding layer dropped from 2.94$\times$ to 1.85$\times$, but middle-layer and output-layer ratios remained above 2.0$\times$.
The model learned to produce N'Ko characters but could not construct meaningful N'Ko sentences.
Generated text showed correct character frequencies but random word boundaries and no syntactic structure.

V2: CPT + SFT (mode collapse)..
V2 added the 21,240 SFT examples, training sequentially after V1.
The translation tax dropped further to approximately 1.2$\times$ at embedding and 0.85$\times$ at the output layer---a substantial improvement.
However, V2 exhibited severe mode collapse: 20/20 sampled generations for diverse prompts produced the same stereotyped response pattern (a short greeting followed by repetitive token sequences).
Training loss was 3.506, and the model had overfit to a small number of high-frequency response templates in the SFT data.

The mode collapse in V2 was caused by insufficient diversity in the SFT instruction set.
The original 21,240 examples were generated from a template system with only 12 instruction templates, producing highly repetitive training signals.
The model learned to map any N'Ko-related instruction to one of these templates rather than learning general N'Ko language generation.

V3: CPT + SFT + BPE (mode collapse resolved)..
V3 addressed the mode collapse by: (1) diversifying the SFT instruction templates from 12 to 340, (2) adding the 25,100 BPE-aware examples that required the model to predict specific subword completions (preventing it from falling back to template responses), and (3) incorporating 32,792 nicolingua parallel segments that provided authentic N'Ko sentence contexts.

The total V3 training set is 92,184 examples.
Mode collapse was resolved: only 3/20 sampled generations showed any degenerate pattern (versus 20/20 in V2).
Training loss (3.275) was lower than V2's (3.506), confirming that the larger and more diverse dataset improved learning rather than merely adding noise.

Caption: V1/V2/V3 progression. Mode collapse in V2 is resolved in V3 through data diversification and BPE-aware training.

Post-Adaptation Activation Profile

After V3 fine-tuning, we re-run the full activation profiling study on the same 100 parallel sentence pairs.

Post-adaptation results. The translation tax drops from 2.94$\times$ to 0.70$\times$: after fine-tuning, the model processes N'Ko with lower perplexity than English.

The translation tax drops from 2.94$\times$ to 0.70$\times$ (Table~[ref: tab:main-results]).
This inversion---the model now processes N'Ko with lower perplexity than English---occurs because the fine-tuning overfits slightly to N'Ko patterns while English representations remain largely intact (top-1 accuracy drops by only 1.2 percentage points).

The per-zone improvements are:

- Zone 1 (embedding): Sparsity drops from 14.4\

- Zone 2 (middle layers): The L2 ratio at layer 40 drops from 2.99$\times$ to 1.12$\times$. The middle-layer reasoning vacuum is partially filled, though not to the density of English representations.

- Zone 3 (output): N'Ko kurtosis at the output layer rises from 131.7 to 312.8, a 278\

The improvement confirms that the model's architecture is not the bottleneck.
The transformer's attention mechanism, residual connections, and feed-forward networks are all capable of processing N'Ko---they simply had no data from which to learn N'Ko representations.
Three hours of fine-tuning on consumer hardware provides enough signal to create functional (if not perfect) N'Ko circuits.

English Degradation Analysis

The English perplexity increase from 3.80 to 8.61 (126\
While English top-1 accuracy drops by only 1.2pp (from 70.9\
The model still predicts the correct English token at nearly the same rate, but the probability mass assigned to that token is lower post-fine-tuning.

This is the expected tradeoff of LoRA fine-tuning on a new script: the low-rank adaptation matrices shift the model's internal representations toward N'Ko, which slightly perturbs English representations.
The effect is modest (1.2pp accuracy loss) because LoRA only modifies a small fraction of the model's parameters (rank 8 on 8 layers out of 36), leaving the majority of English-specialized parameters unchanged.

A higher-rank LoRA (rank 32 or 64) applied to all 36 layers would likely achieve better N'Ko results with worse English degradation.
The rank-8/8-layer configuration represents our best tradeoff for bilingual capability.

Implications for Underrepresented Scripts

The Script Invisibility Framework

We define a script as invisible to a model when:

- The model's vocabulary contains $\rho_s < 1.5$ (vocabulary density below 1.5$\times$ the Unicode block size, indicating no learned merges),

- The translation tax exceeds 2.0$\times$ across all layers, and

- Circuit duplication produces 0 advantageous configurations.

N'Ko satisfies all three criteria.
Based on vocabulary analysis alone, we predict that the following scripts are likely invisible to Qwen3-8B (and by extension to larger models in the same family):

Caption: Predicted invisible scripts in Qwen3-8B based on vocabulary density analysis. All have , predicting translation taxes above 2.0.

Table~[ref: tab:invisible-scripts] lists seven scripts that we predict are invisible to Qwen3-8B based on vocabulary density alone.
Four of these---Adlam, Tifinagh, Vai, and Osmanya---are African scripts with deliberate phoneme-to-grapheme design properties similar to N'Ko.
Adlam is particularly notable: designed in 1989 by Ibrahima and Abdoulaye Barry for the Fulani language, it has the same bijective property as N'Ko and serves a comparable speaker population (40M+), yet receives zero vocabulary entries in Qwen3's tokenizer.

Predictions and Proposed Diagnostics

Based on our findings, we predict that for any script with $\rho_s < 1.0$:

- The translation tax will exceed 2.0$\times$ at the embedding layer (correlation between $\rho_s$ and embedding tax is approximately logarithmic based on our three-script comparison).

- Circuit duplication will produce 0 advantageous configurations.

- LoRA fine-tuning with 15,000--20,000 examples of target-script text will reduce the tax by at least 50\

- Mode collapse risk increases when SFT instruction diversity is below 100 unique templates (based on our V2 failure at 12 templates and V3 success at 340 templates).

We propose that model developers adopt the following diagnostics:

- Vocabulary audit: For each Unicode script block, compute $\rho_s$ and flag scripts with $\rho_s < 1.5$.

- Translation tax spot check: For flagged scripts, compute L2 norm ratios at the embedding, middle, and output layers on 20 parallel sentence pairs. Total cost: approximately 10 minutes on consumer hardware.

- Remediation threshold: Any script with tax $> 2.0\times$ should receive targeted CPT and SFT data in the model's next training iteration.

Decolonizing NLP Infrastructure

The pattern we observe is not coincidental.
The scripts most likely to be invisible to LLMs are disproportionately scripts invented by colonized peoples as acts of linguistic self-determination.
N'Ko was created in response to a claim that African languages could not be written.
Adlam was created by two Guinean brothers to give Fulani its own writing system.
Osmanya was created by Osman Yusuf Kenadid for Somali independence.

These scripts share a historical pattern: they were designed as alternatives to Latin (or Arabic) orthographies imposed by colonial or religious authorities.
Their absence from LLM training data is a second-order effect of the same colonial dynamics that motivated their creation.
The languages themselves---Bambara, Fulani, Somali---have Latin-script resources because colonial institutions produced Latin-script materials.
The indigenous scripts, created as acts of cultural resistance, were not adopted by the institutions (universities, governments, international organizations) that produce the digital text corpora on which LLMs are trained.

We are not making a political argument.
We are making a technical observation: the training data pipeline for large language models systematically excludes scripts that were created by indigenous communities, and this exclusion has measurable computational consequences (2.94$\times$ translation tax, 0/55 circuit configurations, 78.1\
The fix is also technical: include these scripts in pre-training data and tokenizer vocabulary.
The cost is negligible relative to the total cost of LLM training: our entire fine-tuning pipeline costs \$0 on consumer hardware, and including N'Ko Wikipedia (3.7M characters) in a model's pre-training corpus would add approximately 0.0001\

Discussion

Dead Circuits versus Weak Circuits

Our central finding---that N'Ko circuits in Qwen3-8B are absent rather than weak---has implications for how the field approaches multilingual model development.

A weak circuit can be strengthened through small amounts of targeted data.
If N'Ko's circuits were weak (processing N'Ko with reduced accuracy but non-random performance), we would expect:

- Circuit duplication to produce modest but above-random improvements (we observe 0/55 above random).

- Fine-tuning to require fewer examples (our V1 with 17,360 examples only partially closes the gap).

- The translation tax to vary across layers (some layers stronger, some weaker), rather than being uniformly $\sim$3$\times$ across all 36 layers.

The dead circuit pattern we observe is qualitatively different.
The model has no N'Ko representations to build on.
Fine-tuning must create representations from a blank slate, which requires more data, more iterations, and more careful curriculum design (as evidenced by the V2 mode collapse).

This distinction maps onto a broader taxonomy of multilingual model failures:

- Type A (Weak circuits): The model has some pre-training data for the script but not enough. Translation tax 1.2--2.0$\times$. Circuit duplication sometimes helps. Small-scale fine-tuning (1,000--5,000 examples) is effective. Examples: minority European languages in Latin script.

- Type B (Dead circuits): The model has zero or near-zero pre-training data for the script. Translation tax $> 2.0\times$. Circuit duplication never helps. Large-scale fine-tuning (15,000+ examples) with curriculum design is required. Examples: N'Ko, Adlam, Tifinagh.

- Type C (Structural incompatibility): The model's architecture cannot process the script's structural properties (e.g., extremely long sequences for logographic scripts without proper positional encoding). Translation tax may be moderate, but error patterns are qualitatively different from Types A and B. This is rare in modern transformer architectures.

N'Ko is a clear Type B failure.
The remediation path is well-defined: include N'Ko text in pre-training and allocate vocabulary entries.
The cost is trivial.
The barrier is awareness, not resources.

Adlam: The Most Urgent Case

Among the scripts we predict to be invisible (Table~[ref: tab:invisible-scripts]), Adlam deserves special attention.
Designed in 1989 by Ibrahima and Abdoulaye Barry, two Guinean brothers, Adlam serves the Fulani (Pular/Fulfulde) language community of approximately 40 million speakers across 20+ West African countries.
Like N'Ko, Adlam is:

- A deliberately engineered phonemic alphabet (not an evolved script)

- Right-to-left

- Bijective in its phoneme-grapheme mapping

- Standardized in Unicode (U+1E900--U+1E95F, Unicode 9.0, 2016)

- Used for education, signage, social media, and personal correspondence

Unlike N'Ko, Adlam receives zero vocabulary entries in Qwen3's tokenizer.
Not even single-character fallback tokens.
Adlam text fed to Qwen3 would be processed as raw byte sequences, producing even worse representations than N'Ko's 32 character tokens.
We predict a translation tax exceeding 4.0$\times$ for Adlam---worse than N'Ko's 2.94$\times$---based on the complete absence of Adlam-specific tokenization.

The research infrastructure we have built for N'Ko (activation profiling, circuit duplication, LoRA fine-tuning pipeline) is directly transferable to Adlam.
Adlam Wikipedia exists (smaller than N'Ko Wikipedia but sufficient for CPT), and the phonemic regularity of the script means a cross-script bridge from Latin Fulani to Adlam is feasible with the same architecture as our N'Ko bridge.
We estimate the total cost of an Adlam activation study and LoRA fine-tuning at under \$5.

The Tokenizer as Gatekeeper

Our analysis identifies the tokenizer as the single most consequential decision point for script equity.
A script that receives no BPE merges in the tokenizer training process receives no subword vocabulary entries, which means:

- Character-level tokenization at 3--5$\times$ the token rate of well-represented scripts.

- Embedding-layer representations that carry no compositional information (each character embedding is trained independently with no knowledge of character co-occurrence patterns).

- Downstream attention patterns that must operate over much longer sequences with much weaker individual token representations.

The tokenizer is typically trained once, early in the model development process, on a fixed corpus.
Scripts absent from that corpus are permanently excluded from the model's representational capacity.
No amount of downstream fine-tuning can fully compensate for the absence of dedicated vocabulary entries, because fine-tuning operates on the fixed vocabulary of the pre-trained model.

We advocate for tokenizer auditing as a mandatory step in multilingual model release.
A simple metric---the number of scripts with $\rho_s < 1.0$---would immediately surface the most severe equity gaps.

Limitations

Single model family and scale.
Our activation profiling is conducted on Qwen3-8B-Instruct (8-bit quantized).
While we expect the qualitative findings (dead circuits, three-zone failure) to generalize to other language models with similar tokenizer design decisions (GPT-4, Claude, Llama, Gemma), and to larger models in the Qwen3 family, we have not verified this.
The specific numbers (3.3$\times$ transformer-block tax, 77.9\

Arabic comparison is limited.
Our Arabic baseline uses 50 sentence pairs versus 30 for the English-N'Ko comparison.
A full Arabic activation profiling study with matched sample sizes and all four metrics at every layer would strengthen the comparison.

Quantization effects.
We use 8-bit quantization, which introduces some activation distortion relative to full-precision inference.
Our spot-check against fp16 inference on a subset of 10 sentence pairs shows equivalent ratios ($\pm$0.03), but we cannot rule out larger deviations from full-precision behavior in specific layers or for specific inputs.

Single model scale.
Both activation profiling and LoRA fine-tuning are conducted on Qwen3-8B-Instruct.
The 79\

V3 data includes synthetic examples.
The 92,184 V3 training examples include substantial synthetic data (BPE completions, template-generated instructions).
The quality of the fine-tuned model's outputs has not been evaluated by native N'Ko readers at scale; our evaluation is limited to automated metrics (perplexity, accuracy, translation tax).

Causal claims are limited.
We observe correlations between vocabulary density and translation tax, but our three-script comparison (English, Arabic, N'Ko) is insufficient for strong causal claims.
A comprehensive study across 10+ scripts with varying $\rho_s$ values would strengthen the vocabulary-to-representation-quality relationship.

Attention Pattern Analysis

Beyond the aggregate metrics reported in \S[ref: sec:translation-tax], we examine the structure of attention patterns for N'Ko versus English inputs.

Attention Entropy by Layer

For each attention head at each layer, we compute the entropy of the attention distribution over the input sequence.
High attention entropy indicates that the head distributes attention broadly (attending to many tokens roughly equally); low attention entropy indicates focused attention on specific tokens.

Caption: Mean attention entropy per head by layer group. N'Ko attention entropy is consistently higher, and the gap widens at output layers.

Table~[ref: tab:attention-entropy] reveals that N'Ko attention patterns are dramatically more diffuse than English at every layer group.
At the output layers (29--36), N'Ko attention entropy is 121\
This means attention heads that should be focusing on specific context tokens to make predictions are instead distributing attention nearly uniformly---the attention mechanism cannot identify which tokens are informative because no token carries meaningful representation.

For English, attention entropy decreases with depth: from 3.42 at embedding to 1.87 at output.
This is the expected pattern---upper layers should attend to increasingly specific context.
For N'Ko, attention entropy barely decreases: from 4.87 to 4.14.
The attention mechanism has nothing to focus on.

Head Specialization

In well-functioning multilingual models, individual attention heads develop specializations: syntactic heads that attend to grammatical dependencies, positional heads that track relative token positions, and semantic heads that attend to semantically related tokens [citation: clark2019attention].

We measure head specialization by computing the standard deviation of each head's attention entropy across the 30 examples.
High standard deviation indicates a head that behaves differently for different inputs (specialization); low standard deviation indicates a head that behaves identically regardless of input (no specialization).

For English inputs, the mean attention entropy standard deviation across all heads in layers 12--24 is 0.83.
For N'Ko inputs, it is 0.21.
This 4$\times$ reduction in head-level variability confirms that N'Ko inputs do not trigger the specialized attention patterns that English inputs produce.
The heads are not malfunctioning---they simply have nothing to specialize on.

Cross-Attention Between N'Ko Characters

We examine whether attention patterns between adjacent N'Ko characters show any evidence of the model learning character composition (combining individual character tokens into subword or word-level representations).

For a typical English word tokenized as a single subword token, no composition is needed.
For a N'Ko word tokenized as 4--8 individual character tokens, the model must learn to combine these tokens into a coherent representation through attention.

We compute the mean attention weight between adjacent N'Ko character tokens belonging to the same word (as determined by space tokenization) across layers 0--10.

Caption: Attention weight analysis. N'Ko character tokens attend to each other less than English subword tokens attend to adjacent subwords.

Table~[ref: tab:cross-attention] shows that N'Ko character tokens attend to adjacent characters less than English subword tokens attend to adjacent subwords (0.047 vs. 0.082).
This is the opposite of what we would need for successful character composition: the model should attend more to adjacent characters within a word to build up word-level representations from character-level inputs.
Instead, it attends less, treating each character as largely independent---consistent with the dead circuit hypothesis.

Reproducibility and Data Release

Computational Requirements

All experiments can be reproduced on the following hardware:

- Activation profiling: Apple M4 with 16GB unified memory (or any GPU with $\geq$8GB VRAM). We use 8-bit quantization via Hugging Face Transformers, which requires approximately 9GB for Qwen3-8B. Profiling 30 sentence pairs takes approximately 15 minutes. Total profiling time in our experiments: 896.8 seconds.

- LoRA fine-tuning: Apple M4 with 16GB unified memory (or any GPU with $\geq$8GB VRAM). We use MLX v0.29 on macOS; PyTorch + bitsandbytes achieves equivalent results on CUDA GPUs. Total training time: 185 minutes across three stages.

- Circuit duplication: Same hardware as profiling. 45 configurations at approximately 3 minutes each. Total: approximately 2.25 hours.

Data Availability

All data used in this study is either publicly available or released with this paper:

- N'Ko Wikipedia: 1,693 articles, 3.7M characters. Available via Wikimedia dumps.

- Parallel sentence pairs: 30 English-N'Ko pairs used for profiling. Released in our repository.

- Activation profiles: Per-layer metric values for all 30 pairs $\times$ 37 layers $\times$ 4 metrics. Released as JSON.

- Circuit duplication scores: All 45 configurations with English and N'Ko scores. Released as CSV.

- Fine-tuned model weights: LoRA adapter weights for all three stages. Released on HuggingFace.

- BPE tokenizer: 512-merge N'Ko tokenizer trained on 62,035 word occurrences. Released in repository.

Evaluation Protocol

Our translation tax metric (L2 norm ratio) is computed as follows:

- For each sentence pair $(s_e, s_n)$ where $s_e$ is English and $s_n$ is N'Ko:

- Tokenize independently using Qwen3's tokenizer.

- Forward pass through the model, storing hidden states at every layer.

- Compute per-token L2 norm at each layer, then average across tokens in each sentence.

- The translation tax at layer $l$ is $\|h_l^e\|_2 / \|h_l^n\|_2$.

- Average across all 30 sentence pairs.

The per-token averaging in step 4 ensures that the metric is not biased by sequence length differences.
The sentence-level averaging in step 5 provides a stable estimate across the $N=30$ pairs.

Future Work

Vocabulary Injection

The most direct intervention for script invisibility is vocabulary injection: adding BPE-trained subword tokens for the target script to the model's tokenizer before pre-training.
This approach has been validated for Arabic [citation: antoun2020arabert] and Indic languages [citation: kakwani2020indicnlpsuite] but has not been systematically studied for script-invisible languages.

We propose a controlled experiment: take an open-source LLM (e.g., Llama 3.1 or Qwen3), extend its tokenizer with 512 N'Ko BPE merges, re-initialize the corresponding embedding rows, and perform continued pre-training on N'Ko Wikipedia.
The prediction from our analysis is that vocabulary injection followed by modest CPT (10,000--20,000 examples) will produce lower translation taxes than the LoRA approach alone, because the model will have proper subword representations from the start.

Multi-Script Profiling Study

Our three-script comparison (English, Arabic, N'Ko) establishes the methodology but is insufficient for strong causal claims about the vocabulary-to-representation relationship.
A comprehensive study across 15--20 scripts with vocabulary density $\rho_s$ ranging from 0.0 (Adlam) to 609 (Latin) would enable regression analysis relating $\rho_s$ to translation tax, circuit duplication scores, and fine-tuning data requirements.
Such a study would produce a practical guide for model developers: given a target $\rho_s$, how much additional training data is needed to achieve translation tax $<$ 1.5$\times$?

Candidate scripts for this study span three categories:

- Well-represented ($\rho_s > 10$): Latin, CJK, Arabic, Cyrillic---providing the high-$\rho_s$ reference points.

- Moderately represented ($\rho_s \in [1, 10]$): Devanagari, Thai, Tamil, Georgian---scripts with dedicated vocabulary entries but smaller than Latin or CJK.

- Invisible ($\rho_s < 1$): N'Ko, Adlam, Tifinagh, Vai, Osmanya, Ol Chiki, Bamum---scripts with zero or near-zero vocabulary entries.

The expected relationship is logarithmic: $\text{tax} \sim -\alpha \log(\rho_s) + \beta$, where $\alpha$ captures the sensitivity of representation quality to vocabulary density and $\beta$ is the baseline tax for a well-represented script.
Our three data points (Latin: $\rho_s = 609$, tax $\approx 1.0$; Arabic: $\rho_s = 16.4$, tax $\approx 1.05$; N'Ko: $\rho_s = 0.5$, tax $= 2.94$) are consistent with a logarithmic relationship but insufficient to fit it precisely.

Cross-Model Comparison

Our analysis uses Qwen3-8B-Instruct exclusively.
Extending the study to other model families (Llama 3.1, Gemma 2, GPT-4, Claude) would establish whether script invisibility is a universal phenomenon or specific to particular tokenizer designs.
We hypothesize that the qualitative findings (dead circuits for zero-vocabulary scripts) will be universal, but the specific translation tax values will vary based on:

- Tokenizer training data: Models trained on more diverse web crawls may include incidental N'Ko or Adlam text, producing non-zero (but still weak) representations.

- Byte-level tokenization: Models using byte-level BPE (like GPT-4) process all Unicode text without fallback, potentially producing better (but still weak) representations for invisible scripts.

- Architecture scale: Larger models may develop stronger character-composition abilities through their larger middle-layer capacity, partially compensating for tokenizer deficiency.

Real-Time Translation Tax Monitoring

We envision a translation tax monitoring system integrated into model training pipelines.
At regular intervals during pre-training (every 10,000 steps), the system would compute translation tax estimates for a set of probe scripts (100 sentence pairs per script, 10 scripts), flagging any script where the tax exceeds 2.0$\times$.
This would enable model developers to intervene during training---increasing the proportion of that script's data in the training mixture---rather than discovering script invisibility after training is complete.

The computational cost of such monitoring is modest: 1,000 sentence pairs $\times$ 37 layers $\times$ 4 metrics can be computed in approximately 30 minutes on a single GPU, a negligible fraction of the total training compute for any large-scale model.

Ethical Considerations

This work involves the computational representation of a script created by a colonized people as an act of cultural self-determination.
We approach this with awareness that technical interventions---adding N'Ko to a tokenizer, fine-tuning a model---are not substitutes for the political and cultural work of script revitalization.
The decision of which scripts to support in language technology should involve the communities that use those scripts, not only the engineers who build the models.

Our training data is drawn from N'Ko Wikipedia, which represents a particular register (encyclopedic) and a particular community of contributors.
N'Ko is used for a wider range of purposes---religious texts, poetry, personal correspondence, commercial signage---that our training data does not capture.
The model's capabilities after fine-tuning are therefore biased toward encyclopedic N'Ko.

Conclusion

We have performed the first per-layer activation profiling study of a large language model processing N'Ko script, establishing quantitative baselines for script invisibility: a 2.94$\times$ translation tax, 0/55 productive circuit duplication configurations, 78.1\

Three-zone failure analysis reveals structurally distinct collapse modes: comprehension failure at the embedding layer (sparse, weak representations from 32 character-level tokens), reasoning vacuum in the middle layers (empty circuits that cannot be amplified by duplication), and incoherent output at the final layers (near-maximum entropy, flat kurtosis, near-random token prediction).

Comparison with Arabic---another RTL script that the same model processes competently---isolates vocabulary allocation as the mechanistic root cause.
Arabic receives 4,200 vocabulary entries with rich subword coverage; N'Ko receives 32 single-character tokens.
The difference in representation quality maps directly to this 131$\times$ vocabulary disparity.

Three-stage LoRA fine-tuning reduces the translation tax from 2.94$\times$ to 0.70$\times$ using 92,184 training examples and 185 minutes on an Apple M4, demonstrating that the model's architecture is capable---the data was the bottleneck.
We document the V1/V2/V3 progression, including mode collapse in V2 and its resolution through data diversification.

The framework generalizes to at least six other scripts that we predict are invisible to current LLMs: Adlam, Tifinagh, Vai, Osmanya, Ol Chiki, and Bamum.
We propose vocabulary density ($\rho_s$) as a diagnostic metric, translation tax spot-checking as a validation protocol, and tokenizer auditing as a mandatory step in multilingual model releases.

The circuits are dead.
The cost of bringing them to life is three hours on consumer hardware.
The barrier is not compute.
It is the invisible filter of training data curation that systematically excludes the scripts created by the peoples who needed them most.

Code, data, and activation profiles: https://github.com/Diomandeee/nko-brain-scanner

Total compute cost: \$1.72 (activation profiling, Vast.ai A100) + \$0.00 (LoRA fine-tuning, Apple M4 16GB) = \$1.72

References

#1

antoun2020arabert
Wissam Antoun, Fady Baly, and Hazem Hajj. 2020.
\newblock {AraBERT}: Transformer-based model for {Arabic} language understanding.
\newblock In LREC Workshop on Open-Source Arabic Corpora and Processing Tools.

wmt2023nko
Lo\"{i}c Barrault et~al. 2023.
\newblock {WMT} 2023 shared task: Machine translation for {N'Ko}.
\newblock In Proceedings of the Eighth Conference on Machine Translation (WMT).

conneau2020unsupervised
Alexis Conneau et~al. 2020.
\newblock Unsupervised cross-lingual representation learning at scale.
\newblock In Proceedings of ACL 2020.

dossou2022afrolm
Bonaventure F.~P. Dossou, Atnafu~Lambebo Tonja, et~al. 2022.
\newblock {AfroLM}: A self-active learning-based multilingual pretrained language model for 23 {African} languages.
\newblock In SustaiNLP Workshop at EMNLP.

doumbouya2021
Moussa Doumbouya et~al. 2021.
\newblock Using radio archives for low-resource speech recognition: Towards an automatic transcription of {Bambara} radio broadcasts.
\newblock In Proceedings of NAACL.

hu2022lora
Edward~J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu~Wang, and Weizhu Chen. 2022.
\newblock {LoRA}: Low-rank adaptation of large language models.
\newblock In ICLR 2022.

kakwani2020indicnlpsuite
Divyanshu Kakwani et~al. 2020.
\newblock {IndicNLPSuite}: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for {Indian} languages.
\newblock In Findings of EMNLP.

magueresse2020lowresource
Alexandre Magueresse, Vincent Carles, and Evan Heetderks. 2020.
\newblock Low-resource languages: A review of past work and future challenges.
\newblock In Proceedings of the 1st Workshop on NLP for Positive Impact (ACL).

ng2024rys
David~Noel Ng. 2024.
\newblock Revisit your shoulders: A circuit analysis of transformer layers for reasoning enhancement.
\newblock arXiv preprint.

pfeiffer2020adapterhub
Jonas Pfeiffer et~al. 2020.
\newblock {AdapterHub}: A framework for adapting transformers.
\newblock In Proceedings of EMNLP 2020 (Demo).

sennrich2016bpe
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.
\newblock Neural machine translation of rare words with subword units.
\newblock In Proceedings of ACL 2016.

tenney2019bert
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.
\newblock {BERT} rediscovers the classical {NLP} pipeline.
\newblock In Proceedings of ACL 2019.

tonja2023natural
Atnafu~Lambebo Tonja et~al. 2023.
\newblock Natural language processing in {Ethiopian} languages: Current state, challenges, and opportunities.
\newblock In AfricaNLP Workshop at ACL 2023.

unicode2006nko
{Unicode Consortium}. 2006.
\newblock N'Ko block: {U+07C0--U+07FF}.
\newblock The Unicode Standard, Version 5.0+.

clark2019attention
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher~D. Manning. 2019.
\newblock What does {BERT} look at? {An} analysis of {BERT}'s attention.
\newblock In Proceedings of the 2019 ACL Workshop BlackboxNLP.

aji2022bloom
Alham~Fikri Aji et~al. 2022.
\newblock One country, 700+ languages: {NLP} challenges for underrepresented languages and dialects in {Indonesia}.
\newblock In Proceedings of ACL 2022.

adelani2022masakhaner2
David~Ifeoluwa Adelani et~al. 2022.
\newblock {MasakhaNER} 2.0: {Africa}-centric transfer learning for named entity recognition.
\newblock In Proceedings of EMNLP 2022.

joshi2020state
Pratik Joshi et~al. 2020.
\newblock The state and fate of linguistic diversity and inclusion in the {NLP} world.
\newblock In Proceedings of ACL 2020.

rust2021good
Phillip Rust et~al. 2021.
\newblock How good is your tokenizer? {On} the monolingual performance of multilingual language models.
\newblock In Proceedings of ACL 2021.

bayelemabaga2025
Adama Coulibaly et~al. 2025.
\newblock Bayelemabaga: A {Bambara}-{French} parallel corpus for machine translation.
\newblock In Proceedings of NAACL 2025.

devlin2019bert
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.
\newblock {BERT}: Pre-training of deep bidirectional transformers for language understanding.
\newblock In Proceedings of NAACL 2019.

muller2021first
Benjamin M\"{u}ller et~al. 2021.
\newblock First align, then predict: Understanding the cross-lingual ability of multilingual {BERT}.
\newblock In Proceedings of EACL 2021.

ponti2019modeling
Edoardo~Maria Ponti et~al. 2019.
\newblock Modeling language variation and universals: A survey on typological linguistics for natural language processing.
\newblock Computational Linguistics, 45(3):559--601.

lauscher2020zero
Anne Lauscher et~al. 2020.
\newblock From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers.
\newblock In Proceedings of EMNLP 2020.

wang2020extending
Zihan Wang et~al. 2020.
\newblock Extending multilingual {BERT} to low-resource languages.
\newblock In Findings of EMNLP 2020.