in Editing
Spelling invented words the same way every time
An invented word has no dictionary to check it against. The book's own glossary has to do that job, and someone has to keep it.
Writers of fantasy, science fiction and other speculative work create vocabulary: names of places, peoples, creatures, technologies, titles and rituals. Some writers invent whole languages. Every invented word is a word with no dictionary entry, which means no spellchecker will catch it when it goes wrong. And it does go wrong, because words drift over a long drafting process.
How drift happens
Say an invented city is called Vareth in chapter one. By chapter twelve it has become Varreth, and in a late revision, Vâreth. The writer knows these are the same place and may not see the differences. A reader does, and may briefly wonder whether they are different places.
Drift comes from several sources. The writer revises the spelling of a word partway through and does not go back. The writer types quickly and a variant slips in. A word is capitalised in some places and not in others. A hyphen appears and disappears. Plurals are formed inconsistently. Diacritics, such as accents and circumflexes, are especially vulnerable, because they are easy to drop when typing fast.
Keep a glossary
The fix is a glossary or lexicon, kept alongside the manuscript from the beginning. For each invented term, record the exact spelling, whether it is capitalised, how the plural is formed, whether it takes an article ("the Vareth" or just "Vareth"), and a short definition. If a word has a possessive or adjectival form, record that too.
This document is part of the style sheet, and it is often the longest part in a speculative novel.
Spelling rules for invented languages
If the book includes several words from an invented language, it helps to set a few spelling conventions for that language and keep to them. Which letter combinations are used, whether double consonants appear, how particular sounds are written. Readers may not know the rules, but they will sense whether the words feel as if they come from one language or several.
It is also worth thinking about pronounceability. Words that readers cannot sound out tend to be skimmed, and skimmed words are harder to remember and easier to confuse. That is a stylistic choice rather than a rule, but it affects how readers experience the text.
Checking the manuscript
Most word processors let you add words to a custom dictionary. Adding every invented term, spelled correctly, means that variants will be flagged. It is not foolproof, since a variant that happens to match another invented word will pass, but it catches a great deal.
Another approach is to search for fragments. Searching for "Var" in the example above would surface every form of the city name, correct or not.
In a series, the glossary becomes more valuable with every book. Readers who have followed several volumes will notice if a name they know well suddenly appears differently. A glossary that travels from book to book, and is handed to each editor and proofreader, keeps the series consistent across years of writing.
Invented words are part of a world's texture. Spelling them consistently is what makes that world feel real.