feat(string): add UTF-8 string conversion and validation functions

[bobtista]

• Relates to refactor(string): Add functions for handling UTF8 encoded strings #2045

Adds UTF-8 string handling to WWLib and plumbs it through the codebase, replacing the GameSpy-specific Win32 wrappers with a shared, portable implementation.

Picks up the work in #2045 with API adjustments per the review from @xezon.

New: WWLib/utf8.h / utf8.cpp

• Utf8_Validate(const char* str, size_t length) — returns true if the range is valid UTF-8 per RFC 3629 (rejects overlong encodings, surrogate codepoints, and codepoints above U+10FFFF)
• Utf16Le_To_Utf8_Len(const wchar_t* src, size_t srcLen) / Utf8_To_Utf16Le_Len(const char* src, size_t srcLen) — required output size, not counting the null terminator
• Utf16Le_To_Utf8(char* dest, size_t destLen, const wchar_t* src, size_t srcLen)
• Utf8_To_Utf16Le(wchar_t* dest, size_t destLen, const char* src, size_t srcLen)

Naming follows the Snake_Case convention used in WWVegas. The caller queries the _Len helper to size the destination, then calls the matching conversion; the conversion writes a null terminator if room remains and returns the number of units written, or 0 on invalid input or an empty source.

The implementation is an RFC 3629 transcoder using no platform text APIs (no Win32, no ICU), so it builds on every platform. The wide side adapts to wchar_t width at compile time: UTF-16 with surrogate pairs where wchar_t is 16-bit (Windows), UTF-32 codepoints where it is 32-bit.

AsciiString::translate / UnicodeString::translate

Replaces the broken implementations that only worked for 7-bit ASCII (marked @todo since the original code) with proper UTF-8 conversion. UnicodeString::translate validates first and falls back to a 1:1 byte cast for data that is not valid UTF-8 (e.g. legacy CP1252), so such content keeps its original characters instead of turning into replacement characters, and data written by the old lossy path reads back byte-identical.

ThreadUtils.cpp

Replaces raw Win32 API calls in MultiByteToWideCharSingleLine and WideCharStringToMultiByte with the new WWLib functions, using std::string::resize / std::wstring::resize to avoid duplicate allocation.

Changes from the current body: dropped the two removed functions and the single-arg validate; corrected the return-value contract; replaced "Windows-only / Win32 wrappers" with the portable-transcoder + wchar_t-width description; added the surrogate note; documented the CP1252 fallback in the translate section.

Todo

• testing — codec verified locally via standalone RFC 3629 unit tests, both wide widths
• testing - Windows + VC6 CI build

[greptile-apps]

Greptile Summary

Adds a portable UTF-8/wide-character transcoder and integrates it into engine string and GameSpy conversion paths.

• Validates UTF-8 by rejecting malformed, overlong, surrogate, and out-of-range encodings.
• Replaces the previous ASCII-only AsciiString and UnicodeString translations with UTF-8 conversion and a legacy byte-cast fallback.
• Replaces GameSpy’s Win32 conversion calls with shared WWLib helpers.
• Registers the new implementation in the WWLib build.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
Core/Libraries/Source/WWVegas/WWLib/utf8.cpp	Implements width-aware UTF-8 transcoding with checks for malformed, overlong, surrogate, and out-of-range sequences.
Core/Libraries/Source/WWVegas/WWLib/utf8.h	Defines the portable conversion API, invalid-input sentinel, sizing contracts, and destination-buffer behavior.
Core/GameEngine/Source/Common/System/AsciiString.cpp	Replaces lossy wide-to-byte casting with correctly sized UTF-8 encoding.
Core/GameEngine/Source/Common/System/UnicodeString.cpp	Adds validated UTF-8 decoding while preserving legacy non-UTF-8 data through byte-wise fallback.
Core/GameEngine/Source/GameNetwork/GameSpy/Thread/ThreadUtils.cpp	Migrates GameSpy string conversion from Win32 APIs to WWLib and handles invalid UTF-8 through the documented fallback.
Core/Libraries/Source/WWVegas/WWLib/CMakeLists.txt	Adds the portable UTF-8 source and header to the WWLib target.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Engine or GameSpy text] --> B{Conversion direction}
    B -->|UTF-8 to wide| C[Utf8_To_Wide_Len]
    C --> D{Valid UTF-8?}
    D -->|Yes| E[Utf8_To_Wide]
    D -->|No| F[Legacy one-to-one byte fallback]
    B -->|Wide to UTF-8| G[Wide_To_Utf8_Len]
    G --> H[Wide_To_Utf8]
    E --> I[UnicodeString or std::wstring]
    F --> I
    H --> J[AsciiString or std::string]

Loading

_{Reviews (17): Last reviewed commit: "build: Use qualified include path for WW..." | Re-trigger Greptile}

[bobtista]

• Fixed the if formatting
• added RFC 3629 overlong and out-of-range checks
• RE the theoretical memory leak, can that even happen here? set() allocates via the engine's custom memory allocator which crashes on failure rather than throwing, so the leak path can't really be reached right?

[xezon]

Get_Utf8_Size should not include the null terminator in its size.

[xezon]

What happens if dest_size does not have enough room for a null terminator?

[bobtista]

The doc says "Does not write a null terminator" - should we add more comments? Change the functions to always null-terminate? What do you want here?

[xezon]

Maybe make it behave like strncpy? Writes null if there is room, otherwise not.

The only issue with the current interface then is that we will not know if it wrote the null terminator. Maybe it should return size_t instead, returning the number of characters it writes or would like to write? MultiByteToWideChar also does that.

I suggest to think this through and design the function interface in a way that it can be conveniently be used for fixed size strings (std::string, AsciiString) and large throwaway buffers (char arr[512]).

The behavior definitely needs to be documented.

[bobtista]

Done. It writes the terminator when there is room, otherwise not, and returns the units written. Bad input returns UTF8_INVALID, so it is distinct from an empty result.

[xezon]

The diff now shows unrelated changes.

[bobtista]

The diff now shows unrelated changes.

Try again cleaned up the commits and force pushed

[Mauller]

So is this translating UTF16LE from windows into UTF8 that is then stored within AsciiString?

If so this may help with the paths issue with usernames and paths not using Latin characters, but file handling functions will need updating to use unicode variants instead of Ascii.

[bobtista]

Yes. It does not fix path handling on its own though, the file APIs still take the narrow string. Should that be a separate change?

[Mauller]

I wonder if we should also add a flag to state that the Ascii string is holding a UTF8 string?

I guess all normal ascii characters will display properly, it's just extended character sets that will look garbled.

[OmniBlade]

Slight nitpick on function naming, shouldn't it be Utf16 rather than Wchar as Wchar is only Utf16 on windows yet we will likely want the conversion functions on other platforms as well for at least csf parsing if nothing else.

[bobtista]

Slight nitpick on function naming, shouldn't it be Utf16 rather than Wchar as Wchar is only Utf16 on windows yet we will likely want the conversion functions on other platforms as well for at least csf parsing if nothing else.

Yeah, but it's consistent with the other naming as it is for now. How about we keep the naming and using a uint16_t/char16_t type internally rather than wchar_t when we make non windows paths? Or would you rather we rename to something like Utf16_To_Utf8?

[Mauller]

Slight nitpick on function naming, shouldn't it be Utf16 rather than Wchar as Wchar is only Utf16 on windows yet we will likely want the conversion functions on other platforms as well for at least csf parsing if nothing else.

Yeah, but it's consistent with the other naming as it is for now. How about we keep the naming and using a uint16_t/char16_t type internally rather than wchar_t when we make non windows paths? Or would you rather we rename to something like Utf16_To_Utf8?

If anything it would be Utf16Le_To_Utf8, windows uses the little endian utf16 format. Not sure if Utf16Be is used much anywhere but worth being concise with it.

[githubawn]

Could the unconditional replacement of translate() silently break callers passing legacy CP1252 data, causing strings that are valid CP1252 but invalid UTF-8 to corrupt or clear?

Maybe something like:
Check if the source is valid UTF-8 via Utf8_Validate.
If valid: Proceed with UTF-8 conversion.
If invalid: Fall back to the legacy 1:1 byte-to-wide cast (treating it as CP1252).

[xezon]

Maybe put a breakpoint and check usage patterns.

[xezon]

What is the current state of this? Can it be finalized?

[bobtista]

What is the current state of this? Can it be finalized?

Yeah, I just pushed cleanups addressing the open threads, it should be ready for another pass

[bobtista]

Pushed updates addressing the open comments:

• Portable transcoder, dropped the Win32 WideCharToMultiByte/MultiByteToWideChar wrappers and the #else #error for a hand-rolled RFC 3629 encode/decode in utf8.cpp. It compiles on all platforms now, and the wide side adapts to wchar_t width at compile time (UTF-16 with surrogate pairs on Windows, UTF-32 elsewhere), so it's reusable for CSF parsing off-Windows. @OmniBlade — I kept the Utf16Le_* naming for now per the earlier thread; happy to rename if you'd prefer.
• CP1252 fallback, UnicodeString::translate now validates first. Valid UTF-8 decodes as UTF-8; anything else (legacy CP1252) falls back to the old 1:1 byte cast instead of producing replacement characters. Bonus: data written by the old lossy path reads back byte-identical.
• Scope, reverted the ensureUniqueBufferOfSize null-terminator change, so the string files now carry only the translate rewrite.

Tested the codec in isolation: ASCII / 2- / 3- / 4-byte round-trips on both wide widths, plus RFC 3629 rejection of overlong, surrogate, out-of-range, and truncated sequences.

Ready for another pass

[bobtista]

Reworked the conversion interface:

Utf8_To_Wide_Len / Utf8_To_Wide return UTF8_INVALID on bad input. 0 now means empty only, so callers can tell the two apart.
The encoder substitutes U+FFFD for unpaired surrogates and out of range wide values. Before this, a lone surrogate encoded to ED A0 80, which the decoder then rejected.
Renamed to Wide_To_Utf8 / Utf8_To_Wide. The wide side is wchar_t, which is UTF-32 off Windows, so the Utf16Le name was wrong.
MultiByteToWideCharSingleLine falls back to a byte cast on non-UTF-8, matching UnicodeString::translate. It returned an empty string before.
Dropped Utf8_Validate, Utf8_To_Wide_Len covers it and nothing else called it.