Conversation
Phoenix always force-decoded newly opened files as UTF-8. A file that declares a different charset (e.g. a legacy HTML file with <meta charset=windows-1252>) or starts with a UTF-16/UTF-32 BOM got silently and irreversibly corrupted - every undecodable byte replaced with U+FFFD - both in the editor and in live preview, which serves the same in-memory text. Add EncodingDetector, wired into DocumentCommandHandlers' file-open flow, that sniffs a file's raw bytes before the first (lossy) read: - Honors a BOM for any non-binary file (previously not detected at all, even for plain .txt files). - Honors a self-declared <meta charset>/Content-Type for markup files, but only when the bytes aren't already valid UTF-8, so a stale/incorrect meta tag on a real UTF-8 file is never second-guessed. - Never overrides a previously user-picked encoding for a path. Adds unit tests (EncodingDetector-test.js) and extends the existing file-encoding-integ-test.js with real fixtures covering both the windows-1252 meta-charset case and the BOM auto-detection gap.
…ection
Two follow-up correctness fixes for the encoding auto-detection added in the
previous commit, plus the perf optimization that surfaced them:
- Skip re-running EncodingDetector when a file's encoding is already known
(File instances are cached/reused per path for the session), so reopening a
tab doesn't re-read + re-detect every time.
- That skip-check needed to be more careful than a plain truthiness check:
several unrelated features (ProjectManager's "Download" command,
AIChatPanel's image-attach, MediaViewer, etc.) read a File instance with
{encoding: fs.BYTE_ARRAY_ENCODING} and no doNotCache for their own non-text
purposes - which, per File.read()'s own caching, leaves that sentinel value
cached in file._encoding even though the file was never opened as a
document. Added EncodingDetector.isKnownTextEncoding() to tell a real text
codec apart from that sentinel (notably still present in
fs.SUPPORTED_ENCODINGS, so that list alone can't distinguish them).
- Found a deeper, pre-existing bug while writing the regression test for the
above: File.read()'s cache-hit check only compares options.encoding to
this._encoding - it doesn't invalidate _contents/_stat when something
bare-reassigns file._encoding (which _doOpen has always done for the
explicit-option and stored-preference branches too, not just detection).
So: a prior raw-byte read followed by reassigning _encoding to the correct
detected value could make the next real open cache-hit on stale raw bytes
instead of re-reading as text. Fixed via a _setFileEncoding() helper that
clears the stale cache whenever the encoding actually changes.
Extends EncodingDetector-test.js and file-encoding-integ-test.js with
regression coverage for both.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phoenix always force-decoded newly opened files as UTF-8. A file that declares a different charset (e.g. a legacy HTML file with ) or starts with a UTF-16/UTF-32 BOM got silently and irreversibly corrupted - every undecodable byte replaced with U+FFFD - both in the editor and in live preview, which serves the same in-memory text.
Add EncodingDetector, wired into DocumentCommandHandlers' file-open flow, that sniffs a file's raw bytes before the first (lossy) read:
Adds unit tests (EncodingDetector-test.js) and extends the existing file-encoding-integ-test.js with real fixtures covering both the windows-1252 meta-charset case and the BOM auto-detection gap.