Methylation reference data: EWAS Catalog and cross-reactive probes - #16
Merged
Conversation
Which traits each CpG is already reported for, and in how many study analyses - the "is this hit known or new?" annotation on an EWAS hit list. Bundled rather than queried at runtime. The app asked ewascatalog.org/api/ one HTTP GET per CpG: slow enough to need a button, dead in a container with no outbound network, and pinned to whatever happened to be live that day. The catalog publishes the whole thing as a flat file, so we ship a release we control. The pin is recorded three times - in the build script header, as attr(EWAS_CATALOG, "version"), and as an assertion in the consumer's tests. Reduced from 174 MB gzipped and 8.0M association rows to 6.2 MB and 6.3M rows of three columns. Nothing but the CpG and the study it came from is ever rendered, so beta, se, p, position and gene are dropped, the study id is resolved to its trait, and the pairs are counted: 724,456 CpGs and 6,515 traits. inst/extdata rather than data/ on purpose. A LazyData object stays in the package namespace for the life of the session once touched, so every Shiny worker would carry 129 MB for a table read once per hit list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Illumina probes that map to more than one genome location, so their signal is not the CpG it is attributed to. The Methylome Profiler excludes them from an EWAS behind a "Cross-reactive" checkbox. Three published blacklists, 104,124 rows over 53,498 unique probes, with the source column kept so a probe flagged by two lists keeps both. Moved here from components/app_methylome/inst/masking/, where it was a 2.7 MB CSV surviving behind a .gitignore negation. It is published reference data, not app code, and it belongs beside the EWAS catalog. Two things improve: as an xz .rds it is 140 KB rather than 2.7 MB, and the app's loader can stop guessing at candidate paths - it tried several because the working directory varies, resolved to nothing once, and left 53,498 probes tested while the UI reported a mask had been applied. get_file() is system.file(mustWork = TRUE). The build script records a real gap: the CSV arrived pre-assembled with only three source labels and no download URLs, so it converts what we have rather than rebuilding from source. The labels identify the papers; the exact supplements and accession dates were never recorded and should be resolved before the next refresh. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0176Man7WMFLZp6mTypp1afU
ESCRI11
added a commit
to bigomics/omicsplayground
that referenced
this pull request
Aug 31, 2026
It was a 2.7 MB CSV in components/app_methylome/inst/masking/, kept in the repo behind a .gitignore negation. It is published reference data, not app code, so it moves to playdata beside the EWAS catalog it is used with (bigomics/playdata#16), where it is 140 KB as an xz .rds. The loader loses its candidate-path search. It tried several paths because the app's working directory varies, and that resolved to nothing once - 53,498 cross-reactive probes tested while the UI reported a mask applied. playdata::get_file() is system.file(mustWork = TRUE), so a playdata without the list throws and the panel refuses, rather than quietly masking nothing. Verified against the real list: 53,498 probes load, and 1,552 of GSE43976-methyl-mini's 23,471 probes are masked. Requires playdata with bigomics/playdata#16. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0176Man7WMFLZp6mTypp1afU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two reference datasets the Methylome Profiler needs, bundled rather than fetched or carried in the app repo.
EWAS Catalog (
ewas-catalog.rds, 6.2 MB) — which traits each CpG is already reported for. The app previously askedewascatalog.org/api/one HTTP GET per CpG: slow, dead in a container with no outbound network, and pinned to whatever was live that day. Reduced from 8.0M rows of summary statistics to the 6.3M CpG–trait pairs actually rendered.Cross-reactive probes (
cross-reactive-probes.rds, 140 KB) — probes mapping to more than one genome location, excluded from an EWAS. Moved out ofcomponents/app_methylome/inst/masking/, where it was a 2.7 MB CSV surviving behind a.gitignorenegation.Both in
inst/extdatarather thandata/: a LazyData object stays in the namespace for the life of the session once touched, and these are read once per fit.Build scripts in
data-raw/. The cross-reactive one records a gap — that CSV arrived pre-assembled with three source labels and no download URLs, so it converts what we have; the labels identify the papers but the exact supplements were never recorded.Needed by bigomics/omicsplayground#1875.