[Paper] TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records (2023)

15 Feb 2024     #bio #ehr #transformer

Yang, Zhichao, et al. “TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records.” Nature Communications 14.1 (2023): 7857.

Paper Link

Points

1. New pre-training objective: predicting all diseases or outcomes of a future visit

   • Helps the model uncover the complex interrelations among different diseases and outcomes

2. TransformEHR, the generative encoder-decoder framework to predict patients’ ICD codes using their longitudinal EHRs

3. Validation of the generalizability using both internal and external datasets

   • Demonstrated a strong transfer learning capability of the model

   • Could be great with limited data and computing resources

Background

• Longitudinal electronic health records (EHRs) have been successfully used to predict clinical diseases or outcomes (congestive heart failure, sepsis mortality, mechanical ventilation, septic shock, diabetes, PTSD, etc.)
• With the availability of large cohorts and computational resources, deep learning (DL) based models outperform traditional machine learning (ML) models (Med-BERT, BEHRT, BRLTM, etc.)
• The existing pre-training tasks were limited in predicting a fraction of ICD codes within each visit :arrow_right: A novel pre-training strategy, which predicts the complete set of diseases and outcomes within a visit, might improve clinical predictive modeling

Method

Data

*VHA: Veterans Health Administration, the largest integrated healthcare system in the US, providing care at 1,321 healthcare facilities

Pre-training data: around 6M patients who received care from more than 1,200 facilities of the US VHA

• Two common and uncommon disease/outcome agnostic prediction (DOAP) datasets

  • ICD-10CM codes with more than a 2% prevalence ratio for common dataset
  • Those with a 0.04%-0.05% prevalence ratio for uncommon dataset

• Non-VHA dataset: from MIMIC-IV dataset (29,482)

  • Only selected objects with ICD-10CM records to match the cohorts from VHA

Longitudinal EHRs

• Include demographic information (gender, age, race, and marital status) and ICD-10CM codes as predictors

• Group ICD codes at the visit level

• Order the codes by priority, where the primary diagnosis is typically given the highest priority

• Form multiple visits as a time-stamped input of a sequence by date of visit

Embeddings

Multi-level embeddings: visit embeddings + time embeddings + code embeddings

• Time embeddings: embed days difference as relative time information by getting the difference between a certain visit and the last visit in the EHR
  • Includes the date of each visit to integrate temporal information, not only sequential order
  • Date is important as the importance of predictor in a visit can vary over time

Model Architecture

Encoder-decoder transformer-based architecture

• Encoder: performs cross-attention, unlikely BERT, over representations and assigns an attention weight for each representation
  • Cross-attention is implemented by masking the complete set of ICD codes of a future visit as shown in Fig. 2b
• Decoder: generates ICD codes of the masked future visit with the weighted representations from the encoder
  • Generates the codes following the order of code priority within a visit

Evaluation

Metrics: PPV (precision), AUROC, AUPRC

Baseline models: logistic regression, LSTM, BERT without pre-training, BERT with pre-training   # what’s the objective when pre-training BERT? MLM or the objective proposed in this paper?

Pre-training

Task: Disease or outcome agnostic prediction (DOAP); Predicting the ICD codes of a patient’s future visit based on longitudinal information up to the current visit

Ablation study

1. Visit masking vs. code (part of visit) masking for an encoder-decoder model
   • Visit masking performed better; pre-training of all diseases outperform traditional pre-training objective (2.52-2.96% in AUROC)
2. Encoder-decoder vs. encoder-only (BERT) on DOAP
   • ​Encoder-decoder outperformed; 0.74-1.16% in AUROC   # the possibility if the parameter size affected this result?
3. Time embeddings O vs. X
   • The model with the time embeddings outperformed moderately; 0.43 in AUROC
   • Days difference is more effective than specific date as the embeddings

Fine-tuning

Tasks: the pancreatic cancer onset prediction (Table 3) and intentional self-harm prediction in patients with PTSD (Table 4)

• TransformEHR outperforms on the both tasks

• AUPRC was consistent when using different set of demographics

• Results with all visits was better than with recent few(five) visits
• In generalizability evaluation,
  • When testing with internal dataset which is included data from VHA facilities not used for pre-training, there’s no statistical difference in AUPRC on the intentional self-harm prediction task among PTSD

[Study] white matter hyperintensity (WMH)와 아밀로이드 베타 (β-amyloid) 및 타우 축적 양상 간의 관계 논문 스터디

31 Jan 2024     #bio #brainImaging #demensia #atn

WMH: 뇌 백질 변성, 피질인 회백질 간 통로 역할을 하는 수질인 백질이 뇌혈관 변성에 의해 확장되어 손상된 형태. 보통 MR FLAIR 이미지를 통해 확인

References

WMH와 Alzhheimer’s disease biomarker인 Amyloid beta, Tau 간의 관계를 조사하기 위해 다음의 세 논문을 읽었습니다.

Alban, Sierra L., et al. “The association between white matter hyperintensities and amyloid and tau deposition.” NeuroImage: Clinical 38 (2023): 103383. link

Graff-Radford, Jonathan, et al. “White matter hyperintensities: relationship to amyloid and tau burden.” Brain 142.8 (2019): 2483-2491. link

Hedden, Trey, et al. “Cognitive profile of amyloid burden and white matter hyperintensities in cognitively normal older adults.” Journal of Neuroscience 32.46 (2012): 16233-16242. link

Keys

위 세 논문에 따르면

• WMH와 Amyloid beta의 축적 양상 간의 관계는 아직 명확하게 밝혀진 바가 없다: 관련성의 유무부터 관련된다면 어떻게 관련되는지까지

• 점차 WMH와 Ab간의 관계를 밝히려는 연구가 늘어나고 있는 추세이다

• WMH와 Tau의 축적 양상 간에는 관계성이 없다

---

The association between white matter hyperintensities and amyloid and tau deposition (2023)

Abstract

… Finally, the regions where β-amyloid and WMH count were most positively associated were the middle temporal region in the right hemisphere (r = 0.18, p = 0.002) and the fusiform region in the left hemisphere (r = 0.017, p = 0.005). β-amyloid and WMH have a clear association, though the mechanism facilitating this association is still not fully understood. The associations found between β-amyloid and WMH burden emphasize the relationship between β-amyloid and vascular lesion formation while factors like CVRFs, age, and sex affect AD development through various mechanisms

Data

The subset of ADNI-3 participants who had all the T1-weighted, 3D FLAIR, Amyloid, and Tau PET modalities available

Snippets

• The percentage of white matter volume occupied by WMH was significantly and positively correlated with β-amyloid PET SUVR (Fig. 1; r = 0.28, p < 0.001). We observed WMH volume to significantly predict global amyloid accumulation when controlling for age, sex, years of education, and scanner manufacturer (F(1, 309) = 13.9, p = 0.0002).
• The correlational analyses were repeated after the log transformation of WMH volume and outcomes remained the same, resulting in a significant positive correlation between WMH volume and β-amyloid (r = 0.24, p = 4.9e-5), and a nonsignificant positive correlation between WMH volume and meta-temporal tau (r = 0.09, p = 0.12).
• The inclusion of MOCA, MMSE, and Global CDR, as covariates, did not change the significant relationship between WMH volume and β-amyloid. We observed a significant effect of hippocampal volume fraction on WMH volume (F(1, 580) = 16.9, p = 4.5e-5) and β-amyloid (F(1, 309) = 32.5, p = 2.8e-8).
• WMH volume percent of participants with either amyloid (A+) or tau (T+) pathology was higher than controls (A-/T-) (Fig. 2). We observed a significantly higher WMH percent in AD pathology participants (A+/T+) compared to controls (A-/T-) (p = 0.007, Cohen’s d = 0.4, t = -2.5). … No significant association was found in the A-/T- group (r = 0.06, p = 0.45). A significant positive correlation was observed between β-amyloid SUVR and WMH count in the A+/T+ group only.
• WMH count was used as another method of measuring WMH burden. … Both statistical tests on the WMH volume and the WMH count showed similar results, confirming that WMH count is an accurate measure to use alongside WMH volume. Correlations of WMH count with β-amyloid within A/T pathological groups also paralleled the WMH volume analysis result.
• Our regional analysis showed that β-amyloid and WMH accumulation in the precentral, cuneus, fusiform, isthmus cingulate, lateral occipital, lingual, superior parietal, and supramarginal regions were most significantly associated across all pathological groups when averaged across hemispheres. Variations in the locations of increased WMHs are indicative of AD and its phase of progression, some of these regions being more implicated in cognitive decline than others.

• We observed neither a correlation nor an association between WMH and Tau uptake in the entire cohort (Fig. 4: correlation p = 0.25; association p = 0.4, controlling for age, sex, years of education, and scanner manufacturer).

• Additionally, WMH volume was only predicted by CN and MCI diagnoses, not AD. The relationship between WMH volume often predicts AD in the preclinical stages, likely accounting for the relationship we observed.

• The inclusion of cognitive scores MMSE, MOCA, and Global CDR had no effect on the associations found between β-amyloid and WMH volume, therefore not significantly impacting this relationship. In the literature, worse performance on these cognitive tests has been associated with increased WMH volume (Wang et al., 2020).

• The relationship between higher MMSE scores and increased WMH volume showed significance with multiple comparison corrections while controlling for age and sex in the isthmus cingulate region

• For Global CDR scores, this spatial distribution was significant in the isthmus cingulate, temporal pole, and pars triangularis

• Lastly, the MOCA scores showed significant positive spatial relationships in the isthmus cingulate, linguistic, and temporal pole regions

• We also did not observe any significant effect of APOE-ε4 presence on the established relationship between β-amyloid and WMH volume or WMH count. … Although, we did have a small sample size of individuals with the homozygous APOE-ε4 genotype.

---

White matter hyperintensities: relationship to amyloid and tau burden (2019)

Abstract

White matter hyperintense volumes in the detected topographic pattern correlated strongly with lobar cerebral microbleeds (P < 0.001, age and sex-adjusted Cohen’s d = 0.703). In contrast, there were no white matter hyperintense regions significantly associated with increased tau burden using voxel-based analysis or region-specific analysis, among non-demented elderly, amyloid load correlated with a topographic pattern of white matter hyperintensities. Further, the amyloid-associated, white matter hyperintense regions strongly correlated with lobar cerebral microbleeds suggesting that cerebral amyloid angiopathy contributes to the relationship between amyloid and white matter hyperintensities.

Data

• Participants, aged 50 to 89, were enrolled in the Mayo Clinic Study of Aging (MCSA), a population-based study of Olmsted County, Minnesota residents.
• 434 non-demented participants with FLAIR-MRI, tau-PET (AV-1451), and Pittsburgh compound B (PiB)-PET (amyloid) scans to assess the relationship between FLAIR WMH and Alzheimer’s disease pathologies.

Snippets

• In the study of non-demented individuals, we found that amyloid burden measured by PET was associated with a topographic pattern of WMH. These amyloid-related WMH regions were associated with lobar CMBs suggesting that regional changes correlate with CAA. We found no evidence to support an association between tau burden and WMH burden.

• We did not detect an association between tau burden and WMH in either the voxel-level analyses or region-level analyses.

---

Cognitive Profile of Amyloid Burden and White Matter Hyperintensities in Cognitively Normal Older Adults (2012)

Abstract

Amyloid burden and WMH were not correlated with one another. Age was associated with lower performance in all cognitive domains, while higher estimated verbal intelligence was associated with higher performance in all domains. Hypothesis-driven tests revealed that amyloid burden and WMH had distinct cognitive profiles, with amyloid burden having a specific influence on episodic memory and WMH primarily associated with executive function but having broad (but lesser) effects on the other domains. These findings suggest that even before clinical impairment, amyloid burden, and WMH likely represent neuropathological cascades with distinct etiologies and dissociable influences on cognition.

Data

• 168 (95 female) cognitively normal, community-dwelling older adults (aged 65–86, M=73.24, SD=5.80).

• Participants in the Harvard Aging Brain Study, an ongoing longitudinal study currently in the baseline assessment phase

• Because of the staged nature of the visits (all baseline visits must be completed within 6 months), positron emission tomography (PET) amyloid imaging and magnetic resonance imaging (MRI) estimates of WMH were currently available for 109 of the older adults.

[Paper] BEHRT: Transformer for Electronic Health Records (2020)

22 Jan 2024     #bio #ehr #transformer

Li, Yikuan, et al. “BEHRT: transformer for electronic health records.” Scientific reports 10.1 (2020): 7155.

Paper Link

Points

1. BEHRT (BERT for EHR): BERT-based architecture

   • In EHR, certain diseases can be reversed, or the time interval between two diagnoses can be shorter or longer than recorded.

     → Bidirectional contextual awareness of the model’s representation is a big advantage with EHR data.

2. Transfer Learning: pre-training on predicting of masked disease words, such as Masked Language Modeling (MLM), and then fine-tuning on three disease prediction tasks
3. Disease embeddings: show the relations between the various diseases

Background

• In traditional research on EHR data, individuals are represented by models as features. Experts had to define the appropriate features.

• Studies applying Deep Learning (DL) to EHR started to show that DL models can outperform the traditional Machine Learning (ML) methods
• With DL architectures for sequence data, such as Recurrent Neural Networks (RNNs), the application of DL models for EHR was improved in terms of capturing the long-term dependencies among events.
• Similarities between sequences in EHR and natural language lead to the successful transferability of techniques

Method

Data

Clinical Practice Research Datalink (CPRD)

• one of the largest linked primary care EHR systems

• Contains longitudinal data from a network of 674 general practitioner practices in the UK

• from ​8 million patients (eligible for linkage to HES, meet CPRD’s quality standards) to 1.6 million patients (having at least 5 visits in the EHR)

• Only the data from GP practices considered in this study, which consented to record linkage with HES

Input Features

• Only consider the diagnoses and ages

The patient $p$’s EHR: \[ V_p={v^1_p},{v^2_p},{v^3_p}, …,{v^n_p} \]

The patient $p$’s EHR of the $j$th visit: $v^j_p$

$v^j_p$ is a list consisting of single or multiple $m$ diagnoses: $v^j_p={d_1, …d_m}$

Input sequence: \[ I_p={CLS, v^1_p, SEP, v^2_p, SEP,…,v^{n_p}_p, SEP} \]

• $CLS$ token: the start of a medical history

• $SEP$ token: between visits

Embeddings

• A combination of 4 embeddings: disease + position + age + visit segment

• Disease embeddings: past diseases can improve the accuracy of the prediction for future diagnoses

• Positional encodings: determine the relative position in the EHR sequence

Tasks

Pre-training: prediction of masked disease words (MLM)

• 86.5% of the disease words were unchanged, 12% of them replaced with the mask token, and the remaining 1.5% words replaced with randomly-chosen disease words

Fine-tuning: 3 different disease prediction tasks

• Prediction of diseases in the next visit (T1)

  1. Randomly choose an index $j (3<j<n_p)$ for each patient
  2. Form input as $x_p={v^1_p, …, v^j_p}$
  3. Output $y_p=w_{j+1}$, $w_{j+1}$ is a multi-one-hot vector, indexed for disease that exist in $v^{j+1}_p$
  • Each patient contributes only an input-output pair to the training and evaluation process.

• Prediction of diseases in the next 6 months (T2) & in the next 12 months (T3)

  1. Not include patients that don’t have 6 or 12 months worth of EHR in the analysis
  2. choose $j$ randomly from $(3, n*)$, where $n*$ is the highest index after 6 or 12 months
  3. output $y_p=w_{6m}$ and $y_p=w_{12m}$

• The number of patients for each task: 699K, 391K, 342K

Results

Disease Embeddings

T-SNE Results

• The natural stratification of gender-specific diseases

• The natural clusters are formed that in most cases consist fo disease of the same chapter

• 10 closest diseases by cosine similarity of their embeddings are founded as similar as those provided by a clinical researcher (0.757 overlab) - Supplementary table S3

Attention and Interpretability

Found the relationships among events which goes beyond temporal/sequence adjacency

Analysed the attention-based patterns by Vig

• Strong connections between rheumatoid arthritis and enthesopathies and synovial disorders → Attention can go beyond recent events and find long-range dependencies among diseases

Disease Prediction

Metrics: average precision score(APS), AUROC

• APS: a weighted mean of precision and recall achieved at different thresholds

1. Performance scores

   

   → Outperformed the best model by more than around 8% in predicting for a range of more than 300 diseases

2. Comparing performance for each disease

   APS and AUROC scores with the all \(y_p\) and \(y*_p\) vectors of a disease

Discussion

• Flexibility of BEHRT from employed 4 key concepts from EHR: disease, age, segment, position
  • gains insights about underlying generating process of EHR
  • Distributed/complex representations that are capable of capturing concepts of disease
  • Future work: adding new concepts to the embeddings
• Disease embeddings provide great insights into how various diseases are related to each other: co-occurrence and closeness of diseases
  • Could be used for future research as reliable disease vectors

• Important features of EHR for prediction

  • Robust, gender-specific predictions without inclusion of gender

  • Position was important

  • Age embeddings might be vital in diagnosing age-related diseases

[LearnMRI] MRI로 관찰되는 알츠하이머 치매 관련 혈관성 변화

26 Dec 2023     #bio #brainImaging #mri

Amyloid beta (A$\beta$)가 침착되면 Alzheimer’s demensia가 발병될 확률이 높아진다. A$\beta$ 침착에 의한 뇌 손상 이전에 혈관성 변화들이 관찰 되는데, 혈관성 변화는 생활 습관 개선이나 약물 치료로 개선될 수 있다. 다양한 형식의 MRI를 통해 관찰할 수 있는 혈관성 변화는 여러가지가 있다.

Choroid plexus (ChP)

• MRI type: T1-weighted image (T1WI)
• ChP: 뇌실(ventricle)에서 발견되는 혈관과 세포의 네트워크
  • 혈액-뇌 사이에서 면역세포의 관문 역할을 한다.
  • 뇌척수액(cerebrospinal fluid; CSF)을 생산하여 뇌세포의 노폐물 및 독성 제거에 도움을 준다.
• ChP의 부피가 클수록 인지 장애 정도가 심해진다.

Perivascular space (PVS)

• MRI type: T2-weighted image (T2WI)
• PVS: 뇌를 관통하는 세동맥을 둘러싸고 있는 공간
  • 혈관벽 내 및 주위의 체액, 세포 조직 등을 아우르는 개념이다.
  • Blood-brain barrier (BBB)의 한 구성 요소이다.
• Fluid dynamic
  • CFS와 간질액(interstitial fluid; ISF) 사이 네트워크 역할을 해 대뇌의 부산물을 정화한다.
  • 뇌 속 림프계의 역할을 한다고 알려져 있다. → glymphatic system
• PVS의 확장(Enlarged PVS; EPVS)은 퇴행성뇌질환 및 다양한 뇌 질병과 관련이 있다.
• PVS volume과 Amyloid beta (A$\beta$) 양성 확률은 비례한다.
  • Temporal lobe에서 경향성이 두드러진다: Temporal PVS를 A$\beta$ 양성 환자들이 많이 가지고 있다.

White matter hyperintensity (WMH)

• MRI type: FLAIR
• WMH: 백색물질 과집중
  • 노화가 주 원인으로, 혈류의 상실, 세포 손상으로 나타난다.
  • Subcortical hyperintensity: basal ganglia 근처의 WMH
• 알츠하이머 치매와 직접적인 연관이 있다고 아직 확정되지는 않았으나, 환자들을 관찰했을 때 A$\beta$ 양성과 WMH volume 간 비례 관계가 있는 것으로 보인다고 한다.

[LearnMRI] MR Modality 종류와 관찰 가능한 뇌 양상

26 Dec 2023     #bio #brainImaging #mri

Spin echo 기법에 따라 T1 강조 영상(T1 weighted image; T1WI)과 T2 강조 영상(T2 weighted image; T2WI)을 얻을 수 있다. 이들에 영상 조작을 더하여 다양한 MR modality 영상을 생성할 수 있다. 영상마다 병변 조직의 신호 강도(signal intensity)가 달리 나타나므로, 강조되는 병변의 종류가 다르다.

T1 강조 영상 (T1-weighted image; T1WI)

• Spin echo: TR과 TE를 모두 짧게 준다
  • TR이 짧아지면 조직마다 다른 $Mz$의 90도 pulse 회복 시간(T1 relaxation time)이 강조된다. 어느 조직은 모두 회복되고, 어느 조직은 덜 된 상태로 두 번째 pulse의 영향을 받으면, 전자는 영향을 많이, 후자는 덜 받게 된다. 이 차이가 영상에 반영 된다.
  • TE를 짧게 주어야 T2 relxation time 값에 영향을 덜 줌으로써 이를 통제할 수 있다.
• Signal intensity:
  • Signal intensity가 T2-weighted image보다 높아 해부학적(anatomical) 구조물이 더 명확히 구별된다.
  • 피하지방(fat) 및 혈액(blood)은 밝게(hyperintense), 근육은 중간 정도의 밝기로, 수분(water)은 어둡게(hypointense) 보인다.
  • 골수(marrow)는 지방이 풍부하므로, 골피질(cortex)은 수분이 적으므로 hyperintense하게 나타난다.
  • 병변: 지방종, 아급성 출혈, 고단백질 함유 병소(점액낭종(mococele) 등)
• Observe: 뇌 피질 형태(anatomical detail), 혈관 변화(vascular changes), 혈뇌장벽(blood-brain barrier) 손상 여부
• Feature: cortical thickness, choroid plexus (ChP)

T2 강조 영상 (T2-weighted image; T2WI)

• Spin echo: TR과 TE를 모두 길게 준다
  • TR을 길게 주어 T1 relaxation time에 영향을 덜 준다.
  • TE가 길수록 조직마다 $Mxy$가 감소되는 수준의 차이가 강조되어 영상에 달리 표현된다.
• Signal intensity:
  • 수분(water)이 hyperintense한데, 병적 조직은 수분을 많이 함유하므로 병소(lesion)가 쉽게 관찰된다.
    • 대부분의 병변은 T1에서는 저신호, T2에서는 고신호로 나타난다.
    • 수분 함유량이 많은 조직일 수록 hyperintense하다: 낭종 > 부종 > 정상 조직 순으로 밝게 보인다.
  • 뇌척수액(cerebrospinal fluid; CSF)도 hyperintense하여 병변과 구분이 잘 되지 않는다.
  • Muscle, fat, blood는 hypointense하다.
• Observation: 병소(lesion), hypo 병변 (급성혈종, 진균구 등), 동맥 (정맥은 혈류 속도가 느려 환자마다 신호 강도가 다르다)
• Feature: Perivascular space (PVS)

FLAIR (Fluid Attenuation Inversion Recovery)

• T2에서 CSF를 까맣게 처리한 이미지이다.
• Signal intensity: Non-free-flowing water는 hyperintense, fat은 hypointense로 나타난다.
• Observation: 뇌실(ventricle) 부근의 병소, 부종(edema, 부종은 정체된 수분이라 밝게 나타난다), 회백질의 차이(grey-white differentiation)
• Feature: lesion, white matter hyperintensity (WMH)

GRE (Gradient Echo; T2*)

• Signal intensity: 상자성체 물질(paramegnetic substances, blood, calcium, metal 등)이 hyperintense로 나타나, 철의 침착과 관련된 사항을 관찰하기 용이하다.
• Observation: 뇌출혈 초기/말기, 미만성 축삭손상(diffuse axonal injury)에서 보이는 microbleed 검출에 탁월
  • *diffuse axonal injury: 뇌진탕 등 뇌손상의 구성요소 중 하나, 축삭 손상으로 인해 외상 후 혼수상태를 보이는 특징이 있다.
• Feature: bleeding