And a false positive screening result is not innocuous: it incurs costs in a variety of different ways, including human health.
If you're otherwise healthy and would have a 1/1,000,000 chance of having the disease before the test, and then you test positive with a test that is 99% accurate, you are ~100x more likely to have the disease than before - but that's still only 1/10,000 - not at all 99% likely, even though the test was "99% accurate"
That said, I think with this knowledge the test still confers helpful information. I might decide to spend $1000 on an additional diagnostic, even knowing that I'm still very likely to be negative. Depends on how wealthy I am, and how serious the disease is, and what the treatments for it are.
The problem is that diagnostics aren't necessarily risk-free. For example, there's a non-zero risk of death while getting a colonoscopy, to the point that false positives from unnecessary testing can increase all-cause mortality for patients.
Late 20's Hispanic lady shows up in the ER with what they think is probably food poisoning. But they do a CT. Which shows changes in her liver which probably is a fatty liver. But they do a biopsy just in case. Biopsy results in a bleed which requires a transfusion and 4 days in the hospital. Biopsy result, fatty liver.
Surprisingly few ER docs anywhere in the world have even a rudimentary understanding of the risks of CT scanning patients. There's a lot of information around about this, but my own first hand (anecdotal) experience is that I've had ER docs try to convince me that it's basically the same as an X-Ray and act like I'm a crazy person when I explain that it's orders of magnitude higher and cumulative over a lifetime. On one hand, it's not their job to care about your long term health - they need to rule out an emergency and get you out the door as quickly as possible - but it's very concerning.
It's a bit like how general practitioners aren't taught about nutrition at all, so give out really poor advice for heart disease patients (the leading cause of mortality in Western economies).
The trial followed 25,000 adults from the US and Canada over a year, with nearly one in 100 getting a positive result. For 62% of these cases, cancer was later confirmed.
(It also had a false negative rate of 1%:)
The test correctly ruled out cancer in over 99% of those who tested negative.
Based on your quoted sections, we can infer:
1. About 250 people got a positive result ("nearly one in 100")
2. Of those 250 people, 155 (62%) actually had cancer, 95 did not.
3. About 24,750 people got a negative test result.
4. Assuming a false negative rate of 1% (the quote says "over 99%") it means of those 24,750 people, about 248 actually did have cancer, while about 24,502 did not.
When you write it out like that (and I know I'm making some rounding assumptions on the numbers), it means the test missed the majority of people who had cancer while subjecting over 1/3 of those who tested positive to fear and further expense.
Nope, there is another important thing that matters: some of the cancers tested are really hard to detect early by other means, and very lethal when discovered late.
I would not be surprised if out of the 155 people who got detected early, about 50 lives were saved that would otherwise be lost.
That is quite a difference in the real world. Even if the statistics stays the same, the health consequences are very different when you test for something banal vs. for pancreatic cancer.
This is a bizarre thing to say in response to... a clear statement of the positive and negative predictive value. PPV is 62% and NPV is "over 99%".
Your calculations don't appear to have any connection to your criticism. You're trying to back into sensitivity ("the test missed the majority of people who had cancer") from reported PPV and NPV, while complaining that sensitivity is misleading and honest reporting would have stated the PPV and NPV.
Reading the article, I'm still not sure about the accuracy, and don't have the time to carefully parse the whole article. I see at least the following statements (there may be more):
The trial followed 25,000 adults from the US and Canada over a year, with nearly one in 100 getting a positive result. For 62% of these cases, cancer was later confirmed.
and
The test correctly ruled out cancer in over 99% of those who tested negative.
They're "trained to understand this type of stuff" in the sense that it will get a mention in medical school. Overwhelmingly, they aren't "trained to understand this type of stuff" in the sense that if you pose them a simple problem of this type, they'll be able to calculate the answers.
> Next, suppose I told you that most doctors get the same wrong answer on this problem – usually, only around 15% of doctors get it right. (“Really? 15%? Is that a real number, or an urban legend based on an Internet poll?” It’s a real number. See Casscells, Schoenberger, and Grayboys 1978; Eddy 1982; Gigerenzer and Hoffrage 1995; and many other studies. It’s a surprising result which is easy to replicate, so it’s been extensively replicated.)
Edit: There is in fact another comment on this thread of someone doing exactly this: https://news.ycombinator.com/item?id=45652535
Also, these tests are a grand a pop if I'm reading it directly (which I may not be)
Running the test using multiple different labs helps eliminate contamination and handling errors but most false positives are due to genetics and long term environmental factors.
The trial followed 25,000 adults from the US and Canada over a year, with nearly one in 100 getting a positive result. For 62% of these cases, cancer was later confirmed.
Lead researcher Dr Nima Nabavizadeh, associate professor of radiation medicine at Oregon Health & Science University, said the data showed that the test could "fundamentally change" their approach to cancer screening.
A year ago:
Galleri promises to detect multiple cancers—but new evidence casts doubt on this much hyped blood test (August 2024))
The NHS is currently running a £150m trial of the test, funded by Grail and involving more than 100 000 participants in England.
...
Documents leaked to The BMJ indicate that the criteria being used, unpublished until now, are unsuitable to justify a new national screening programme aimed at saving lives.
They show that even Mike Richards, the chair of the independent UK National Screening Committee, has privately voiced “serious concerns” to Amanda Pritchard, NHS England’s chief executive, about the trial and its ability to provide sufficient evidence “on whether the benefits of testing outweigh any potential harms and at reasonable cost.”
Company site: https://www.galleri.com/what-is-galleri/types-of-cancer-dete...
Forgive me if I'm wrong, but isn't this the textbook example of understanding false negatives in testing people at scale?
That said, at what level of risk of follow up diagnostic would you baulk? Any procedure which requires a general is bad news, and if you are over 70 its a lot more bad.
It's not perfect but it's easy/fast and a good way to screen for big problems.
1) You could develop cancer tomorrow, notice the symptoms, and assume it's not worth getting screened again because you were just supposedly cleared of cancer. A common logical fallacy for our human brains is that we think, "Oh, I just got a test, so it's less likely I have cancer today," which is not how probability works.
2) You could have gotten a false positive, which would have led to unnecessary additional screening. Many methods of cancer screening have some risk, whether from anesthesia, infection, or further false positives leading to unnecessary treatment.
Given my interactions with my doctors and their tests, they did seem to hold this belief, too.
In any case, the real mistake people make is failing to update their beliefs when new evidence (symptoms) appears. Rookie mistake. My doctors love to do (or rather, not do) this. Who needs differential diagnosis and re-evaluation when they can just keep the diagnosis and continue receiving a fuckton of money after you based on a secore-based system we have here?
I am not going to avoid any reasonable treatment/screen because of it. It was intended to catch asymptomatic cancer. Additional invasive screenings are voluntary and like all treatments they carry risk. I weigh all treatments based on their risks at the time.
For everyday people increased screening of all types has risks, but overall the benefits massively outweigh the risks. If I was a frail 80yo, I might see the risk profile differently.
In my career I've encountered many people who "don't want to know" about medical tests of any kind. I'm not one of those people. Minimally invasive screens early and often please.
This is just not how math works, and it's why we still need doctors to order tests -- to protect people from themselves. You clearly don't know what you don't know, but you have a huge amount of confidence that you do, apparently.
Here's a list of different types of cancer screenings and where the risk/benefit falls: https://publichealth.jhu.edu/2023/balancing-the-benefits-and...
The risk of any cancer screening has to be calculated with variables like:
- how risky is the test?
- what are the risks of a false positive?
- how does a true negative affect the person's behavior in the future?
- what is the likelihood that the patient has asymptomatic cancer, based on risks like genetics and age?
- how difficult is the cancer to treat in different stages?
Without looking at all of those things, you don't know if the test is going to increase or decrease all-cause mortality risk.
That's exciting. The first step of many. The bar is so low because it remains to be seen if this earlier detection prevents deaths. And of course we still don't know the root causes or triggers, which promotes the same thing occurring a few years later. Remission is not solved, we need cure still.
I’m about to order it.
If they don’t take the test, then presumably they also think they don’t have cancer.
I suspect that this will remain a niche product. It would, in theory, be applicable to most people on earth. Such wide spread screenings simply aren't viable in a world of massive compliance costs and subsidized health care.
It took an act of god for simple COVID testing to become somewhat inexpensive.
I look forward to getting this test every few years until it is killed by our regulatory apparatus.
The article made it read like it was some trial that just completed.
Does anyone know if (or when) these are regularly available in Canada? And the costs associated?
There are two regulatory regimes for diagnostics in the US:
1) Lab developed tests (LDTs) licensed under the CLIA legislation (I think from the 1980s). These are verified by a Lab Director with a professional license in diagnostics, that allows them to sign out clinical results from the test. There are professional organizations that perform regular inspections of the lab, its condition, its paperwork, its tests, the SOPs, and the internal validations that have been performed at the lab director's direction to assess performance of the tests. These are limited to a single site, the kits for the test can not be sold except for Research Use Only, and if a second site wants to start doing a similar test the lab director at the other site needs to do all the same validation all over again as at the first site.
2) FDA approval for diagnostic medical devices. These can be simple and straightforward for Class 1/2 devices, which do not directly provide medical advice but mere physical readouts (to greatly bastardize the distinction between Class 1/2 and 3). Or the device approval can be quite complex for Class 3 devices, and would require a huge trial like the one described here. If you want to sell the device for others to use, rather than just testing as service, you want to go this route. Though there are still single-site "devices" especially for DNA sequencing tests, that want the FDA label.
Neither of these will result in getting reimbursement for a test. For that, you need to pursue coverage determinations from all the payers, basically one on one. For complex sequencing tests like this that mostly affects older populations, getting CMS coverage (Medicare) can pave the path for others. For other conditions... well... get all your trials and papers together and hope that your patient population is super sympathetic or you can show the insurance company some savings.
A large clinical trial like this one can help with getting coverage for the test, but it has to either show a big medical benefit, or show economic benefits within five years for the payor. Or ideally both. Early cancer detection has potential for this, but I have not heard optimistic things up until now at Galleri's chance for reimbursement any time soon.
I paid the $950 rate (it’s occasionally discounted to $800, such as now until the end of the year, and you might be able to use FSA/HSA funds depending on plan administrator) and thought it was worth it (to detect potentially asymptomatic early stage cancer).
(no affiliation)
It’s not a false promise — rather, a technology that is still far from perfect because of existing technical barriers.