Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > ger.ct > #579807

Re: Nathan, die lunare Biopositronik (Perry Rhodan)

Show all headers | View raw

Am 27.12.2022 um 23:05 schrieb Stefan Froehlich:
> (Allerdings bin ich mir nicht sicher, dass ich ein System zur
> Beantwortung von Fragen verwenden möchte, welches mir zwar in
> geschliffenster Hochsprache und stets felsenfester Überzeugung
> antwortet, dessen Antworten aber keinerlei Gewähr auf Richtigkeit
> bieten können. Bevorzugter Einsatzzweck dürften derzeit einfache
> Pressemeldungen und einfache bis mittel komplexe Schulaufgaben
> sein).
Ach wieso. Wenn ich das richtig verstanden habe, reicht es immerhin für 
eine ausreichende Simulation medizinischer Kompetenz:

https://arxiv.org/abs/2207.08143

| Although large language models (LLMs) often produce impressive
| outputs, it remains unclear how they perform in real-world scenarios
| requiring strong reasoning skills and expert domain knowledge. We set
| out to investigate whether GPT-3.5 (Codex and InstructGPT) can be
| applied to answer and reason about difficult real-world-based
| questions. We utilize two multiple-choice medical exam questions
| (USMLE and MedMCQA) and a medical reading comprehension dataset
| (PubMedQA). We investigate multiple prompting scenarios: Chain-of-
| Thought (CoT, think step-by-step), zero- and few-shot (prepending the
| question with question-answer exemplars) and retrieval augmentation
| (injecting Wikipedia passages into the prompt). For a subset of the
| USMLE questions, a medical expert reviewed and annotated the model's
| CoT. We found that InstructGPT can often read, reason and recall
| expert knowledge. Failure are primarily due to lack of knowledge and
| reasoning errors and trivial guessing heuristics are observed, e.g.\
| too often predicting labels A and D on USMLE. Sampling and combining
| many completions overcome some of these limitations. Using 100
| samples, Codex 5-shot CoT not only gives close to well-calibrated
| predictive probability but also achieves human-level performances on
| the three datasets. USMLE: 60.2%, MedMCQA: 57.5% and PubMedQA: 78.2%

Google scheint das sogar noch besser zu können:

https://arxiv.org/abs/2212.13138

| Large language models (LLMs) have demonstrated impressive capabilities
| in natural language understanding and generation, but the quality bar
| for medical and clinical applications is high. Today, attempts to
| assess models' clinical knowledge typically rely on automated
| evaluations on limited benchmarks. There is no standard to evaluate
| model predictions and reasoning across a breadth of tasks. To address
| this, we present MultiMedQA, a benchmark combining six existing open
| question answering datasets spanning professional medical exams,
| research, and consumer queries; and HealthSearchQA, a new free-
| response dataset of medical questions searched online. We propose a
| framework for human evaluation of model answers along multiple axes
| including factuality, precision, possible harm, and bias. In addition,
| we evaluate PaLM (a 540-billion parameter LLM) and its instruction-
| tuned variant, Flan-PaLM, on MultiMedQA. Using a combination of
| prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on
| every MultiMedQA multiple-choice dataset (MedQA, MedMCQA, PubMedQA,
| MMLU clinical topics), including 67.6% accuracy on MedQA (US Medical
| License Exam questions), surpassing prior state-of-the-art by over
| 17%. However, human evaluation reveals key gaps in Flan-PaLM
| responses. To resolve this we introduce instruction prompt tuning, a
| parameter-efficient approach for aligning LLMs to new domains using a
| few exemplars. The resulting model, Med-PaLM, performs encouragingly,
| but remains inferior to clinicians. We show that comprehension, recall
| of knowledge, and medical reasoning improve with model scale and
| instruction prompt tuning, suggesting the potential utility of LLMs in
| medicine. Our human evaluations reveal important limitations of
| today's models, reinforcing the importance of both evaluation
| frameworks and method development in creating safe, helpful LLM models
| for clinical applications.

Wobei ich mir da mit mir noch nicht so ganz einig bin, ob das was über 
die Qualität des Sprachmodells oder über die Qualität der Medizin aussagt.

Auch hier dürfte gelten: Muß ja gar nicht perfekt sein. Reicht 
vollkommen, wenn es besser ist als ein menschlicher Arzt.

Und natürlich, wenn es billiger ist als ein menschlicher Arzt. Dann wird 
die Versuchung groß.
-- 
Neue Woche, neues Kondom.

Back to ger.ct | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Nathan, die lunare Biopositronik (Perry Rhodan) Dr. Joachim Neudert <neudert@5sl.org> - 2022-12-27 06:25 +0000
  Re: Nathan, die lunare Biopositronik (Perry Rhodan) Ulf Kutzner <Ulf.Kutzner@web.de> - 2022-12-26 23:18 -0800
    Re: Nathan, die lunare Biopositronik (Perry Rhodan) Dietz Proepper <dietz-usenet@rotfl.franken.de> - 2022-12-27 08:32 +0100
  Re: Nathan, die lunare Biopositronik (Perry Rhodan) Dietz Proepper <dietz-usenet@rotfl.franken.de> - 2022-12-27 08:28 +0100
    Re: Nathan, die lunare Biopositronik (Perry Rhodan) Stefan+Usenet@Froehlich.Priv.at (Stefan Froehlich) - 2022-12-27 08:02 +0000
      Re: Nathan, die lunare Biopositronik (Perry Rhodan) Dietz Proepper <dietz-usenet@rotfl.franken.de> - 2022-12-27 09:33 +0100
        Re: Nathan, die lunare Biopositronik (Perry Rhodan) Gerald Gruner <gerald314@yahoo.de> - 2022-12-27 16:08 +0100
          Re: Nathan, die lunare Biopositronik (Perry Rhodan) Ulf Kutzner <Ulf.Kutzner@web.de> - 2022-12-27 08:21 -0800
        Re: Nathan, die lunare Biopositronik (Perry Rhodan) Stefan+Usenet@Froehlich.Priv.at (Stefan Froehlich) - 2022-12-27 16:20 +0000
          Re: Nathan, die lunare Biopositronik (Perry Rhodan) Ulf Kutzner <Ulf.Kutzner@web.de> - 2022-12-27 08:24 -0800
          Re: Nathan, die lunare Biopositronik (Perry Rhodan) Dietz Proepper <dietz-usenet@rotfl.franken.de> - 2022-12-27 18:40 +0100
            Re: Nathan, die lunare Biopositronik (Perry Rhodan) Bernd Ohm <invalid@invalid.invalid> - 2022-12-27 22:45 +0100
              Re: Nathan, die lunare Biopositronik (Perry Rhodan) Dietz Proepper <dietz-usenet@rotfl.franken.de> - 2022-12-27 23:20 +0100
            Re: Nathan, die lunare Biopositronik (Perry Rhodan) Stefan+Usenet@Froehlich.Priv.at (Stefan Froehlich) - 2022-12-27 22:13 +0000
              Re: Nathan, die lunare Biopositronik (Perry Rhodan) Dietz Proepper <dietz-usenet@rotfl.franken.de> - 2022-12-27 23:34 +0100
                Re: Nathan, die lunare Biopositronik (Perry Rhodan) Stefan+Usenet@Froehlich.Priv.at (Stefan Froehlich) - 2022-12-27 22:54 +0000
          Re: Nathan, die lunare Biopositronik (Perry Rhodan) Martin Pochert <pochert.no.spam@gmx.de> - 2022-12-27 22:02 +0100
      Re: Nathan, die lunare Biopositronik (Perry Rhodan) "Dr. Joachim Neudert" <neudert@5sl.org> - 2022-12-27 09:41 +0100
      Re: Nathan, die lunare Biopositronik (Perry Rhodan) "Dr. Joachim Neudert" <neudert@5sl.org> - 2022-12-27 09:48 +0100
        Re: Nathan, die lunare Biopositronik (Perry Rhodan) Ulf Kutzner <Ulf.Kutzner@web.de> - 2022-12-27 00:51 -0800
        Re: Nathan, die lunare Biopositronik (Perry Rhodan) Stefan+Usenet@Froehlich.Priv.at (Stefan Froehlich) - 2022-12-27 15:55 +0000
          Re: Nathan, die lunare Biopositronik (Perry Rhodan) "Dr. Joachim Neudert" <neudert@5sl.org> - 2022-12-27 17:26 +0100
            Re: Nathan, die lunare Biopositronik (Perry Rhodan) Lars Gebauer <lgebauer@live.de> - 2022-12-27 18:32 +0100
            Re: Nathan, die lunare Biopositronik (Perry Rhodan) Fidel Sebastián Hunrichse-Lara <Fidel-Sebastian_Hunrichse_Lara@b.maus.de> - 2022-12-27 15:28 -0500
            Re: Nathan, die lunare Biopositronik (Perry Rhodan) Stefan+Usenet@Froehlich.Priv.at (Stefan Froehlich) - 2022-12-27 22:05 +0000
              Re: Nathan, die lunare Biopositronik (Perry Rhodan) Dr. Joachim Neudert <neudert@5sl.org> - 2022-12-28 06:24 +0000
              Re: Nathan, die lunare Biopositronik (Perry Rhodan) Michael Bode <m.g.bode@web.de> - 2022-12-28 07:51 +0100
                Re: Nathan, die lunare Biopositronik (Perry Rhodan) "Dr. Joachim Neudert" <neudert@5sl.org> - 2022-12-28 07:56 +0100
                Re: Nathan, die lunare Biopositronik (Perry Rhodan) Ulf Kutzner <Ulf.Kutzner@web.de> - 2022-12-27 23:18 -0800
                Re: Nathan, die lunare Biopositronik (Perry Rhodan) Peter Veith <veith@snafu.de> - 2022-12-27 23:22 -0800
              Re: Nathan, die lunare Biopositronik (Perry Rhodan) Lars Gebauer <lgebauer@live.de> - 2022-12-30 09:08 +0100
          Re: Nathan, die lunare Biopositronik (Perry Rhodan) Martin Ebert <mx300@gmx.net> - 2022-12-28 02:59 +0100
        Re: Nathan, die lunare Biopositronik (Perry Rhodan) Jörg Tewes <jogi1964@gmx.net> - 2022-12-27 21:18 +0100

csiph-web