Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > alt.comp.os.windows-10 > #182131

Re: Source Code: My Beginnings

Cross-posted to 2 groups.

Show key headers only | View raw

On Sat, 2/8/2025 7:45 AM, Newyana2 wrote:

>    This is the first I've heard of "NPU". I'm surprised the author
> didn't explain the acronym, or TOPS.

The "Neural Processing Unit" does Fused Multiply Add (FMA) for matrix math.
Think of rows and columns of a matrix, being multiplied, then added.

If you can do 10^12 of those FMA per second, that is 1 TOPS.

Intel, in a first generation ("unusable") NPU, put *10000* FMA
on a silicon die. Some FPGAs intended for this sort of work
(inference), have *3000* DSP cores. And similar to the CUDA
cores on a video card, the DSP cores are programmable. The
RTX 5090, has around *26000* cores that can run shader programs.

The game is all about multitudes of fairly simple, reproducible blocks.

But there is also a degree of deception going on here, in that
to have real time performance ("an AI that talks to you"), that
requires hardware that at scale is 100x what is on your desktop.

What you get on the desktop, is something that might be able
to do a DNN or an OpenCL type calculation. These are older techniques
and standards. Maybe the first desktop offerings (like... Recall),
those won't be Large Language Models. They'll be more purpose built
neural networks or static calculation approaches. Maybe if there
is any language model at all, the OCR that Recall does, is
buttressed by grammer checks (it recognizes language better
than it recognizes single letters in isolation). Recall is
using a "very similar" OCR to what is in the W11 SnippingTool
right now (the Text Action button).

On a desktop system, even with some degree of acceleration,
there just isn't the memory bandwidth or facilities for
real time LLM AI answers. You'll be waiting some number of
seconds, before your AI says "Hello! Maya". If you distill the
models down, and use "INT1" math to do the calc, it just
makes the model dumber and more mistake prone. Note that
this is not the counting the letter "R" in strawberry problem,
which is an architectural/approach issue.

I can see your home AI computer, being air gapped and
dedicated to doing Wikipedia-type things. "How do you
mix green paint?" "why don't you drive to the Home
Depot and ask that guy in the paint section?" That's
the kind of help I expect.

   Paul

Back to alt.comp.os.windows-10 | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Source Code: My Beginnings Book Review <invalid@invalid.com> - 2025-02-08 03:00 +0000
  Re: Source Code: My Beginnings Newyana2 <newyana@invalid.nospam> - 2025-02-07 23:10 -0500
    Re: Source Code: My Beginnings Paul <nospam@needed.invalid> - 2025-02-08 01:40 -0500
      Re: Source Code: My Beginnings Newyana2 <newyana@invalid.nospam> - 2025-02-08 07:45 -0500
        Re: Source Code: My Beginnings Paul <nospam@needed.invalid> - 2025-02-08 17:35 -0500
          Re: Source Code: My Beginnings Newyana2 <newyana@invalid.nospam> - 2025-02-08 19:29 -0500
            Re: Source Code: My Beginnings Paul <nospam@needed.invalid> - 2025-02-08 23:39 -0500
    Re: Source Code: My Beginnings Stan Brown <the_stan_brown@fastmail.fm> - 2025-02-08 05:03 -0800
      Re: Source Code: My Beginnings Newyana2 <newyana@invalid.nospam> - 2025-02-08 10:24 -0500
        Re: Source Code: My Beginnings sticks <wolverine01@charter.net> - 2025-02-09 17:17 -0600
        Re: Source Code: My Beginnings Daniel70 <daniel47@eternal-september.org> - 2025-02-15 21:30 +1100
    Re: Source Code: My Beginnings Chris <ithinkiam@gmail.com> - 2025-02-15 11:10 +0000
  Re: Source Code: My Beginnings Anton Shepelev <anton.txt@gmail.moc> - 2025-02-08 19:32 +0300
  Re: Source Code: My Beginnings MikeS <mikes@is.invalid> - 2025-02-10 21:15 +0000

csiph-web