NPUs (Neural Processing Units) are the new normal (Was: 25-30% is insane, Neural Network Branch Prediction?)

From	Mild Shock <janburse@fastmail.fm>
Newsgroups	comp.lang.prolog
Subject	NPUs (Neural Processing Units) are the new normal (Was: 25-30% is insane, Neural Network Branch Prediction?)
Date	2025-10-15 16:04 +0200
Message-ID	<10co9kp$12u6u$1@solani.org> (permalink)
References	<107t0fi$6bgl$1@solani.org> <10cmqea$s93$1@solani.org> <10cn157$10qq$1@solani.org>

Show all headers | View raw

Hi,

It seems I am having problems pacing with
all the new fancy toys. Wasn't able to really
benchmark my NPU from a Desktop AI machine,

picked the wrong driver. Need to try again.
What worked was benchmarking Mobile AI machines.
I just grabbed Geekbench AI and some devices:

USA Fab, M4:

     sANN    hANN    qANN
iPad CPU    4848    7947    6353
iPad GPU    9752    11383    10051
iPad NPU    4873    36544    *51634*

China Fab, Snapdragon:

     sANN    hANN    qANN
Redmi CPU    1044    950    1723
Redmi GPU    480    905    737
Redmi NNAPI    205    205    469
Redmi QNN    226    226    *10221*

Speed-Up via NPU is factor 10x. See the column
qANN which means quantizised artificial neural
networks, when NPU or QNN is picked.

The mobile AI NPUs are optimized using
mimimal amounts of energy, and minimal amounts
of space squeezing (distilling) everything

into INT8 and INT4.

Bye

Mild Shock schrieb:
> Hi,
> 
> The change from 378 ms to 286 ms is around 25-30%
> is insane. But I did both tests on a novel AI CPU.
> To be precise on a AMD Ryzen AI 7 350.
> 
> But somehow I picked up rumors that AI CPUs now
> might do Neural Network Branch Prediction. The
> idea seems to exist in hardware at least since (2012):
> 
> Machine learning and artificial intelligence are
> the current hype (again). In their new Ryzen
> processors, AMD advertises the Neural Net
> Prediction. It turns out this is was already
> used in their older (2012) Piledriver architecture
> used for example in the AMD A10-4600M. It is also
> present in recent Samsung processors such as the
> one powering the Galaxy S7. What is it really?
> https://chasethedevil.github.io/post/the_neural_network_in_your_cpu/
> 
> It can be done with Convoluted Neural Networks (CNN):
> 
> BranchNet: A Convolutional Neural Network to
> Predict Hard-To-Predict Branches
> To this end, Tarsa et al. proposed using convolutional
> neural networks (CNNs) that are trained at
> compiletime to accurately predict branches that
> TAGE cannot. Given enough profiling coverage, CNNs
> learn input-independent branch correlations.
> https://microarch.org/micro53/papers/738300a118.pdf
> 
> Interstingly the above shows cases a PGO based
> Machine Learning for Branch Predictors. No clue
> how they construct the CPU, that they can feed
> 
> it with offline constructed neural neutworks for
> their own execution. Maybe an optimizer uses it?
> But I guess a more modern  solutions would not only
> 
> use CNN, but also an Attention Mechanism.
> 
> Bye
> 
> Mild Shock schrieb:
>> Hi,
>>
>> I spent some time thinking about my primes.pl
>> test. And came to the conclusion that it
>> mainly tests the Prolog ALU. Things like
>>
>> integer successor or integer modulo. Then
>> I found that Java has Math.floorMod() which
>> I wasn't using yet. And peng results are better:
>>
>> /* Dogelog Player 2.1.2 for Java, today */
>> ?- time(test).
>> % Zeit 286 ms, GC 1 ms, Lips 26302430, Uhr 15.10.2025 02:31
>> true.
>>
>> Maybe the Java backend picks a CPU instruction
>> for Math.floorMod() instead of executing the
>> longer code sequence that is needed to correct
>>
>> rem/2 into mod/2. Who knows. I also reorganized
>> the code a little bit, and eliminated an extra
>> method call in all arithmetic functions, by
>>
>> inlining the arithmetic function body in the
>> evaluable predicate definition code. Comparison
>> to old measurements and some measurements of
>>
>> other Prolog systems:
>>
>> /* Dogelog Player 2.1.2 for Java, weeks ago */
>> ?- time(test).
>> % Zeit 378 ms, GC 1 ms, Lips 19900780, Uhr 28.08.2025 17:44
>> true.
>>
>> /* SWI-Prolog 9.0.4 */
>> ?- time(test).
>> % 7,506,639 inferences, 0.363 CPU in 0.362 seconds
>> (100% CPU, 20693560 Lips)
>> true.
>>
>> /* Scryer Prolog 0.9.4-639 */
>> ?- time(test).
>> % CPU time: 0.365s, 7_517_613 inferences
>> true.
>>
>> /* Trealla Prolog 2.82.23-3 */
>> ?- time(test).
>> % Time elapsed 0.868s, 11263917 Inferences, 12.983 MLips
>> true.
>>
>> Bye
>>
>> P.S.: The code uses the hated mathematical mod/2,
>> and not the cheaper rem/2 that CPUs usually have:
>>
>> test :-
>>     len(L, 1000),
>>     primes(L, _).
>>
>> primes([], 1).
>> primes([J|L], J) :-
>>     primes(L, I),
>>     K is I+1,
>>     search(L, K, J).
>>
>> search(L, I, J) :-
>>     mem(X, L),
>>     I mod X =:= 0, !,
>>     K is I+1,
>>     search(L, K, J).
>> search(_, I, I).
>>
>> mem(X, [X|_]).
>> mem(X, [_|Y]) :-
>>     mem(X, Y).
>>
>> len([], 0) :- !.
>> len([_|L], N) :-
>>     N > 0,
>>     M is N-1,
>>     len(L, M).
>>
>> Mild Shock schrieb:
>>> Hi,
>>>
>>> WebPL is already outdated I guess. It doesn't
>>> show the versions of the other Prolog systems
>>> it is using. While I had these results for
>>>
>>> the primes example in the WebPL playground:
>>>
>>> /* Trealla Prolog WASM */
>>> (23568.9ms)
>>>
>>> When I run the example here:
>>>
>>> https://php.energy/trealla.html
>>>
>>> I get better results:
>>>
>>> /* trealla-js 0.27.1 */
>>>
>>> ?- time(test).
>>> % Time elapsed 9.907s, 11263917 Inferences, 1.137 MLips
>>>
>>> Bye
>>
>

Thread

WebPL is already outdated Mild Shock <janburse@fastmail.fm> - 2025-08-17 18:37 +0200
  Heap/Stack versus WAM resp. ZIP (Was: WebPL is already outdated) Mild Shock <janburse@fastmail.fm> - 2025-08-18 14:52 +0200
    Trealla knows Program Sharing (PS) Tricks ? (Was: Heap/Stack versus WAM resp. ZIP) Mild Shock <janburse@fastmail.fm> - 2025-08-18 15:06 +0200
      Smarter Partial Strings would use Program Sharing ['$append'/3] (Was: Trealla knows Program Sharing (PS) Tricks ?) Mild Shock <janburse@fastmail.fm> - 2025-08-18 15:42 +0200
        Static Shunting is even better [Dogelog Player] (Was: Smarter Partial Strings would use Program Sharing ['$append'/3]) Mild Shock <janburse@fastmail.fm> - 2025-08-18 15:49 +0200
  The Artificial Intelligence Flip: Acer Swift Go! (Was: WebPL is already outdated) Mild Shock <janburse@fastmail.fm> - 2025-08-31 23:56 +0200
    2025 will be last year we hear of Python (Re: The Artificial Intelligence Flip: Acer Swift Go!) Mild Shock <janburse@fastmail.fm> - 2025-09-01 00:45 +0200
      Apertus: With love, from Switzerland [02 Sept 2025] (Re: 2025 will be last year we hear of Python) Mild Shock <janburse@fastmail.fm> - 2025-09-05 00:36 +0100
      Don't try this (Was: Apertus: With love, from Switzerland [02 Sept 2025] ) Mild Shock <janburse@fastmail.fm> - 2025-09-05 01:03 +0100
      AI means ambracing Non-Linearity (Was: 2025 will be last year we hear of Python) Mild Shock <janburse@fastmail.fm> - 2025-09-19 10:01 +0200
        AI soaked PCs: Is there a Copilot+ Prolog? (Was: AI means ambracing Non-Linearity) Mild Shock <janburse@fastmail.fm> - 2025-09-19 10:10 +0200
          The morning coffee incident [Prolog Community] (Was: AI soaked PCs: Is there a Copilot+ Prolog?) Mild Shock <janburse@fastmail.fm> - 2025-09-19 14:38 +0200
            Root Cause Prediction for Your Brain (Was: The morning coffee incident [Prolog Community]) Mild Shock <janburse@fastmail.fm> - 2025-09-19 18:22 +0200
              Please delete my account and all my posts on SWI-Prolog discourse (Re: Root Cause Prediction for Your Brain) Mild Shock <janburse@fastmail.fm> - 2025-09-19 18:38 +0200
                I will consult a Lawyer of mine (Was: Please delete my account and all my posts on SWI-Prolog discourse) Mild Shock <janburse@fastmail.fm> - 2025-09-19 18:42 +0200
  Scryer Prolog unify_with_occurs_check/2 doesn't scale (Was: WebPL is already outdated) Mild Shock <janburse@fastmail.fm> - 2025-09-19 16:08 +0200
    How bad is Rust, can JavaScript beat it? (Was: Scryer Prolog unify_with_occurs_check/2 doesn't scale) Mild Shock <janburse@fastmail.fm> - 2025-09-19 16:18 +0200
    unify_with_occurs_check/2 might have been fixed (Was: Scryer Prolog unify_with_occurs_check/2 doesn't scale) Mild Shock <janburse@fastmail.fm> - 2025-09-25 01:50 +0200
      Had to Rollback my Jaffar Unification (Was: unify_with_occurs_check/2 might have been fixed) Mild Shock <janburse@fastmail.fm> - 2025-09-25 01:59 +0200
        Trealla Prolog might apply "frozeness" to cyclic terms (Was: Had to Rollback my Jaffar Unification) Mild Shock <janburse@fastmail.fm> - 2025-09-25 02:06 +0200
          Non-intrusive through "frozen" subcategories (Was: Trealla Prolog might apply "frozeness" to cyclic terms) Mild Shock <janburse@fastmail.fm> - 2025-09-25 02:21 +0200
    Scryer Prolog occurs check cannot do hydra (Was: Scryer Prolog unify_with_occurs_check/2 doesn't scale) Mild Shock <janburse@fastmail.fm> - 2025-09-26 12:19 +0200
  WebPL and Scryer Prolog are bad examples (Was: WebPL is already outdated) Mild Shock <janburse@fastmail.fm> - 2025-10-13 09:49 +0200
    Who will win Shift-Reduce or Tabled DCG? [AI Boom] (Was: WebPL and Scryer Prolog are bad examples) Mild Shock <janburse@fastmail.fm> - 2025-10-13 15:09 +0200
  primes.pl mainly tests the Prolog ALU [mod/2 vs rem/2] (Was: WebPL is already outdated) Mild Shock <janburse@fastmail.fm> - 2025-10-15 02:38 +0200
    25-30% is insane, Neural Network Branch Prediction? (Was: primes.pl mainly tests the Prolog ALU) Mild Shock <janburse@fastmail.fm> - 2025-10-15 04:33 +0200
      NPUs (Neural Processing Units) are the new normal (Was: 25-30% is insane, Neural Network Branch Prediction?) Mild Shock <janburse@fastmail.fm> - 2025-10-15 16:04 +0200
        Ask Phind: AI inflection point right now [End 2025] (Was: NPUs (Neural Processing Units) are the new normal) Mild Shock <janburse@fastmail.fm> - 2025-10-15 16:10 +0200
          Eat Tteokbokki before SkyNet kills you [$100 ChatGPT] (Was: Ask Phind: AI inflection point right now [End 2025]) Mild Shock <janburse@fastmail.fm> - 2025-10-18 15:57 +0200
            Give Julio Di Egidio the bloody money [3 RMB¥ MiniMind] (Re: Eat Tteokbokki before SkyNet kills you [$100 ChatGPT]) Mild Shock <janburse@fastmail.fm> - 2025-10-18 16:19 +0200
              Vertex AI Training is more expensive? (Was: Give Julio Di Egidio the bloody money [3 RMB¥ MiniMind]) Mild Shock <janburse@fastmail.fm> - 2025-10-21 00:32 +0200
        The Love Affair: OpenAI and AMD (Was: NPUs (Neural Processing Units) are the new normal) Mild Shock <janburse@fastmail.fm> - 2025-10-18 18:59 +0200
        The NPU in your Browser [WebNN by W3C] (Was: NPUs (Neural Processing Units) are the new normal) Mild Shock <janburse@fastmail.fm> - 2025-10-26 08:39 +0100
          Fuzzy Alert: Boris the Loris on the Dancefloor (Was: The NPU in your Browser [WebNN by W3C]) Mild Shock <janburse@fastmail.fm> - 2025-10-26 11:33 +0100

csiph-web