Groups > sci.physics.relativity > #671689

The evolution of hardware and GPT-2 training (Was: Why forget Bulgarians, never on my mind)

From	Mild Shock <janburse@fastmail.fm>
Newsgroups	sci.math, sci.physics.relativity
Subject	The evolution of hardware and GPT-2 training (Was: Why forget Bulgarians, never on my mind)
Date	2026-07-27 10:56 +0200
Message-ID	<11476g4$ftjl$1@solani.org> (permalink)
References	(1 earlier) <1140bc7$bdpp$1@solani.org> <1140ciu$bue1$1@solani.org> <1145hhc$fdqk$1@solani.org> <ZDGdnbf3T4PLTPv3nZ2dnZfqnPudnZ2d@giganews.com> <11470h4$gapi$1@solani.org>

Cross-posted to 2 groups.

Show all headers | View raw

Hi,

While Huggingfaces hired GG in 2026,
AK was hired by Anthropic in 2026:

Andrej Karpathy (born 23 October 1986[3])
is a Slovak-Canadian AI researcher, who
co-founded and formerly worked at OpenAI
In 2026 he joined Anthropic as part of
the pretraining team.
https://en.wikipedia.org/wiki/Andrej_Karpathy

But his nanochat archivement has an
interesting time line:

168 hours , Original OpenAI GPT-2 checkpoint, 2019
3 hours , d24 baseline, slightly overtrained, Jan 29 2026
1 1/2 hour, autoresearch round 2, Mar 14 2026
The best ChatGPT that $100 can buy.
https://github.com/karpathy/nanochat

But what hardware was the enabler. What is the
NVIDIA H100 GPU even. Well the thingy is surely not
a Budget Laptop, performance pretty much

dependence on data elememt size, the H100 NVL
version (*), and when using tensor operations,
and not only scalar operations:

8-bit towards 3000 tera flops
16-bit towards 1500 tera flops
32-bit towards 900 tera flops

Cool! I guess this experiment would tap into 60
tera flops, since it only uses scalar operations so far:

11.4 Giga Lips with a Budget Laptop
https://github.com/Jean-Luc-Picard-2021/gigabudget

You could perform it by migration the web application
using WebGPU into a node.js standalone application
using the dawn library for GPU access.

Bye

(*) 
https://resources.nvidia.com/en-us-hopper-architecture/nvidia-tensor-core-gpu-datasheet

Mild Shock schrieb:
> Hi,
> 
> Whats this "forget" trope of glue sniffing
> Rossy Boy with his herpes blisters?
> 
>  > Bulgarians, that's some real Boris and Natasha crap,
>  > forget Hungarians and Bulgarians.
> 
> Why should I forget Bulgarians,
> they are never on my mind. Do you
> see me doing ggml stuff?
> 
> I only hypothesized that it is
> over for Python as the machine
> learning language or AI inferencing
> 
> locally on AI laptops language, and
> made the ggml case, so I already forgot
> about them. Which might give you a glimps,
> 
> why WebGPU was used for this here:
> 
> 11.4 Giga Lips with a Budget Laptop
> https://github.com/Jean-Luc-Picard-2021/gigabudget
> 
> Is an interesting choice. Even
> github has some Languages statistics,
> giving an account what I used:
> 
> HTML 67.5% JavaScript 23.1% CSS 9.4%
> 
> Have Fun!
> 
> Bye
> 
> P.S.: The example below is not p-adics,
> you complete imbecil moron. Its just:
> 
> 7-11 cubic Solution by Pritchard & Gries
> https://www.cs.cornell.edu/gries/TechReports/83-574.pdf
> 
> Ross Finlayson schrieb:
>> On 07/26/2026 10:52 AM, Mild Shock wrote:
>>> Hi,
>>>
>>> You see it all boils down to find your inner peace
>>> by an immaculate inception of some queue datatype.
>>>
>>> KOAN/Fortran-S was an early 1990s research programming
>>> system for distributed-memory multiprocessors . Developed
>>> at ENS Lyon in the early 1990s . Often listed alongside
>>> other historical parallel programming efforts.
>>>
>>> The Message Passing: The research explicitly
>>> compared the SVM approach against message passing
>>> on the same hardware . The finding was that SVM
>>> could achieve good performance without the low-level
>>>
>>> complexity of managing explicit messages, though
>>> the best results often came from a hybrid approach (sic!)
>>> Here is an interesting baseline, from Java,
>>> a class ElevenSingle that only does:
>>>
>>>      public static void run() {
>>>          for (int A = 1; A < 192; A++) {
>>>              int Y = (771-A)/3;
>>>              for (int B = A; B < Y; B++) {
>>>                  int Z = (771-A-B)/2;
>>>                  for (int C = B; C < Z; C++) {
>>>                      int D = 711-A-B-C;
>>>                      if (A*B*C == 711000000/D &&
>>>                            711000000 % D == 0)
>>>      System.out.println("A="+A+", B="+B+", C="+C+", D="+D);
>>>                  }
>>>              }
>>>          }
>>>      }
>>>
>>> And then compare it to ElevenMulti, doing some
>>> Work Balancing Scheduler Tetris Game with 8 cores:
>>>
>>> ElevenSingle
>>> A=120, B=125, C=150, D=316
>>> 6.628 ms
>>>
>>> ElevenMulti
>>> A=120, B=125, C=150, D=316
>>> 1.941 ms
>>>
>>> Not great, not terrible!
>>>
>>> Bye
>>
>> Oh, that's just "tricks of p-adic arithmetic".
>>
>> Like other sock-puppet howler trolls, when confronted
>> with its base incredulity, it will descend to its
>> lower levers of the pathos variety.
>>
>> You might be happier learning about Julia trees and
>> raster ops, instead of shilling yet another Ramanujan
>> series without saying how it's made.
>>
>> Bulgarians, that's some real Boris and Natasha crap,
>> forget Hungarians and Bulgarians.
>>
>>
>

Thread

The Wuhan Virus that destroyed Python [ggml Manifesto] Mild Shock <janburse@fastmail.fm> - 2026-07-22 21:00 +0200
  Deadlock Exorcism: Switch from Push to Pull [A pi-calculus Specification of Prolog] (Re: The Wuhan Virus that destroyed Python [ggml Manifesto]) Mild Shock <janburse@fastmail.fm> - 2026-07-23 00:23 +0200
    Why do you even need a mpmc queue? [Thunder Kittens] (Re: Deadlock Exorcism: Switch from Push to Pull) Mild Shock <janburse@fastmail.fm> - 2026-07-23 08:43 +0200
      Trivial balancing example for (int i=0; i<global_id; i++) (Re: Why do you even need a mpmc queue? [Thunder Kittens]) Mild Shock <janburse@fastmail.fm> - 2026-07-23 08:57 +0200
        Enqueue/dequeue need not be fast and can spinn ["fairness" questions] (Was: Trivial balancing example for (int i=0; i<global_id; i++)) Mild Shock <janburse@fastmail.fm> - 2026-07-23 09:11 +0200
          The Pixel Phone AI Experiment Song (Re: Enqueue/dequeue need not be fast and can spinn ["fairness" questions] ) Mild Shock <janburse@fastmail.fm> - 2026-07-23 09:21 +0200
      Re: Why do you even need a mpmc queue? [Thunder Kittens] (Re: Deadlock Exorcism: Switch from Push to Pull) Ross Finlayson <ross.a.finlayson@gmail.com> - 2026-07-23 08:24 -0700
  Potential Python Recovery: Free Threading [3.13 release] (Was: The Wuhan Virus that destroyed Python [ggml Manifesto]) Mild Shock <janburse@fastmail.fm> - 2026-07-23 10:19 +0200
    Re: Potential Python Recovery: Free Threading [3.13 release] (Was: The Wuhan Virus that destroyed Python [ggml Manifesto]) Ross Valikhanov <kavna@rl.ru> - 2026-07-23 16:01 +0000
  Re: The Wuhan Virus that destroyed Python [ggml Manifesto] Ramon Dubenkov <omd@nnk.ru> - 2026-07-23 13:38 +0000
  The things XILINX braught to the AMD table (Was: The Wuhan Virus that destroyed Python [ggml Manifesto]) Mild Shock <janburse@fastmail.fm> - 2026-07-23 18:47 +0200
    NIVIDIA evacuated its Chinese market [Tau Scaling] (Was: The things XILINX braught to the AMD table) Mild Shock <janburse@fastmail.fm> - 2026-07-23 19:11 +0200
    NVIDIA evacuated its Chinese market [Tau Scaling] (Re: The things XILINX braught to the AMD table) Mild Shock <janburse@fastmail.fm> - 2026-07-23 19:12 +0200
      Re: NVIDIA evacuated its Chinese market [Tau Scaling] (Re: The things XILINX braught to the AMD table) Lane W <cactus_DAC@yahoo.com> - 2026-07-23 11:22 -0600
        Micro penis mother sung arias (Was: NVIDIA evacuated its Chinese market [Tau Scaling]) Mild Shock <janburse@fastmail.fm> - 2026-07-24 14:38 +0200
          Re: Micro penis mother sung arias (Was: NVIDIA evacuated its Chinese market [Tau Scaling]) Lane W <cactus_DAC@yahoo.com> - 2026-07-24 07:15 -0600
            Micro penis brain is in constant hiatus (Was: Micro penis mother sung arias) Mild Shock <janburse@fastmail.fm> - 2026-07-24 15:24 +0200
              Re: Micro penis brain is in constant hiatus (Was: Micro penis mother sung arias) Mild Shock <janburse@fastmail.fm> - 2026-07-24 15:36 +0200
              Ignoramus or Ignorabimus: I don't care (π-WAM) (Re: Micro penis brain is in constant hiatus) Mild Shock <janburse@fastmail.fm> - 2026-07-24 15:38 +0200
                Re: Ignoramus or Ignorabimus: I don't care (π-WAM) (Re: Micro penis brain is in constant hiatus) Lane W <cactus_DAC@yahoo.com> - 2026-07-24 08:31 -0600
                You are a moron, brainless putin payed (Was: Ignoramus or Ignorabimus: I don't care (π-WAM)) Mild Shock <janburse@fastmail.fm> - 2026-07-24 18:01 +0200
                Re: You are a moron, brainless putin payed (Was: Ignoramus or Ignorabimus: I don't care (π-WAM)) Lane W <cactus_DAC@yahoo.com> - 2026-07-24 10:27 -0600
                Yeah keep reading my posts, uninspired fool (Was: You are a moron, brainless putin payed) Mild Shock <janburse@fastmail.fm> - 2026-07-24 19:45 +0200
                Re: Yeah keep reading my posts, uninspired fool (Was: You are a moron, brainless putin payed) Lane W <cactus_DAC@yahoo.com> - 2026-07-24 12:11 -0600
                LoL (Was: Yeah keep reading my posts, uninspired fool ) Mild Shock <janburse@fastmail.fm> - 2026-07-24 20:12 +0200
                Re: LoL (Was: Yeah keep reading my posts, uninspired fool ) Lane W <cactus_DAC@yahoo.com> - 2026-07-24 12:53 -0600
                Out of the blue accusation span 15 days [Empirical USENET study] (Was: Ignoramus or Ignorabimus: I don't care (π-WAM)) Mild Shock <janburse@fastmail.fm> - 2026-07-24 20:26 +0200
  Little Data Center on Your Palm [AI Laptops for 500 USD] (Was: The Wuhan Virus that destroyed Python [ggml Manifesto] Mild Shock <janburse@fastmail.fm> - 2026-07-24 17:58 +0200
    2008: 4 Blades + Tesla S1070 versus 2026: 1 AI Laptop (Re: Little Data Center on Your Palm [AI Laptops for 500 USD]) Mild Shock <janburse@fastmail.fm> - 2026-07-24 18:16 +0200
    Re: Little Data Center on Your Palm [AI Laptops for 500 USD] (Was: The Wuhan Virus that destroyed Python [ggml Manifesto] Bradford Babkoff <ffb@odbb.ru> - 2026-07-24 18:05 +0000
      LoL (Was: Little Data Center on Your Palm [AI Laptops for 500 USD]) Mild Shock <janburse@fastmail.fm> - 2026-07-24 20:11 +0200
  Hurry the blue bus doesnt stop indefinitely (Was: The Wuhan Virus that destroyed Python [ggml Manifesto]) Mild Shock <janburse@fastmail.fm> - 2026-07-24 20:36 +0200
    Not SIMD, a MIMD design for NVIDIA Volta (Re: Hurry the blue bus doesnt stop indefinitely) Mild Shock <janburse@fastmail.fm> - 2026-07-24 20:57 +0200
      Could take 3-4 months find machine / browser (Was Not SIMD, a MIMD design for NVIDIA Volta) Mild Shock <janburse@fastmail.fm> - 2026-07-24 21:15 +0200
      The Koan of pi-WAM queues [FORTRAN-S] (Was: Not SIMD, a MIMD design for NVIDIA Volta) Mild Shock <janburse@fastmail.fm> - 2026-07-26 19:52 +0200
        The turbo capping of AI Laptops (Re: The Koan of pi-WAM queues [FORTRAN-S]) Mild Shock <janburse@fastmail.fm> - 2026-07-26 20:01 +0200
        Re: The Koan of pi-WAM queues [FORTRAN-S] (Was: Not SIMD, a MIMD design for NVIDIA Volta) Ross Finlayson <ross.a.finlayson@gmail.com> - 2026-07-26 20:33 -0700
          Why forget Bulgarians, never on my mind (Re: The Koan of pi-WAM queues [FORTRAN-S] (Was: Not SIMD, a MIMD design for NVIDIA Volta) Mild Shock <janburse@fastmail.fm> - 2026-07-27 09:14 +0200
            miniTriton CUDA is an alternative to torch variants (Was: Why forget Bulgarians, never on my mind) Mild Shock <janburse@fastmail.fm> - 2026-07-27 09:40 +0200
              Andrej Karpathy original gangster of Budget Laptop (Was: miniTriton CUDA is an alternative to torch variants) Mild Shock <janburse@fastmail.fm> - 2026-07-27 09:51 +0200
                Re: Andrej Karpathy original gangster of Budget Laptop (Was: miniTriton CUDA is an alternative to torch variants) Ross Finlayson <ross.a.finlayson@gmail.com> - 2026-07-27 01:41 -0700
            Re: Why forget Bulgarians, never on my mind (Re: The Koan of pi-WAM queues [FORTRAN-S] (Was: Not SIMD, a MIMD design for NVIDIA Volta) Ross Finlayson <ross.a.finlayson@gmail.com> - 2026-07-27 01:38 -0700
            The evolution of hardware and GPT-2 training (Was: Why forget Bulgarians, never on my mind) Mild Shock <janburse@fastmail.fm> - 2026-07-27 10:56 +0200
              How speed up π-WAM with vector operations (Was: The evolution of hardware and GPT-2 training) Mild Shock <janburse@fastmail.fm> - 2026-07-27 11:08 +0200
                AI accelerator extend from GPU to CPU [Zero Copying] (Was: How speed up π-WAM with vector operations) Mild Shock <janburse@fastmail.fm> - 2026-07-27 11:21 +0200
                The invention of vector and matrix registers [NVIDIA Volta] (Was: AI accelerator extend from GPU to CPU [Zero Copying]) Mild Shock <janburse@fastmail.fm> - 2026-07-27 13:20 +0200
                Maybe they should have named it NVIDIA Einstein [Rossy Boy Toe Sucking] (Re: The invention of vector and matrix registers [NVIDIA Volta] (Was: AI accelerator extend from GPU to CPU [Zero Copying]) Mild Shock <janburse@fastmail.fm> - 2026-07-27 17:14 +0200
                π-WAM is not adding decimals, it is removing decimals (Re: The invention of vector and matrix registers [NVIDIA Volta]) Mild Shock <janburse@fastmail.fm> - 2026-07-27 18:36 +0200
                In Budget Laptops the TOPS come with low energy footprint (Was: π-WAM is not adding decimals, it is removing decimals) Mild Shock <janburse@fastmail.fm> - 2026-07-27 18:44 +0200
    Potato Computer owner impressed by Ukraine Tech [Rossy Boys Brother?] (Re: Hurry the blue bus doesnt stop indefinitely) Mild Shock <janburse@fastmail.fm> - 2026-07-27 16:57 +0200
  Got it. Or are you too stupid? [New Usenet Mantra] (Re: The Wuhan Virus that destroyed Python [ggml Manifesto]) Mild Shock <janburse@fastmail.fm> - 2026-07-27 19:00 +0200

csiph-web