The evolution of hardware and GPT-2 training (Re: Why forget Bulgarians, never on my mind)

From	Mild Shock <janburse@fastmail.fm>
Newsgroups	sci.logic, comp.lang.prolog, sci.physics
Subject	The evolution of hardware and GPT-2 training (Re: Why forget Bulgarians, never on my mind)
Date	2026-07-27 10:57 +0200
Message-ID	<11476i7$ftjl$2@solani.org> (permalink)
References	<113r42n$8g50$2@solani.org> <1140bdm$bdpp$2@solani.org> <1140cld$bue1$2@solani.org> <1145hle$fdqk$2@solani.org> <11470l1$gapi$3@solani.org>

Cross-posted to 3 groups.

Show all headers | View raw

Hi,

While Huggingfaces hired GG in 2026,
AK was hired by Anthropic in 2026:

Andrej Karpathy (born 23 October 1986[3])
is a Slovak-Canadian AI researcher, who
co-founded and formerly worked at OpenAI
In 2026 he joined Anthropic as part of
the pretraining team.
https://en.wikipedia.org/wiki/Andrej_Karpathy

But his nanochat archivement has an
interesting time line:

168 hours , Original OpenAI GPT-2 checkpoint, 2019
3 hours , d24 baseline, slightly overtrained, Jan 29 2026
1 1/2 hour, autoresearch round 2, Mar 14 2026
The best ChatGPT that $100 can buy.
https://github.com/karpathy/nanochat

But what hardware was the enabler. What is the
NVIDIA H100 GPU even. Well the thingy is surely not
a Budget Laptop, performance pretty much

dependence on data elememt size, the H100 NVL
version (*), and when using tensor operations,
and not only scalar operations:

8-bit towards 3000 tera flops
16-bit towards 1500 tera flops
32-bit towards 900 tera flops

Cool! I guess this experiment would tap into 60
tera flops, since it only uses scalar operations so far:

11.4 Giga Lips with a Budget Laptop
https://github.com/Jean-Luc-Picard-2021/gigabudget

You could perform it by migration the web application
using WebGPU into a node.js standalone application
using the dawn library for GPU access.

Bye

(*) 
https://resources.nvidia.com/en-us-hopper-architecture/nvidia-tensor-core-gpu-datasheet


Mild Shock schrieb:
> Hi,
> 
> Whats this "forget" trope of glue sniffing
> Rossy Boy with his herpes blisters?
> 
>  > Bulgarians, that's some real Boris and Natasha crap,
>  > forget Hungarians and Bulgarians.
> 
> Why should I forget Bulgarians,
> they are never on my mind. Do you
> see me doing ggml stuff?
> 
> I only hypothesized that it is
> over for Python as the machine
> learning language or AI inferencing
> 
> locally on AI laptops language, and
> made the ggml case, so I already forgot
> about them. Which might give you a glimps,
> 
> why WebGPU was used for this here:
> 
> 11.4 Giga Lips with a Budget Laptop
> https://github.com/Jean-Luc-Picard-2021/gigabudget
> 
> Is an interesting choice. Even
> github has some Languages statistics,
> giving an account what I used:
> 
> HTML 67.5% JavaScript 23.1% CSS 9.4%
> 
> Have Fun!
> 
> Bye
> 
> P.S.: The example below is not p-adics,
> you complete imbecil moron. Its just:
> 
> 7-11 cubic Solution by Pritchard & Gries
> https://www.cs.cornell.edu/gries/TechReports/83-574.pdf
> 
> Ross Finlayson schrieb:
>  > On 07/26/2026 10:52 AM, Mild Shock wrote:
>  >> Hi,
>  >>
>  >> You see it all boils down to find your inner peace
>  >> by an immaculate inception of some queue datatype.
>  >>
>  >> KOAN/Fortran-S was an early 1990s research programming
>  >> system for distributed-memory multiprocessors . Developed
>  >> at ENS Lyon in the early 1990s . Often listed alongside
>  >> other historical parallel programming efforts.
>  >>
>  >> The Message Passing: The research explicitly
>  >> compared the SVM approach against message passing
>  >> on the same hardware . The finding was that SVM
>  >> could achieve good performance without the low-level
>  >>
>  >> complexity of managing explicit messages, though
>  >> the best results often came from a hybrid approach (sic!)
>  >> Here is an interesting baseline, from Java,
>  >> a class ElevenSingle that only does:
>  >>
>  >>      public static void run() {
>  >>          for (int A = 1; A < 192; A++) {
>  >>              int Y = (771-A)/3;
>  >>              for (int B = A; B < Y; B++) {
>  >>                  int Z = (771-A-B)/2;
>  >>                  for (int C = B; C < Z; C++) {
>  >>                      int D = 711-A-B-C;
>  >>                      if (A*B*C == 711000000/D &&
>  >>                            711000000 % D == 0)
>  >>      System.out.println("A="+A+", B="+B+", C="+C+", D="+D);
>  >>                  }
>  >>              }
>  >>          }
>  >>      }
>  >>
>  >> And then compare it to ElevenMulti, doing some
>  >> Work Balancing Scheduler Tetris Game with 8 cores:
>  >>
>  >> ElevenSingle
>  >> A=120, B=125, C=150, D=316
>  >> 6.628 ms
>  >>
>  >> ElevenMulti
>  >> A=120, B=125, C=150, D=316
>  >> 1.941 ms
>  >>
>  >> Not great, not terrible!
>  >>
>  >> Bye
>  >
>  > Oh, that's just "tricks of p-adic arithmetic".
>  >
>  > Like other sock-puppet howler trolls, when confronted
>  > with its base incredulity, it will descend to its
>  > lower levers of the pathos variety.
>  >
>  > You might be happier learning about Julia trees and
>  > raster ops, instead of shilling yet another Ramanujan
>  > series without saying how it's made.
>  >
>  > Bulgarians, that's some real Boris and Natasha crap,
>  > forget Hungarians and Bulgarians.
>  >
>  >
>

Thread

The Wuhan Virus that destroyed Python [ggml Manifesto] Mild Shock <janburse@fastmail.fm> - 2026-07-22 21:01 +0200
  Deadlock Exorcism: Switch from Push to Pull [A pi-calculus Specification of Prolog] (Re: The Wuhan Virus that destroyed Python [ggml Manifesto]) Mild Shock <janburse@fastmail.fm> - 2026-07-23 00:25 +0200
    Why do you even need a mpmc queue? [Thunder Kittens] (Re: Deadlock Exorcism: Switch from Push to Pull) Mild Shock <janburse@fastmail.fm> - 2026-07-23 08:45 +0200
      Trivial balancing example for (int i=0; i<global_id; i++) (Re: Why do you even need a mpmc queue? [Thunder Kittens]) Mild Shock <janburse@fastmail.fm> - 2026-07-23 08:55 +0200
        The Pixel Phone AI Experiment Song (Enqueue/dequeue need not be fast and can spinn ["fairness" questions]) Mild Shock <janburse@fastmail.fm> - 2026-07-23 09:19 +0200
          Enqueue/dequeue need not be fast and can spinn ["fairness" questions] (Re: The Pixel Phone AI Experiment Song (Enqueue/dequeue need not be fast and can spinn ["fairness" questions]) Mild Shock <janburse@fastmail.fm> - 2026-07-23 09:23 +0200
      And, where did I talk about rockets? [Hint its about xAI's Grok] (Re: Why do you even need a mpmc queue? [Thunder Kittens]) Mild Shock <janburse@fastmail.fm> - 2026-07-25 01:25 +0200
        Why forget something, that was never on my mind (Re: And, where did I talk about rockets? [Hint its about xAI's Grok]) Mild Shock <janburse@fastmail.fm> - 2026-07-25 09:49 +0200
      Example Mandel Brot rendering [Faster with MIMD] (Was: Why do you even need a mpmc queue? [Thunder Kittens]) Mild Shock <janburse@fastmail.fm> - 2026-07-25 09:56 +0200
  Potential Python Recovery: Free Threading [3.13 release] (Re: The Wuhan Virus that destroyed Python [ggml Manifesto]) Mild Shock <janburse@fastmail.fm> - 2026-07-23 10:21 +0200
  The things XILINX braught to the AMD table (Re: The Wuhan Virus that destroyed Python [ggml Manifesto]) Mild Shock <janburse@fastmail.fm> - 2026-07-23 18:48 +0200
    NVIDIA evacuated its Chinese market [Tau Scaling] (Re: The things XILINX braught to the AMD table) Mild Shock <janburse@fastmail.fm> - 2026-07-23 19:13 +0200
      Micro penis mother sung arias (Re: NVIDIA evacuated its Chinese market [Tau Scaling]) Mild Shock <janburse@fastmail.fm> - 2026-07-24 14:40 +0200
        Micro penis brain is in constant hiatus (Re: Micro penis mother sung arias) Mild Shock <janburse@fastmail.fm> - 2026-07-24 15:27 +0200
          Ignoramus or Ignorabimus: I don't care [(Re: Micro penis brain is in constant hiatus (Re: Micro penis mother sung arias) Mild Shock <janburse@fastmail.fm> - 2026-07-24 15:35 +0200
            You are a moron, brainless putin payed (Re: Ignoramus or Ignorabimus: I don't care) Mild Shock <janburse@fastmail.fm> - 2026-07-24 18:01 +0200
              Yeah keep reading my posts, uninspired fool (Re: You are a moron, brainless putin payed) Mild Shock <janburse@fastmail.fm> - 2026-07-24 19:47 +0200
            Out of the blue accusation span 15 days [Empirical USENET study] (Re: Ignoramus or Ignorabimus: I don't care) Mild Shock <janburse@fastmail.fm> - 2026-07-24 20:27 +0200
  Little Data Center on Your Palm [AI Laptops for 500 USD] (Re: The Wuhan Virus that destroyed Python [ggml Manifesto]) Mild Shock <janburse@fastmail.fm> - 2026-07-24 17:59 +0200
    2008: 4 Blades + Tesla S1070 versus 2026: 1 AI Laptop (Re: Little Data Center on Your Palm [AI Laptops for 500 USD]) Mild Shock <janburse@fastmail.fm> - 2026-07-24 18:15 +0200
  Hurry the blue bus doesnt stop indefinitely (Re: The Wuhan Virus that destroyed Python [ggml Manifesto]) Mild Shock <janburse@fastmail.fm> - 2026-07-24 20:37 +0200
    Not SIMD, a MIMD design for NVIDIA Volta (Re: Hurry the blue bus doesnt stop indefinitely) Mild Shock <janburse@fastmail.fm> - 2026-07-24 20:58 +0200
      Could take 3-4 months find machine / browser (Re: Not SIMD, a MIMD design for NVIDIA Volta) Mild Shock <janburse@fastmail.fm> - 2026-07-24 21:16 +0200
      The Koan of pi-WAM queues [FORTRAN-S] (Re: Not SIMD, a MIMD design for NVIDIA Volta) Mild Shock <janburse@fastmail.fm> - 2026-07-26 19:54 +0200
        The turbo capping of AI Laptops (Was: The Koan of pi-WAM queues [FORTRAN-S]) Mild Shock <janburse@fastmail.fm> - 2026-07-26 20:00 +0200
        Re: The Koan of pi-WAM queues [FORTRAN-S] (Re: Not SIMD, a MIMD design for NVIDIA Volta) Mild Shock <janburse@fastmail.fm> - 2026-07-27 09:16 +0200
        Why forget Bulgarians, never on my mind (Re: The Koan of pi-WAM queues [FORTRAN-S]) Mild Shock <janburse@fastmail.fm> - 2026-07-27 09:16 +0200
          miniTriton CUDA is an alternative to torch variants (Re: Why forget Bulgarians, never on my mind) Mild Shock <janburse@fastmail.fm> - 2026-07-27 09:52 +0200
            Andrej Karpathy original gangster of Budget Laptop (Re: miniTriton CUDA is an alternative to torch variants) Mild Shock <janburse@fastmail.fm> - 2026-07-27 09:54 +0200
          The evolution of hardware and GPT-2 training (Re: Why forget Bulgarians, never on my mind) Mild Shock <janburse@fastmail.fm> - 2026-07-27 10:57 +0200
            How speed up π-WAM with vector operations (Re: The evolution of hardware and GPT-2 training) Mild Shock <janburse@fastmail.fm> - 2026-07-27 11:10 +0200
              AI accelerator extend from GPU to CPU [Zero Copying] (Re: How speed up π-WAM with vector operations) Mild Shock <janburse@fastmail.fm> - 2026-07-27 13:21 +0200
                The invention of vector and matrix registers [NVIDIA Volta] (Re: AI accelerator extend from GPU to CPU [Zero Copying]) Mild Shock <janburse@fastmail.fm> - 2026-07-27 13:22 +0200
                Re: The invention of vector and matrix registers [NVIDIA Volta] (Re: AI accelerator extend from GPU to CPU [Zero Copying]) Ross Finlayson <ross.a.finlayson@gmail.com> - 2026-07-27 07:34 -0700
                Maybe they should have named it NVIDIA Einstein [Rossy Boy Toe Sucking] (Was: The invention of vector and matrix registers [NVIDIA Volta]) Mild Shock <janburse@fastmail.fm> - 2026-07-27 17:12 +0200
                Re: The invention of vector and matrix registers [NVIDIA Volta] (Re: AI accelerator extend from GPU to CPU [Zero Copying]) R Kym Horsell <kym@sdf.com> - 2026-07-27 15:43 +0000
                Re: The invention of vector and matrix registers [NVIDIA Volta] (Re: AI accelerator extend from GPU to CPU [Zero Copying]) R Kym Horsell <kymhorsell@gmail.com> - 2026-07-27 15:46 +0000
                π-WAM is not adding decimals, it is removing decimals (Was: The invention of vector and matrix registers [NVIDIA Volta]) Mild Shock <janburse@fastmail.fm> - 2026-07-27 18:34 +0200
                In Budget Laptops the TOPS come with low energy footprint (Re: π-WAM is not adding decimals, it is removing decimals) Mild Shock <janburse@fastmail.fm> - 2026-07-27 18:45 +0200
    Potato Computer owner impressed by Ukraine Tech [Rossy Boys Brother?] (Was: Hurry the blue bus doesnt stop indefinitely) Mild Shock <janburse@fastmail.fm> - 2026-07-27 16:56 +0200
      Rossy Boy is neither Einstein nor Zweistein (Was: Potato Computer owner impressed by Ukraine Tech) Mild Shock <janburse@fastmail.fm> - 2026-07-27 18:25 +0200
  Got it. Or are you too stupid? [New Usenet Mantra] (Was: The Wuhan Virus that destroyed Python [ggml Manifesto]) Mild Shock <janburse@fastmail.fm> - 2026-07-27 18:59 +0200

csiph-web