Groups > alt.comp.os.windows-11 > #34057

Re: MS Edge was never useful but at least it had worked for Copilot but what AI/LLM now?

From	Paul <nospam@needed.invalid>
Newsgroups	alt.comp.os.windows-11, alt.comp.os.windows-10, alt.comp.microsoft.windows
Subject	Re: MS Edge was never useful but at least it had worked for Copilot but what AI/LLM now?
Date	2026-08-27 05:41 -0400
Organization	A noiseless patient Spider
Message-ID	<116p0nk$1bjo8$1@dont-email.me> (permalink)
References	<116n9nb$2hss$1@nnrp.usenet.blueworldhosting.com> <116ni78$ui00$1@dont-email.me> <116o3de$13gjm$1@dont-email.me> <6245mmxe7e.ln2@Telcontar.valinor>

Cross-posted to 3 groups.

Show all headers | View raw

On Thu, 8/27/2026 3:03 AM, Carlos E.R. wrote:
> On 2026-08-27 03:20, Paul wrote:
>> You can run them locally, but you have to be patient. For one request
>> to write a program, and with high reasoning enabled, it took 50 minutes
>> to half-write the program. The high reasoning took about eight minutes
>> of that, and the vast majority of the time was spent on token output.
>>
>> They tell me it would use a video card for token output,*if* the question
>> fits entirely within the video card. But with my puny hardware resources, it
>> always uses the CPU for token output, which can be slow.
> 
> There is a chip that can accelerate the ai more than gpus. I read about it 
> in the ieee spectrum magazine, but I don't remember the name of the thing. 
> Laptops will come with that hardware so they can handle ai locally with reasonable battery usage.
> 
> probably asking an ai about this it will tell better details than me :-D

Acceleration devices, have support for just one numeric format,
or for multiple numeric formats.

Model files, some of them come pre-quantized to a particular numeric
format, for a reason. If your hardware happens not to have that format,
then something must be done to fix that. This is why there are two
overlapping formats, one an open thing, the other an NVidia version.
If a model comes in the NVidia version, there's a message there for you.

Acceleration devices tend to quote the TOPS figure for their "fastest" format.
Whether they even have a second format, is another question.
Maybe an Intel NPU, they quote the INT8 performance. Now, in LMStudio,
look in the model list, and how many of the models are INT8 ?
Quantizing a model, can break it, so the results are not necessarily
good.

That's why, the single TRIT model that was made, it was "trained in TRITs"
and "Inference is in TRITs". That is to ensure that quantization does
not ruin it. A CPU can handle [-1,0,+1 ] as a representation, but
typical acceleration devices do not have a hardware block for that.

This makes the topic of models and accelerators, a bit of a mine field.
There are things we could be doing... which are not happening. A TRIT
model requires a lot less RAM. And, there are none.

You can see a discussion of what happens when dishing out resources on big machines.

https://aimultiple.com/llm-quantization

   "Int4 degrades code generation more than knowledge"

This is why I use code generation, as my "acceptance test" for local models.
Can it write code ? Is it timid, like one model which only wrote comment
text for the source file ? I don't particularly care, if it can
list all the flavors of toothpaste. That should be pretty easy for it.
Even a small small model, could tell you about peppermint toothpaste.

   Paul

Thread

csiph-web