Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #167963

Re: Idle: C library features wish-list.

Show all headers | View raw

On 10/4/2022 11:56 AM, Kaz Kylheku wrote:
> On 2022-10-04, BGB <cr88192@gmail.com> wrote:
>> In some cases, one "actually needs" a copy operation that will turn the
>> output into a repeating pattern of bytes whenever one forward-copies a
>> chunk of memory over the top of itself.
> 
> You mention this in the context of LZ77 deflate, but I don't see
> any such thing in zlib sources. There is a zmemcpy which
> in some cases is just a macro for memcpy.
> 
> Is Adler missing some clue or something?
> 

No zlib here, rather a custom implementation.

One needs Deflate mostly for things like ZIP and PNG, but one doesn't 
necessarily need to use zlib to do so (nor does one need libpng for PNGs).


Well, and in this case, also not running on a conventional OS.
No Linux or Windows in this case, rather "TestKern", which is sort of like:
Uses a lot of file-formats and other stuff borrowed from Windows
   Uses PE/COFF, RIFF based formats, ...
Uses an API design more modeled after POSIX
General "architecture" generally has more in common with MS-DOS at this 
stage (memory protection is borderline non existent; does more or less 
have virtual memory working, but all of the program instances still 
currently run in a single shared virtual address space).

Also, like DOS (and unlike Linux or Windows), doesn't have preemptive 
multitasking yet. Had written some code for this, but it isn't yet used, 
and work is still needed for "processes" to be a thing. Cooperative 
multithreading isn't exactly the same.


Porting software to it is a little bit of a hassle (things like 
"./configure" aren't going to work when one doesn't even have Bash).

Porting something like Linux or BSD or similar would probably be better, 
but porting these to my ISA look like probably an uphill battle (and 
porting the GNU userland isn't really going to work out well without GCC 
support for this architecture, ...).



As for the way _memlzcpy fits in with LZ77, it is typically that matches 
are expressed as a backwards distance and a length. Where, if the 
distance is less than the length, one gets a repeating pattern.

So:
   ct=_memlzcpy(ct, ct-dist, len);
Can express the typical LZ style match-copy operation.

Say, for decoding an LZ4 style format, one could write:
   ct=dest; cs=src; cse=cs+csize;
   while(cs<cse)
   {
     i=*cs++;
     rl=i>>4; ml=(i&15)+4;
     if(rl==15)
     {
       i=*cs++;
       while(i==255)
         { rl+=i; i=*cs++; }
       rl+=i;
     }
     ct=_memlzcpyf(ct, cs, rl);
     cs+=rl;
     if(cs>=cse)
       break;
     md=_mget_uint16le(cs);
     cs+=2;
     if(ml==19)
     {
       i=*cs++;
       while(i==255)
         { ml+=i; i=*cs++; }
       ml+=i;
     }
     ct=_memlzcpyf(ct, ct-md, ml);
     ct+=ml;
   }


Decided to leave out going a bunch into stuff related to LZ compressors.

They are used for various purposes, among other things, using 
compression as a way to read data from the SDcard faster (at 12.5 MHz, 
the SDcard only does IO at around 1.5 MB/s in SPI mode).



The main target I am dealing with for this is mostly on my BJX2 ISA, 
which is basically a 64-bit 3-wide VLIW, generally runs on FPGA, 
generally at 50MHz.


In some areas, it is kinda meh:
   Runs Doom at ~ 15-20 fps;
   Runs Hexen at ~ 8-10 fps;
   Runs ROTT at ~ 10-12 fps;
   SW Quake at ~ 2-4 fps;
   GLQuake at ~ 5-8 fps;
   At present, only gets ~ 74k in Dhrystone (~ 0.84 DMIPS/MHz, *1);
   ...

But, it does a "surprisingly passable" job at things like software 
OpenGL (*2) rasterization (and a lot of "my own stuff" does rendering 
using the software rasterized OpenGL; also a custom implementation 
optimized for this ISA, with a fair chunk written in ASM).


*1: At Dhrystone, it seems that RISC-V gets better DMIPS/MHz scores.
I suspect some of this is due to GCC being a lot more "clever" than my 
compiler (BGBCC).
Arguably, RISC-V is still a much better option in terms of "being 
practical for general use".

Though, can generally pass timing at higher clock speeds than the SweRV 
core, even if the DMIPS/MHz score is worse. Both need roughly similar 
class FPGAs (XC7S50 or XC7A100 or similar). Had noted that internal 
architecture was very different. Though, have noted that Dhrystone on 
simpler 1-wide scalar RISC-V cores seems to be closer to around 0.6 
DMIPS/MHz (rather than ~ 1.4).

The pipeline is very different, they seem to have the instruction 
pipeline and memory load-store as two independent components (connected 
via a FIFO interface or similar).

In my case, the L1 caches and pipeline operate in lockstep. So, if an L1 
miss happens, the pipeline stalls until the situation is resolved (and 
all instructions in a VLIW bundle advance strictly in lockstep).
Extracting usable ILP from a program is left pretty much entirely to the 
compiler (and/or the person writing ASM code for it).


*2: It implements the OpenGL API, more or less, but is basically a 
software renderer on the backend, and uses affine filtering (with 
dynamic tessellation), so tends to generate output that kinda more 
resembles something like the original PlayStation than a modern GPU.

Software Quake is around 2-4 fps, GLQuake is around 5-8.
Had worked some on trying to porting Quake 3 Arena to it, but this 
fizzled out as Q3A is both memory hungry and very unlikely to be usable.

Partly this is because an OpenGL style rasterizer can make slightly more 
effective use of the CPU's VLIW capabilities.


Ironically, I do have a small custom "Minecraft like" 3D engine running 
on it (though, staying above 5 fps requires limiting it to a 12 meter 
draw distance, which kinda sucks).

Also ironically, still faster than trying to run "actual" Minecraft with 
a similar draw distance on a laptop from 2003 (which has 36x higher 
clock speed). Though, this laptop is plenty fast enough to run Quake and 
similar.


Can do video playback semi passable. But, it is a balancing act between 
computational cost of the video decoding and keeping the bitrate low 
enough that it doesn't get stuck on IO bandwidth (some of the "classic" 
codecs like CRAM or RPZA use need too much IO bandwidth to get the video 
data off the SDcard).

Was generally having best results in this case with hybrid CC/VQ codecs 
with an LZ post-compression stage.

Had observed that it is fast enough at JPEG decoding, that it is at 
least possible that an MPEG style decoder could be used (not tested 
yet). Main "slow parts" of the JPEG decoding in this case being the 
Huffman/VLC decoding, and writing stuff to the output framebuffer 
(things like IDCT and YCbCr->RGB transform mapping "reasonably well" to 
VLIW).

Though, an MPEG-like codec would likely be limited to 160x100 or 
similar, as 320x200 is likely to have more computational cost than the 
CPU could deal with at 50 MHz.

With the VQ+LZ approach was generally able to manage 320x200 video.


>> Also it can be used as a way to implement a "multi byte memset", say for
>> example, if one wants a fast way to flood-fill a chunk of memory with
>> 0xDEADBEEF or similar, ...
> 
> A multi-byte memset that is reading from the area where it is writing
> seems inefficient, particularly if the memory is cache-cold,
> since it will actually be sucking the memory into the processor's
> caches only to turn around a blast it out again.
> 
> You really want an actual multi-byte memset for that use case.
> 

That was also a possible consideration. Depends mostly on use case.

Main hassle with a multi-byte memset is that one (potentially) needs 
multiple versions for each fill size.

But, yeah, something like _memset16, _memset32, or _memset64 could also 
address this use-case.

Though, a multibyte memset would make things like alignment a little 
easier (in my case, calls like "malloc()" always return memory with a 
16-byte alignment).

..

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Idle: C library features wish-list. BGB <cr88192@gmail.com> - 2022-09-30 16:34 -0500
  Re: Idle: C library features wish-list. Tim Rentsch <tr.17687@z991.linuxsc.com> - 2022-10-02 20:38 -0700
    Re: Idle: C library features wish-list. BGB <cr88192@gmail.com> - 2022-10-03 01:14 -0500
      Re: Idle: C library features wish-list. Kaz Kylheku <864-117-4973@kylheku.com> - 2022-10-03 20:40 +0000
        Re: Idle: C library features wish-list. scott@slp53.sl.home (Scott Lurndal) - 2022-10-03 21:10 +0000
          Re: Idle: C library features wish-list. BGB <cr88192@gmail.com> - 2022-10-03 19:07 -0500
            Re: Idle: C library features wish-list. scott@slp53.sl.home (Scott Lurndal) - 2022-10-04 13:43 +0000
              Re: Idle: C library features wish-list. BGB <cr88192@gmail.com> - 2022-10-04 15:20 -0500
                Re: Idle: C library features wish-list. BGB <cr88192@gmail.com> - 2022-10-05 03:58 -0500
            Re: Idle: C library features wish-list. Kaz Kylheku <864-117-4973@kylheku.com> - 2022-10-04 16:56 +0000
              Re: Idle: C library features wish-list. BGB <cr88192@gmail.com> - 2022-10-04 18:30 -0500
        Re: Idle: C library features wish-list. BGB <cr88192@gmail.com> - 2022-10-03 18:19 -0500
      Re: Idle: C library features wish-list. Tim Rentsch <tr.17687@z991.linuxsc.com> - 2022-11-19 06:54 -0800
    Re: Idle: C library features wish-list. gazelle@shell.xmission.com (Kenny McCormack) - 2022-10-03 09:29 +0000

csiph-web