Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.c > #395686 > unrolled thread
| Started by | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| First post | 2025-12-06 01:05 +0000 |
| Last post | 2025-12-17 00:52 -0600 |
| Articles | 19 on this page of 119 — 23 participants |
Back to article view | Back to comp.lang.c
is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-06 01:05 +0000
Re: is_binary_file() Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2025-12-06 01:41 +0000
Re: is_binary_file() Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2025-12-06 02:00 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 17:40 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 11:35 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-10 15:07 +0000
Re: is_binary_file() Michael S <already5chosen@yahoo.com> - 2025-12-10 19:00 +0200
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-10 17:18 +0000
Re: is_binary_file() Richard Heathfield <rjh@cpax.org.uk> - 2025-12-10 19:42 +0000
Re: is_binary_file() bart <bc@freeuk.com> - 2025-12-10 22:37 +0000
Re: is_binary_file() Paul <nospam@needed.invalid> - 2025-12-10 22:35 -0500
Re: is_binary_file() bart <bc@freeuk.com> - 2025-12-11 11:46 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-11 12:53 +0100
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 18:42 +0000
Re: is_binary_file() Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2025-12-10 15:58 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 18:44 +0000
Re: is_binary_file() James Kuyper <jameskuyper@alumni.caltech.edu> - 2025-12-10 12:46 -0500
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 18:45 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 18:41 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 20:57 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-10 22:07 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-11 01:09 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-11 12:33 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-12 19:25 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-12 22:54 +0000
Re: is_binary_file() "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-12-12 15:33 -0800
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-13 00:20 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-13 02:32 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-16 00:26 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-16 17:24 +0100
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-17 03:19 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-17 07:57 +0100
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-17 19:35 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-18 08:44 +0100
Re: is_binary_file() bart <bc@freeuk.com> - 2025-12-18 12:49 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-18 14:06 +0100
Re: is_binary_file() gazelle@shell.xmission.com (Kenny McCormack) - 2025-12-18 13:17 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-18 16:03 +0100
Re: is_binary_file() Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2025-12-05 17:42 -0800
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-06 17:37 +0000
Re: is_binary_file() Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2025-12-06 16:05 -0800
Re: is_binary_file() Louis Krupp <lkrupp@invalid.pssw.com.invalid> - 2025-12-07 03:43 -0700
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-07 16:47 +0000
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-27 03:18 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 17:46 +0000
Re: is_binary_file() Kaz Kylheku <046-301-5902@kylheku.com> - 2025-12-06 02:42 +0000
Re: is_binary_file() bart <bc@freeuk.com> - 2025-12-06 12:42 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-06 17:40 +0000
Re: is_binary_file() Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2025-12-06 18:04 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-06 19:06 +0000
Re: is_binary_file() Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2025-12-06 21:16 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 17:48 +0000
Re: is_binary_file() Kaz Kylheku <046-301-5902@kylheku.com> - 2025-12-08 19:26 +0000
Re: is_binary_file() Richard Heathfield <rjh@cpax.org.uk> - 2025-12-08 19:42 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-09 21:49 +0000
Re: is_binary_file() Paul <nospam@needed.invalid> - 2025-12-06 03:14 -0500
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 17:56 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-08 20:16 +0000
Re: is_binary_file() David Brown <david.brown@hesbynett.no> - 2025-12-09 09:03 +0100
Re: is_binary_file() Richard Heathfield <rjh@cpax.org.uk> - 2025-12-09 09:43 +0000
Re: is_binary_file() Richard Harnden <richard.nospam@gmail.invalid> - 2025-12-09 10:17 +0000
Re: is_binary_file() Kaz Kylheku <046-301-5902@kylheku.com> - 2025-12-09 20:15 +0000
Re: is_binary_file() tTh <tth@none.invalid> - 2025-12-09 12:22 +0100
Re: is_binary_file() Paul <nospam@needed.invalid> - 2025-12-09 20:26 -0500
Re: is_binary_file() Paul <nospam@needed.invalid> - 2025-12-09 06:38 -0500
Re: is_binary_file() Michael S <already5chosen@yahoo.com> - 2025-12-09 17:31 +0200
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-28 02:49 +0000
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-28 00:12 +0000
Re: is_binary_file() richard@cogsci.ed.ac.uk (Richard Tobin) - 2025-12-28 00:43 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-06 17:33 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-07 19:04 +0100
Re: is_binary_file() James Kuyper <jameskuyper@alumni.caltech.edu> - 2025-12-06 20:37 -0500
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 18:02 +0000
Re: is_binary_file() James Kuyper <jameskuyper@alumni.caltech.edu> - 2025-12-09 16:29 -0500
Re: is_binary_file() Michael S <already5chosen@yahoo.com> - 2025-12-10 11:21 +0200
Re: is_binary_file() James Kuyper <jameskuyper@alumni.caltech.edu> - 2025-12-10 12:48 -0500
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 11:38 +0000
Re: is_binary_file() antispam@fricas.org (Waldek Hebisch) - 2025-12-07 03:43 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 18:04 +0000
Re: is_binary_file() bart <bc@freeuk.com> - 2025-12-08 18:44 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-09 19:53 +0000
Re: is_binary_file() Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2025-12-09 15:42 -0800
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 11:41 +0000
Re: is_binary_file() Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2025-12-10 15:20 -0800
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 23:59 +0000
Re: is_binary_file() James Kuyper <jameskuyper@alumni.caltech.edu> - 2025-12-09 16:23 -0500
Re: is_binary_file() Richard Harnden <richard.nospam@gmail.invalid> - 2025-12-07 19:01 +0000
Re: is_binary_file() Richard Heathfield <rjh@cpax.org.uk> - 2025-12-07 21:51 +0000
Re: is_binary_file() Richard Harnden <richard.nospam@gmail.invalid> - 2025-12-07 22:49 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-08 13:51 +0100
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-08 16:04 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-08 19:27 +0100
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-27 05:51 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-29 16:06 +0100
Re: is_binary_file() mjos_examine <m6502x64@gmail.com> - 2025-12-29 11:49 -0500
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-29 20:49 +0100
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-30 01:52 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-08 16:02 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 18:07 +0000
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-27 03:13 +0000
Re: is_binary_file() Paul <nospam@needed.invalid> - 2025-12-27 01:28 -0500
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-27 21:27 +0000
Re: is_binary_file() antispam@fricas.org (Waldek Hebisch) - 2025-12-28 05:46 +0000
Re: is_binary_file() "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-12-07 14:42 -0800
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 18:09 +0000
Re: is_binary_file() "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-12-09 12:45 -0800
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-08 20:36 +0100
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-08 20:50 +0100
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-09 15:09 +0100
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-10 09:18 +0100
Re: is_binary_file() Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2025-12-08 14:43 -0800
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-09 21:38 +0000
Re: is_binary_file() Kaz Kylheku <046-301-5902@kylheku.com> - 2025-12-11 17:33 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-11 19:10 +0100
Re: is_binary_file() "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-12-11 14:56 -0800
Re: is_binary_file() James Kuyper <jameskuyper@alumni.caltech.edu> - 2025-12-11 18:15 -0500
Re: is_binary_file() Janis Papanagnou <janis_papanagnou+ng@hotmail.com> - 2025-12-12 02:19 +0100
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-14 08:27 +0000
Re: is_binary_file() Lynn McGuire <lynnmcguire5@gmail.com> - 2025-12-17 00:52 -0600
Page 6 of 6 — ← Prev page 1 2 3 4 5 [6]
| From | Paul <nospam@needed.invalid> |
|---|---|
| Date | 2025-12-27 01:28 -0500 |
| Message-ID | <10inua3$38636$1@dont-email.me> |
| In reply to | #395988 |
On Fri, 12/26/2025 10:13 PM, Lawrence D’Oliveiro wrote:
> On Sun, 7 Dec 2025 19:01:02 +0000, Richard Harnden wrote:
>
>> A text file is supposed to end with a '\n'
>
> PDF files end with that. The object index comes at the end, and each
> index entry is fixed in length and ends with \015\012.
>
> But the spec makes it very clear that PDF files are not supposed to be
> treated as text files.
>
The best you can do, is for the PDF to be entirely text except for
some bytes near the top (second line). It's not exactly clear what they do,
but I've seen at least one document that misses the binary line. That
binary-thing could be a hash over the document.
At least in this PDF, the document is 99% text. And Mutool can be
used to convert a "mostly binary" PDF, into a "mostly text" PDF.
If a PDF is encrypted, it is unlikely to have a textual representation
when naively opening it.
PDFs can be "anywhere from 99% binary to 99% text". It all depends.
Generally, the ones that are mostly text are the simplest of documents.
Rich media documents will have a lot more binary that cannot be
simplified by simple transformations. You could start in the first place,
by using different source materials that had closer-to-textual representation
to fix that.
***********************************************************************************************************
%PDF-1.4
<=== these can "look like binary" "25 B8 9A 92 9D 0A"
1 0 obj<</Type/Catalog/Pages 3 0 R>>
endobj
2 0 obj<</Producer(GemBox GemBox.Pdf 1.7 (17.0.35.1042; .NET Framework))/CreationDate(D:20211028151721+02'00')>>
endobj
3 0 obj<</Type/Pages/Kids[4 0 R]/Count 1/MediaBox[0 0 595.32 841.92]>>
endobj
4 0 obj<</Type/Page/Parent 3 0 R/Resources<</Font<</F0 6 0 R>>>>/Contents 5 0 R>>
endobj
5 0 obj<</Length 59>>stream
BT
/F0 12 Tf
1 0 0 1 100 702.7366667 Tm
(Hello World!)Tj
ET
endstream
endobj
6 0 obj<</Type/Font/Subtype/Type1/BaseFont/Helvetica/FirstChar 32/LastChar 114/Widths 7 0 R/FontDescriptor 8 0 R>>
endobj
7 0 obj[278 278 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 722 0 0 0 0 0 0 0 0 0 0 0 0 0 0 944 0 0 0 0 0 0 0 0 0 0 0 0 556 556 0 0 0 0 0 0 222 0 0 556 0 0 333]
endobj
8 0 obj<</Type/FontDescriptor/Flags 32/FontName/Helvetica/FontFamily(Helvetica)/FontWeight 500/ItalicAngle 0/FontBBox[-166 -225 1000 931]/CapHeight 718/XHeight 523/Ascent 718/Descent -207/StemH 76/StemV 88>>
endobj
xref
0 9
0000000000 65535 f
0000000015 00000 n
0000000059 00000 n
0000000179 00000 n
0000000257 00000 n
0000000346 00000 n
0000000451 00000 n
0000000573 00000 n
0000000773 00000 n
trailer
<</Root 1 0 R/ID[<9392A59F3BE7B840805D62746E8A4F29><9392A59F3BE7B840805D62746E8A4F29>]/Info 2 0 R/Size 9>>
startxref
988
%%EOF
***********************************************************************************************************
If "there has to be binary in it", it's on the second line.
The other lines can be text... if the tools and print drivers
wanted to do it that way.
Paul
[toc] | [prev] | [next] | [standalone]
| From | Lawrence D’Oliveiro <ldo@nz.invalid> |
|---|---|
| Date | 2025-12-27 21:27 +0000 |
| Message-ID | <10ipivp$3q5oj$2@dont-email.me> |
| In reply to | #395992 |
On Sat, 27 Dec 2025 01:28:18 -0500, Paul wrote: > The best you can do, is for the PDF to be entirely text except for > some bytes near the top (second line). It's not exactly clear what > they do ... The spec recommended the insertion of junk like that simply to dissuade file sniffers from concluding that the file is a text document.
[toc] | [prev] | [next] | [standalone]
| From | antispam@fricas.org (Waldek Hebisch) |
|---|---|
| Date | 2025-12-28 05:46 +0000 |
| Message-ID | <10iqg7o$2g8io$2@paganini.bofh.team> |
| In reply to | #395992 |
Paul <nospam@needed.invalid> wrote:
> On Fri, 12/26/2025 10:13 PM, Lawrence D’Oliveiro wrote:
>> On Sun, 7 Dec 2025 19:01:02 +0000, Richard Harnden wrote:
>>
>>> A text file is supposed to end with a '\n'
>>
>> PDF files end with that. The object index comes at the end, and each
>> index entry is fixed in length and ends with \015\012.
>>
>> But the spec makes it very clear that PDF files are not supposed to be
>> treated as text files.
>>
>
> The best you can do, is for the PDF to be entirely text except for
> some bytes near the top (second line). It's not exactly clear what they do,
> but I've seen at least one document that misses the binary line. That
> binary-thing could be a hash over the document.
I did a little developement on PDF-s. For debugging is is convenient
to have 100% printable form, such PDF-s are perfectly valid. Adobe
encourages putting in a bunch of nonprintable characters to
discourage silly tools from "converting text encoding", which
would mangle PDF-s.
--
Waldek Hebisch
[toc] | [prev] | [next] | [standalone]
| From | "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> |
|---|---|
| Date | 2025-12-07 14:42 -0800 |
| Message-ID | <10h4vt0$3p7o9$2@dont-email.me> |
| In reply to | #395686 |
On 12/5/2025 5:05 PM, Michael Sanders wrote:
> int is_binary_file(const char *path) {
[...]
You can return a float from is_binary_file() to show a probability? Not
exactly sure how you can 100% guarantee it...
[toc] | [prev] | [next] | [standalone]
| From | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| Date | 2025-12-08 18:09 +0000 |
| Message-ID | <10h7484$9q1e$8@dont-email.me> |
| In reply to | #395710 |
On Sun, 7 Dec 2025 14:42:39 -0800, Chris M. Thomasson wrote: > You can return a float from is_binary_file() to show a probability? Not > exactly sure how you can 100% guarantee it... Ha! You know, that's a crazy idea but a darn cool idea at the same time! -- :wq Mike Sanders
[toc] | [prev] | [next] | [standalone]
| From | "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> |
|---|---|
| Date | 2025-12-09 12:45 -0800 |
| Message-ID | <10ha1p5$12lf9$2@dont-email.me> |
| In reply to | #395722 |
On 12/8/2025 10:09 AM, Michael Sanders wrote: > On Sun, 7 Dec 2025 14:42:39 -0800, Chris M. Thomasson wrote: > >> You can return a float from is_binary_file() to show a probability? Not >> exactly sure how you can 100% guarantee it... > > Ha! > > You know, that's a crazy idea but a darn cool idea at the same time! > ;^) It would be funny with a return of .5, lol An error can be a negative result.
[toc] | [prev] | [next] | [standalone]
| From | Bonita Montero <Bonita.Montero@gmail.com> |
|---|---|
| Date | 2025-12-08 20:36 +0100 |
| Message-ID | <10h79a3$bfg2$1@raubtier-asyl.eternal-september.org> |
| In reply to | #395686 |
Am 06.12.2025 um 02:05 schrieb Michael Sanders:
> Am I close? Missing anything you'd consider to be (or not) needed?
>
> <stdio.h>
>
> /*
> * Checks if a file is likely a binary by examining its content
> * for NULL bytes (0x00) or unusual control characters.
> * Returns 0 if text, 1 if binary or file open failure.
> */
>
> int is_binary_file(const char *path) {
> FILE *f = fopen(path, "rb");
> if (!f) return 1; // cannot open file, treat as error/fail check
>
> unsigned char buf[65536];
> size_t n, i;
>
> while ((n = fread(buf, 1, sizeof(buf), f)) > 0) {
> for (i = 0; i < n; i++) {
> unsigned char c = buf[i];
>
> // 1. check for the NULL byte (strong indicator of binary data)
> if (c == 0x00) {
> fclose(f);
> return 1; // IS binary
> }
>
> // 2. check for C0 control codes (0x01-0x1F), excluding known
> // text formatting characters: 0x09 (Tab), 0x0A (LF), 0x0D (CR)
> if (c < 0x20) {
> if (c != 0x09 && c != 0x0A && c != 0x0D) {
> fclose(f);
> return 1; // IS binary (contains unexpected control code)
> }
> }
> }
> }
>
> fclose(f);
> return 0; // NOT binary
> }
>
Much smaller and with error handling for free:
bool binary( path pth )
{
ifstream ifs;
ifs.exceptions( ios_base::badbit );
ifs.open( pth, ios_base::binary | ios_base::ate );
streampos pos = ifs.tellg();
if( pos > (size_t)-1 ) // for 32 bit platforms with large files
throw ios_base::failure( "file too large", error_code(
(int)errc::file_too_large, generic_category() ) );
string buf( (size_t)pos, 0 );
ifs.seekg( 0 );
ifs.read( buf.data(), buf.size() );
auto check = []( unsigned char c ) { return c < 0x20 && c != '\r'
&& c != '\n' && c != '\t'; };
return find_if( buf.begin(), buf.end(), check ) == buf.end();
}
[toc] | [prev] | [next] | [standalone]
| From | Bonita Montero <Bonita.Montero@gmail.com> |
|---|---|
| Date | 2025-12-08 20:50 +0100 |
| Message-ID | <10h7a40$c6jh$1@raubtier-asyl.eternal-september.org> |
| In reply to | #395726 |
And if you like it fast:
bool binary( path pth )
{
static vector<bool> valid = []()
{
vector<bool> ret( numeric_limits<unsigned char>::max() );
for( size_t c = ret.size(); c--; )
ret[c] = c >= 0x20 || c == '\r' || c == '\n' || c == '\t';
return ret;
}();
ifstream ifs;
ifs.exceptions( ios_base::failbit | ios_base::badbit );
ifs.open( pth, ios_base::binary | ios_base::ate );
streampos pos = ifs.tellg();
if( pos > (size_t)-1 )
throw ios_base::failure( "file too large", error_code(
(int)errc::file_too_large, generic_category() ) );
string buf( (size_t)pos, 0 );
ifs.seekg( 0 );
ifs.read( buf.data(), buf.size() );
return find_if( buf.begin(), buf.end(), []( unsigned char c ) {
return !valid[c]; } ) == buf.end();
}
The cool thing about that is that the array valid is initialized only
once and threads-sfe.
[toc] | [prev] | [next] | [standalone]
| From | Bonita Montero <Bonita.Montero@gmail.com> |
|---|---|
| Date | 2025-12-09 15:09 +0100 |
| Message-ID | <10h9ago$s0hs$1@raubtier-asyl.eternal-september.org> |
| In reply to | #395728 |
I made a little benchmark that compares the table code against the
convention
&&-cascaded code. On my Zen4-PC the table code is about 25% faster with
clang.
I'm doing a AVX2 and AVX-512 version now. I guess it's about 20 - 30 times
faster.
#include <iostream>
#include <filesystem>
#include <fstream>
#include <algorithm>
#include <chrono>
using namespace std;
using namespace filesystem;
using namespace chrono;
template<bool Table>
bool binary( string const &buf );
int main()
{
ifstream ifs;
ifs.exceptions( ios_base::failbit | ios_base::badbit );
ifs.open( "main.cpp", ios_base::binary | ios_base::ate );
streampos pos = ifs.tellg();
if( pos > (size_t)-1 )
throw ios_base::failure( "file too large", error_code(
(int)errc::file_too_large, generic_category() ) );
string buf( (size_t)pos, 0 );
ifs.seekg( 0 );
ifs.read( buf.data(), buf.size() );
binary<true>( buf );
auto bench = [&]<bool Table>( bool_constant<Table> ) -> int
{
int ret = 0;
auto start = high_resolution_clock::now();
for( size_t r = 1'000'000; r; --r )
ret += binary<Table>( buf );
double secs = (double)duration_cast<nanoseconds>(
high_resolution_clock::now() - start ).count() / 1.0e9;
cout << (Table ? "table" : "check") << ": " << secs << endl;
return ret;
};
int ret = bench( false_type() );
ret += bench( true_type() );
return ret;
}
template<bool Table>
bool binary( string const &buf )
{
static auto invalid = []( unsigned char c ) static { return c <
0x20 && c != '\r' && c != '\n' && c != '\t'; };
if constexpr( Table )
{
static vector<char> invalidTbl = Table ? []()
{
vector<char> ret( numeric_limits<unsigned char>::max() );
for( size_t c = ret.size(); c--; )
ret[c] = invalid( (unsigned char)c );
return ret;
}() : vector<char>();
return find_if( buf.begin(), buf.end(), [&]( unsigned char c )
{ return invalidTbl[c]; } ) == buf.end();
}
else
return find_if( buf.begin(), buf.end(), invalid ) == buf.end();
}
[toc] | [prev] | [next] | [standalone]
| From | Bonita Montero <Bonita.Montero@gmail.com> |
|---|---|
| Date | 2025-12-10 09:18 +0100 |
| Message-ID | <10hbaab$1c9qn$1@raubtier-asyl.eternal-september.org> |
| In reply to | #395726 |
Now I've developed a benchmark which tests the static comparison approach
vs. the table approach vs. an AVX2-approach vs. an AVX-512 approach. This
are the results with clang 20:
check: 2.17442
table: 2.00056 (109%)
AVX-256: 0.183048 (1093%, 1188%)
AVX-512: 0.0639528 (286%, 3128%, 3400%)
The number in the brackets are the speedups against the before results.
So the AVX-512 solution is 30+ times than the byte-wise solutions.
This is the code:
#include <iostream>
#include <filesystem>
#include <fstream>
#include <algorithm>
#include <chrono>
#include <span>
#include <intrin.h>
#include <array>
#include <functional>
#include "inline.h"
using namespace std;
using namespace filesystem;
using namespace chrono;
template<bool Table>
bool binary( string const &buf );
template<bool Avx512>
bool binaryAvx( string const &buf );
int main()
{
ifstream ifs;
ifs.exceptions( ios_base::failbit | ios_base::badbit );
ifs.open( "main.cpp", ios_base::binary | ios_base::ate );
streampos pos = ifs.tellg();
if( pos > (size_t)-1 )
throw ios_base::failure( "file too large", error_code(
(int)errc::file_too_large, generic_category() ) );
string buf( (size_t)pos, 0 );
ifs.seekg( 0 );
ifs.read( buf.data(), buf.size() );
array<double, 4> results;
using test_fn = function<bool ( string const & )>;
auto bench = [&]( size_t i, char const *what, test_fn const &test )
L_FORCEINLINE -> int
{
int ret = 0;
auto start = high_resolution_clock::now();
#if defined(NDEBUG)
constexpr size_t N = 1'000'000;
#else
constexpr size_t N = 1'000;
#endif
for( size_t r = N; r; --r )
ret += test( buf );
double secs = (double)duration_cast<nanoseconds>(
high_resolution_clock::now() - start ).count() / 1.0e9;
cout << what << ": " << secs;
results[i] = secs;
if( i )
{
cout << " (";
do
{
cout << (int)(100.0 * results[--i] / secs + 0.5) << "%";
if( i )
cout << ", ";
} while( i );
cout << ")";
}
cout << endl;
return ret;
};
struct test { char const *descr; test_fn fn; };
array<test, 4> tests =
{
test( "check", +[]( string const &str ) -> int { return
binary<false>( str ); } ),
test( "table", +[]( string const &str ) -> int { return
binary<true>( str ); } ),
test( "AVX-256", +[]( string const &str ) -> int { return
binaryAvx<false>( str ); } ),
test( "AVX-512", +[]( string const &str ) -> int { return
binaryAvx<true>( str ); } )
};
int ret = 0;
for( size_t t = 0; test const &test : tests )
ret += bench( t++, test.descr, test.fn );
return ret;
}
template<bool Table>
bool binary( string const &buf )
{
static auto invalid = []( unsigned char c ) static { return c <
0x20 && c != '\r' && c != '\n' && c != '\t'; };
if constexpr( Table )
{
static vector<char> invalidTbl = Table ? []()
{
vector<char> ret( numeric_limits<unsigned char>::max() );
for( size_t c = ret.size(); c--; )
ret[c] = invalid( (unsigned char)c );
return ret;
}() : vector<char>();
return find_if( buf.begin(), buf.end(), [&]( unsigned char c )
{ return invalidTbl[c]; } ) == buf.end();
}
else
return find_if( buf.begin(), buf.end(), invalid ) == buf.end();
}
template<bool Avx512>
bool binaryAvx( string const &buf )
{
char const
*pBegin = buf.data(),
*pEnd = pBegin + buf.size();
if constexpr( Avx512 )
{
size_t
head = (size_t)pBegin & 63,
tail = (size_t)pEnd & 63;
span<__m512i const> range( (__m512i *)(pBegin - head), (__m512i
*)(pEnd - tail + (tail ? 64 : 0)) );
__m512i const
printable = _mm512_set1_epi8( (char)0x20 ),
cr = _mm512_set1_epi8( (char)'\r' ),
lf = _mm512_set1_epi8( (char)'\n' ),
tab = _mm512_set1_epi8( (char)'\t' );
uint64_t mask = (uint64_t)-1ll << head;
auto cur = range.begin(), end = range.end();
auto doChunk = [&]() -> bool
{
__m512i chunk = _mm512_loadu_epi8( (void *)to_address( cur ) );
uint64_t
spaMask = _mm512_cmpge_epu8_mask( chunk, printable ),
crMask = _mm512_cmpeq_epi8_mask( chunk, cr ),
lfMask = _mm512_cmpeq_epi8_mask( chunk, lf ),
tabMask = _mm512_cmpeq_epi8_mask( chunk, tab );
return ((spaMask | crMask | lfMask | tabMask) & mask) == mask;
};
for( ; cur != end - (bool)tail; ++cur, mask = -1ll )
if( !doChunk() )
return false;
if( tail )
{
mask = ~((uint64_t)-1ll << tail);
if( !doChunk() )
return false;
}
}
else
{
size_t
head = (size_t)pBegin & 31,
tail = (size_t)pEnd & 31;
span<__m256i const> range( (__m256i *)(pBegin - head), (__m256i
*)(pEnd - tail + (tail ? 32 : 0)) );
__m256i const
zero = _mm256_setzero_si256(),
printable = _mm256_set1_epi8( (char)0xE0 ),
cr = _mm256_set1_epi8( (char)'\r' ),
lf = _mm256_set1_epi8( (char)'\n' ),
tab = _mm256_set1_epi8( (char)'\t' );
uint32_t mask = (uint32_t)-1 << head;
auto cur = range.begin(), end = range.end();
auto doChunk = [&]() -> bool
{
__m256i chunk = _mm256_loadu_epi8( (void *)to_address( cur ) );
uint32_t
spaMask = ~_mm256_movemask_epi8( _mm256_cmpeq_epi8(
_mm256_and_si256( chunk, printable ), zero ) ),
crMask = _mm256_movemask_epi8( _mm256_cmpeq_epi8(
chunk, cr ) ),
lfMask = _mm256_movemask_epi8( _mm256_cmpeq_epi8(
chunk, lf ) ),
tabMask = _mm256_movemask_epi8 (_mm256_cmpeq_epi8(
chunk, tab ) );
return ((spaMask | crMask | lfMask | tabMask) & mask) == mask;
};
for( ; cur != end - (bool)tail; ++cur, mask = -1 )
if( !doChunk() )
return false;
if( tail )
{
mask = ~((uint32_t)-1 << tail);
if( !doChunk() )
return false;
}
}
return true;
}
[toc] | [prev] | [next] | [standalone]
| From | Keith Thompson <Keith.S.Thompson+u@gmail.com> |
|---|---|
| Date | 2025-12-08 14:43 -0800 |
| Message-ID | <871pl4fvbl.fsf@example.invalid> |
| In reply to | #395686 |
Michael Sanders <porkchop@invalid.foo> writes:
[...]
For yet another set of unreliable hueristics for guessing whether a file
is text or binary, you can take a look at Perl's built-in "-T" and "-B"
operators.
The "-T" and "-B" tests work as follows. The first block
or so of the file is examined to see if it is valid
UTF-8 that includes non-ASCII characters. If so, it's a
"-T" file. Otherwise, that same portion of the file is
examined for odd characters such as strange control codes
or characters with the high bit set. If more than a third
of the characters are strange, it's a "-B" file; otherwise
it's a "-T" file. Also, any file containing a zero byte
in the examined portion is considered a binary file. (If
executed within the scope of a use locale which includes
"LC_CTYPE", odd characters are anything that isn't a
printable nor space in the current locale.) If "-T" or
"-B" is used on a filehandle, the current IO buffer is
examined rather than the first block. Both "-T" and "-B"
return true on an empty file, or a file at EOF when testing
a filehandle. Because you have to read a file to do the "-T"
test, on most occasions you want to use a "-f" against the
file first, as in "next unless -f $file && -T $file".
It's not clear how big a "block" is. For an empty file, both -T
and -B are true. I don't know whether there are other cases where
both are true, or where both are false.
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
[toc] | [prev] | [next] | [standalone]
| From | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| Date | 2025-12-09 21:38 +0000 |
| Message-ID | <10ha4tg$142c2$1@dont-email.me> |
| In reply to | #395730 |
On Mon, 08 Dec 2025 14:43:58 -0800, Keith Thompson wrote: > For yet another set of unreliable hueristics for guessing whether a file > is text or binary, you can take a look at Perl's built-in "-T" and "-B" > operators. I guess the key finding in all of these cases really is unreliable. Heuristics is the only 'constant' ie - an educated guess. I wont win this battle, I can see it coming & then as James pointed out, the ambiguous stuff with unicode... -- :wq Mike Sanders
[toc] | [prev] | [next] | [standalone]
| From | Kaz Kylheku <046-301-5902@kylheku.com> |
|---|---|
| Date | 2025-12-11 17:33 +0000 |
| Message-ID | <20251211092839.304@kylheku.com> |
| In reply to | #395686 |
On 2025-12-06, Michael Sanders <porkchop@invalid.foo> wrote:
> Am I close? Missing anything you'd consider to be (or not) needed?
Hi Michael,
I contract for the the defense industry and badly need this function!
I am working with proposed code like:
if (is_binary_file(arg))
launch_nuclear_strike();
So I'm really sweating over the implementation, as you can imagine.
This thread has been very helpful.
I'm still leaning toward my paranoid functionw hich just checks that
every bit of every byte is either 0 or 1 to confirm that the binary
system is used.
In the I/O error case, I will cautiously return a a true value; we would
not want our side to lose due to a storage hardware issue.
--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca
[toc] | [prev] | [next] | [standalone]
| From | Bonita Montero <Bonita.Montero@gmail.com> |
|---|---|
| Date | 2025-12-11 19:10 +0100 |
| Message-ID | <10hf1cb$2d3ua$1@raubtier-asyl.eternal-september.org> |
| In reply to | #395787 |
Please take my AVX-512 code. It's that fast that your nuclear strike hits first and you won't get hit by enemy. Am 11.12.2025 um 18:33 schrieb Kaz Kylheku: > On 2025-12-06, Michael Sanders <porkchop@invalid.foo> wrote: >> Am I close? Missing anything you'd consider to be (or not) needed? > Hi Michael, > > I contract for the the defense industry and badly need this function! > > I am working with proposed code like: > > if (is_binary_file(arg)) > launch_nuclear_strike(); > > So I'm really sweating over the implementation, as you can imagine. > > This thread has been very helpful. > > I'm still leaning toward my paranoid functionw hich just checks that > every bit of every byte is either 0 or 1 to confirm that the binary > system is used. > > In the I/O error case, I will cautiously return a a true value; we would > not want our side to lose due to a storage hardware issue. >
[toc] | [prev] | [next] | [standalone]
| From | "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> |
|---|---|
| Date | 2025-12-11 14:56 -0800 |
| Message-ID | <10hfi75$2i060$1@dont-email.me> |
| In reply to | #395787 |
On 12/11/2025 9:33 AM, Kaz Kylheku wrote: > On 2025-12-06, Michael Sanders <porkchop@invalid.foo> wrote: >> Am I close? Missing anything you'd consider to be (or not) needed? > > Hi Michael, > > I contract for the the defense industry and badly need this function! > > I am working with proposed code like: > > if (is_binary_file(arg)) > launch_nuclear_strike(); any launch_biotoxic_strike(...) in there? ;^) rofl. > > So I'm really sweating over the implementation, as you can imagine. > > This thread has been very helpful. > > I'm still leaning toward my paranoid functionw hich just checks that > every bit of every byte is either 0 or 1 to confirm that the binary > system is used. > > In the I/O error case, I will cautiously return a a true value; we would > not want our side to lose due to a storage hardware issue. > oh my! ;^D
[toc] | [prev] | [next] | [standalone]
| From | James Kuyper <jameskuyper@alumni.caltech.edu> |
|---|---|
| Date | 2025-12-11 18:15 -0500 |
| Message-ID | <10hfja3$2hqlt$1@dont-email.me> |
| In reply to | #395787 |
On 2025-12-11 12:33, Kaz Kylheku wrote: ... > I'm still leaning toward my paranoid functionw hich just checks that > every bit of every byte is either 0 or 1 to confirm that the binary > system is used. I'd be very interested in seeing how you implement that test, and even more interested in what the test data looks like that you use to confirm that a failure of that test is correctly flagged. :-)
[toc] | [prev] | [next] | [standalone]
| From | Janis Papanagnou <janis_papanagnou+ng@hotmail.com> |
|---|---|
| Date | 2025-12-12 02:19 +0100 |
| Message-ID | <10hfqil$24uuc$1@dont-email.me> |
| In reply to | #395787 |
On 2025-12-11 18:33, Kaz Kylheku wrote:
> On 2025-12-06, Michael Sanders <porkchop@invalid.foo> wrote:
>> Am I close? Missing anything you'd consider to be (or not) needed?
>
> Hi Michael,
>
> I contract for the the defense industry and badly need this function!
>
> I am working with proposed code like:
>
> if (is_binary_file(arg))
> launch_nuclear_strike();
else
negotiate_peace_conditions(arg);
I think it's a waste of information to identify some 'arg' as text
and not assume it to be a negotiation proposal for peace treaties!
Or would that be considered just unnecessary feature creep? - Just
bloating the code and having negative impact on runtime performance?
(A few milliseconds could certainly make a difference here between
victory or defeat!)
>
> [...]
>
> In the I/O error case, I will cautiously return a a true value; we would
> not want our side to lose due to a storage hardware issue.
A very considerate decision. Kudos!
Janis
LOL - you made my day, Kaz!
[toc] | [prev] | [next] | [standalone]
| From | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| Date | 2025-12-14 08:27 +0000 |
| Message-ID | <10hlsdr$rhu0$1@dont-email.me> |
| In reply to | #395787 |
On Thu, 11 Dec 2025 17:33:43 -0000 (UTC), Kaz Kylheku wrote: > Hi Michael, > > I contract for the the defense industry and badly need this function! Certainly! CRYPTOGRAM 11/400 > PJJU HFQR FSI HFWWD TS PLAIN-TEXT + HINTS > E P N A -- :wq Mike Sanders
[toc] | [prev] | [next] | [standalone]
| From | Lynn McGuire <lynnmcguire5@gmail.com> |
|---|---|
| Date | 2025-12-17 00:52 -0600 |
| Message-ID | <10htk03$3a4nb$1@dont-email.me> |
| In reply to | #395787 |
On 12/11/2025 11:33 AM, Kaz Kylheku wrote: > On 2025-12-06, Michael Sanders <porkchop@invalid.foo> wrote: >> Am I close? Missing anything you'd consider to be (or not) needed? > > Hi Michael, > > I contract for the the defense industry and badly need this function! > > I am working with proposed code like: > > if (is_binary_file(arg)) > launch_nuclear_strike(); > > So I'm really sweating over the implementation, as you can imagine. > > This thread has been very helpful. > > I'm still leaning toward my paranoid functionw hich just checks that > every bit of every byte is either 0 or 1 to confirm that the binary > system is used. > > In the I/O error case, I will cautiously return a a true value; we would > not want our side to lose due to a storage hardware issue. The probability of this function being correct may be less than 50%. I would hope that any military function would be much higher reliability than this. Lynn
[toc] | [prev] | [standalone]
Page 6 of 6 — ← Prev page 1 2 3 4 5 [6]
Back to top | Article view | comp.lang.c
csiph-web