Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.c > #395686 > unrolled thread
| Started by | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| First post | 2025-12-06 01:05 +0000 |
| Last post | 2025-12-17 00:52 -0600 |
| Articles | 20 on this page of 119 — 23 participants |
Back to article view | Back to comp.lang.c
is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-06 01:05 +0000
Re: is_binary_file() Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2025-12-06 01:41 +0000
Re: is_binary_file() Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2025-12-06 02:00 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 17:40 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 11:35 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-10 15:07 +0000
Re: is_binary_file() Michael S <already5chosen@yahoo.com> - 2025-12-10 19:00 +0200
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-10 17:18 +0000
Re: is_binary_file() Richard Heathfield <rjh@cpax.org.uk> - 2025-12-10 19:42 +0000
Re: is_binary_file() bart <bc@freeuk.com> - 2025-12-10 22:37 +0000
Re: is_binary_file() Paul <nospam@needed.invalid> - 2025-12-10 22:35 -0500
Re: is_binary_file() bart <bc@freeuk.com> - 2025-12-11 11:46 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-11 12:53 +0100
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 18:42 +0000
Re: is_binary_file() Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2025-12-10 15:58 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 18:44 +0000
Re: is_binary_file() James Kuyper <jameskuyper@alumni.caltech.edu> - 2025-12-10 12:46 -0500
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 18:45 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 18:41 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 20:57 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-10 22:07 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-11 01:09 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-11 12:33 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-12 19:25 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-12 22:54 +0000
Re: is_binary_file() "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-12-12 15:33 -0800
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-13 00:20 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-13 02:32 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-16 00:26 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-16 17:24 +0100
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-17 03:19 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-17 07:57 +0100
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-17 19:35 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-18 08:44 +0100
Re: is_binary_file() bart <bc@freeuk.com> - 2025-12-18 12:49 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-18 14:06 +0100
Re: is_binary_file() gazelle@shell.xmission.com (Kenny McCormack) - 2025-12-18 13:17 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-18 16:03 +0100
Re: is_binary_file() Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2025-12-05 17:42 -0800
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-06 17:37 +0000
Re: is_binary_file() Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2025-12-06 16:05 -0800
Re: is_binary_file() Louis Krupp <lkrupp@invalid.pssw.com.invalid> - 2025-12-07 03:43 -0700
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-07 16:47 +0000
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-27 03:18 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 17:46 +0000
Re: is_binary_file() Kaz Kylheku <046-301-5902@kylheku.com> - 2025-12-06 02:42 +0000
Re: is_binary_file() bart <bc@freeuk.com> - 2025-12-06 12:42 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-06 17:40 +0000
Re: is_binary_file() Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2025-12-06 18:04 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-06 19:06 +0000
Re: is_binary_file() Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2025-12-06 21:16 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 17:48 +0000
Re: is_binary_file() Kaz Kylheku <046-301-5902@kylheku.com> - 2025-12-08 19:26 +0000
Re: is_binary_file() Richard Heathfield <rjh@cpax.org.uk> - 2025-12-08 19:42 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-09 21:49 +0000
Re: is_binary_file() Paul <nospam@needed.invalid> - 2025-12-06 03:14 -0500
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 17:56 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-08 20:16 +0000
Re: is_binary_file() David Brown <david.brown@hesbynett.no> - 2025-12-09 09:03 +0100
Re: is_binary_file() Richard Heathfield <rjh@cpax.org.uk> - 2025-12-09 09:43 +0000
Re: is_binary_file() Richard Harnden <richard.nospam@gmail.invalid> - 2025-12-09 10:17 +0000
Re: is_binary_file() Kaz Kylheku <046-301-5902@kylheku.com> - 2025-12-09 20:15 +0000
Re: is_binary_file() tTh <tth@none.invalid> - 2025-12-09 12:22 +0100
Re: is_binary_file() Paul <nospam@needed.invalid> - 2025-12-09 20:26 -0500
Re: is_binary_file() Paul <nospam@needed.invalid> - 2025-12-09 06:38 -0500
Re: is_binary_file() Michael S <already5chosen@yahoo.com> - 2025-12-09 17:31 +0200
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-28 02:49 +0000
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-28 00:12 +0000
Re: is_binary_file() richard@cogsci.ed.ac.uk (Richard Tobin) - 2025-12-28 00:43 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-06 17:33 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-07 19:04 +0100
Re: is_binary_file() James Kuyper <jameskuyper@alumni.caltech.edu> - 2025-12-06 20:37 -0500
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 18:02 +0000
Re: is_binary_file() James Kuyper <jameskuyper@alumni.caltech.edu> - 2025-12-09 16:29 -0500
Re: is_binary_file() Michael S <already5chosen@yahoo.com> - 2025-12-10 11:21 +0200
Re: is_binary_file() James Kuyper <jameskuyper@alumni.caltech.edu> - 2025-12-10 12:48 -0500
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 11:38 +0000
Re: is_binary_file() antispam@fricas.org (Waldek Hebisch) - 2025-12-07 03:43 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 18:04 +0000
Re: is_binary_file() bart <bc@freeuk.com> - 2025-12-08 18:44 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-09 19:53 +0000
Re: is_binary_file() Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2025-12-09 15:42 -0800
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 11:41 +0000
Re: is_binary_file() Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2025-12-10 15:20 -0800
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-10 23:59 +0000
Re: is_binary_file() James Kuyper <jameskuyper@alumni.caltech.edu> - 2025-12-09 16:23 -0500
Re: is_binary_file() Richard Harnden <richard.nospam@gmail.invalid> - 2025-12-07 19:01 +0000
Re: is_binary_file() Richard Heathfield <rjh@cpax.org.uk> - 2025-12-07 21:51 +0000
Re: is_binary_file() Richard Harnden <richard.nospam@gmail.invalid> - 2025-12-07 22:49 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-08 13:51 +0100
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-08 16:04 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-08 19:27 +0100
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-27 05:51 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-29 16:06 +0100
Re: is_binary_file() mjos_examine <m6502x64@gmail.com> - 2025-12-29 11:49 -0500
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-29 20:49 +0100
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-30 01:52 +0000
Re: is_binary_file() scott@slp53.sl.home (Scott Lurndal) - 2025-12-08 16:02 +0000
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 18:07 +0000
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-27 03:13 +0000
Re: is_binary_file() Paul <nospam@needed.invalid> - 2025-12-27 01:28 -0500
Re: is_binary_file() Lawrence D’Oliveiro <ldo@nz.invalid> - 2025-12-27 21:27 +0000
Re: is_binary_file() antispam@fricas.org (Waldek Hebisch) - 2025-12-28 05:46 +0000
Re: is_binary_file() "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-12-07 14:42 -0800
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-08 18:09 +0000
Re: is_binary_file() "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-12-09 12:45 -0800
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-08 20:36 +0100
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-08 20:50 +0100
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-09 15:09 +0100
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-10 09:18 +0100
Re: is_binary_file() Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2025-12-08 14:43 -0800
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-09 21:38 +0000
Re: is_binary_file() Kaz Kylheku <046-301-5902@kylheku.com> - 2025-12-11 17:33 +0000
Re: is_binary_file() Bonita Montero <Bonita.Montero@gmail.com> - 2025-12-11 19:10 +0100
Re: is_binary_file() "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2025-12-11 14:56 -0800
Re: is_binary_file() James Kuyper <jameskuyper@alumni.caltech.edu> - 2025-12-11 18:15 -0500
Re: is_binary_file() Janis Papanagnou <janis_papanagnou+ng@hotmail.com> - 2025-12-12 02:19 +0100
Re: is_binary_file() Michael Sanders <porkchop@invalid.foo> - 2025-12-14 08:27 +0000
Re: is_binary_file() Lynn McGuire <lynnmcguire5@gmail.com> - 2025-12-17 00:52 -0600
Page 2 of 6 — ← Prev page 1 [2] 3 4 5 6 Next page →
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2025-12-10 22:07 +0000 |
| Message-ID | <wam_Q.784$wOxf.255@fx39.iad> |
| In reply to | #395771 |
Michael Sanders <porkchop@invalid.foo> writes:
>On Wed, 10 Dec 2025 18:41:22 -0000 (UTC), Michael Sanders wrote:
>
>> Last version for me (I have to pivot to other things).
>>
>> [...]
>
>smaller look up table still + bit shifting!
>
>*fastest implantation yet* but virtually unreadable =(
>
>#include <stdio.h>
>#include <stddef.h>
>#include <stdint.h>
>
>// is_text_file()
>// Returns:
>// -1 : could not open file
>// 0 : is NOT a text file (binary indicators found)
>// 1 : is PROBABLY a text file (no strong binary signatures)
>
>int is_text_file(const char *path) {
> FILE *f = fopen(path, "rb");
> if (!f) return -1;
>
> unsigned char chunk[4096];
> size_t n, i;
>
> // 128-bit bitmask (16 bytes × 8 bits / byte), 1=allowed, 0=disallowed
> // Allowed bytes: TAB(0x09), LF(0x0A), CR(0x0D), printable ASCII 0x20–0x7E
>
> static const uint8_t MASK[16] = {
> 0x00, 0x24, 0x00, 0x00, // 0x00–0x0F: TAB(09), LF(0A), CR(0D)
> 0xFF, 0xFF, 0xFF, 0xFF, // 0x10–0x2F: SPC!"#$%&'()*+,-./
> 0xFF, 0xFF, 0xFF, 0xFF, // 0x30–0x4F: 0123456789:;<=>?@
> 0xFF, 0xFF, 0xFF, 0x7F // 0x50–0x7F: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdef...
> };
>
> while ((n = fread(chunk, 1, sizeof(chunk), f)) > 0) {
> for (i = 0; i < n; i++) {
> if (chunk[i] < 128 && !(MASK[chunk[i] >> 3] & (1 << (chunk[i] & 7)))) {
> fclose(f);
> return 0; // binary indicator found
> }
> // bytes >= 128 are accepted as probably text
Typically a soi disant extended ASCII character set (e.g. ISO-8859-1)
have the first 32 bytes starting at 128 defined as control characters.
https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Code_page_layout
[toc] | [prev] | [next] | [standalone]
| From | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| Date | 2025-12-11 01:09 +0000 |
| Message-ID | <10hd5l6$1shno$1@dont-email.me> |
| In reply to | #395772 |
On Wed, 10 Dec 2025 22:07:24 GMT, Scott Lurndal wrote:
> Typically a soi disant extended ASCII character set (e.g. ISO-8859-1)
> have the first 32 bytes starting at 128 defined as control characters.
>
> https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Code_page_layout
Many thanks Scott. Here's my final stab at the idea.
Beware word-wrap...
#include <stdio.h>
#include <stdint.h>
/*
* is_text_file()
*
* Determines whether a file is "probably text" or binary, using a heuristic
* based on mostly printable characters.
*
* Detection modes:
* TEXT_LOOSE - Allows ASCII printable bytes (0x20–0x7E), TAB/LF/CR,
* and all high-bit bytes (>=128). Tolerant for UTF-8 or
* ISO-8859-1 text.
* TEXT_STRICT - Rejects ASCII control characters (0x00–0x08, 0x0B–0x0C,
* 0x0E–0x1F) and C1 controls (0x80–0x9F). Counts only
* clearly printable bytes.
* TEXT_ISO8859_1 - Accepts ASCII printable (0x20–0x7E), ISO-8859-1
* printable bytes (0xA0–0xFF), and TAB/LF/CR. Rejects
* C1 controls (0x80–0x9F).
*
* Returns:
* 1 file is probably text (>=90% printable characters)
* 0 file is probably binary (too many non-printable characters)
* -1 empty file
* -2 could not open file
*/
typedef enum {
TEXT_LOOSE, // mostly printable: ASCII + high-bit
TEXT_STRICT, // stricter: reject C1 controls
TEXT_ISO8859_1 // ISO-8859-1 printable (0x20–0x7E + 0xA0–0xFF)
} text_mode_t;
static const uint8_t MASK[16] = {
0x00, 0x24, 0x00, 0x00, // 0x00–0x0F: TAB(09), LF(0A), CR(0D)
0xFF, 0xFF, 0xFF, 0xFF, // 0x10–0x2F: SPC!"#$%&'()*+,-./
0xFF, 0xFF, 0xFF, 0xFF, // 0x30–0x4F: 0123456789:;<=>?@
0xFF, 0xFF, 0xFF, 0x7F // 0x50–0x7F: A–Z [\]^_` a–z (exclude DEL)
};
int is_text_file(const char *path, text_mode_t mode) {
FILE *f = fopen(path, "rb");
if (!f) return -2;
unsigned char chunk[4096];
uint64_t n, i, good = 0, total = 0;
while ((n = fread(chunk, 1, sizeof(chunk), f)) > 0) {
total += n;
for (i = 0; i < n; i++) {
unsigned char c = chunk[i];
switch (mode) {
case TEXT_LOOSE:
if (c >= 128 || (c < 128 && (MASK[c >> 3] & (1 << (c & 7))))) good++;
break;
case TEXT_STRICT: // reject C1 controls 0x80–0x9F
if ((c >= 128 && c <= 159) || (c < 128 && !(MASK[c >> 3] & (1 << (c & 7))))) {
// bad byte, do not count...
} else good++;
break;
case TEXT_ISO8859_1: // accept 0x20–0x7E + 0xA0–0xFF, reject C1 controls
if ((c >= 0x20 && c <= 0x7E) || (c >= 0xA0 && c <= 0xFF)
|| c == 0x09 || c == 0x0A || c == 0x0D) { good++; }
break;
}
}
}
fclose(f);
if (total == 0) return -1; // empty file
return (good * 10 >= total * 9) ? 1 : 0; // 90% threshold
}
--
:wq
Mike Sanders
[toc] | [prev] | [next] | [standalone]
| From | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| Date | 2025-12-11 12:33 +0000 |
| Message-ID | <10hedm9$26p65$1@dont-email.me> |
| In reply to | #395779 |
On Thu, 11 Dec 2025 01:09:59 -0000 (UTC), Michael Sanders wrote:
> [...]
>
> if (c >= 128 || (c < 128 && (MASK[c >> 3] & (1 << (c & 7))))) good++;
>
> [...]
Thinking about it more, the bit-twiddling method while fast,
is certainly not very readable/maintainable. Those who might
want to use any of the variations I've written, will best be
served using the one shown below. Not all the bells & whistles
of the prior offering, but sometimes that's good thing.
Note: If you keep map[] 'out in the open' (globally exposed)
its only computed once at runtime instead everytime...
Well off to work for me.
#include <stdio.h>
#include <stdint.h>
/*
* is_text_file()
*
* Determines whether a file is 'probably text' based on ISO-8859-1 rules.
* Uses a precomputed lookup table for fast byte validation.
*
* Valid bytes:
* - ASCII printable: 0x20–0x7E
* - ISO-8859-1 high printable: 0xA0–0xFF
* - Whitespace/control: TAB (0x09), LF (0x0A), CR (0x0D)
*
* Invalid bytes (binary indicators):
* - NULL byte (0x00)
* - C0 controls (0x01–0x08, 0x0B–0x0C, 0x0E–0x1F)
* - DEL (0x7F)
* - C1 controls (0x80–0x9F)
*
* Returns:
* 1 - file is considered text
* 0 - file is considered binary
* -1 - could not open file
*/
static const uint8_t map[256] = {
0,0,0,0,0,0,0,0,0,1,1,0,0,1,0,0, // 00
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 10
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 20
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 30
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 40
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 50
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 60
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0, // 70
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 80
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 90
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // A0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // B0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // C0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // D0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // E0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 // F0
};
int is_text_file(const char *path) {
FILE *f = fopen(path, "rb");
if (!f) return -1; // could not open file
// larger chunk size means less 'touching' the drive
unsigned char chunk[65536];
size_t n, i;
while ((n = fread(chunk, 1, sizeof(chunk), f)) > 0) {
for (i = 0; i < n; i++) {
if (!map[chunk[i]]) {
fclose(f);
return 0; // binary detected
}
}
}
fclose(f);
return 1; // probally text
}
// eof
--
:wq
Mike Sanders
[toc] | [prev] | [next] | [standalone]
| From | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| Date | 2025-12-12 19:25 +0000 |
| Message-ID | <10hhq7k$39tsd$1@dont-email.me> |
| In reply to | #395783 |
On Thu, 11 Dec 2025 12:33:14 -0000 (UTC), Michael Sanders wrote:
> static const uint8_t map[256] = {...
added 'plugin' maps...
#include <stdio.h>
#include <stdint.h>
/*
* map_strict[]
*
* Valid bytes:
* - ASCII printable: 0x20–0x7E
* - ISO-8859-1 high printable: 0xA0–0xFF
* - Whitespace/control: TAB (0x09), LF (0x0A), CR (0x0D)
*
* Invalid bytes (binary indicators):
* - NULL byte (0x00)
* - C0 controls (0x01–0x08, 0x0B–0x0C, 0x0E–0x1F)
* - DEL (0x7F)
* - C1 controls (0x80–0x9F)
*/
static const uint8_t map_strict[256] = {
0,0,0,0,0,0,0,0,0,1,1,0,0,1,0,0, // 00
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 10
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 20
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 30
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 40
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 50
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 60
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0, // 70
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 80
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 90
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // A0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // B0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // C0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // D0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // E0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 // F0
};
/*
* map_loose[]
*
* Valid bytes:
* - ASCII printable characters: 0x20–0x7E
* - Whitespace/control characters: TAB (0x09), LF (0x0A), CR (0x0D)
* - High bytes: 0x80–0xFF
*
* Invalid bytes (binary indicators):
* - NULL byte: 0x00
* - C0 control codes: 0x01–0x08, 0x0B–0x0C, 0x0E–0x1F
* - DEL character: 0x7F
*/
static const uint8_t map_loose[256] = {
0,0,0,0,0,0,0,0,0,1,1,0,0,1,0,0, // 00
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 10
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 20
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 30
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 40
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 50
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 60
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 70
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 80
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 90
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // A0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // B0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // C0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // D0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // E0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 // F0
};
/*
* is_text_file()
*
* just plugin in your own map[]...
*
* Returns:
* 1 - text
* 0 - binary
* -1 - could not open
*/
int is_text_file(const char *path, const uint8_t map[256]) {
FILE *f = fopen(path, "rb");
if (!f) return -1; // could not open file
// 4KB: 4096, 8KB: 8192, 16KB: 16384, 32KB: 32768, 64KB: 65536
unsigned char buf[65536];
size_t n, i;
while ((n = fread(buf, 1, sizeof(buf), f)) > 0) {
for (i = 0; i < n; i++) {
if (!map[buf[i]]) {
fclose(f);
return 0; // not text (binary indicators)
}
}
}
fclose(f);
return 1; // probably text
}
// eof
--
:wq
Mike Sanders
[toc] | [prev] | [next] | [standalone]
| From | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| Date | 2025-12-12 22:54 +0000 |
| Message-ID | <10hi6g2$3ekvo$1@dont-email.me> |
| In reply to | #395797 |
On Fri, 12 Dec 2025 19:25:41 -0000 (UTC), Michael Sanders wrote:
> [...]
Done.
Features...
- plugin maps
- follows sylinks
- rejects directories, devices, sockets
#include <stdio.h>
#include <stdint.h>
#include <sys/stat.h>
/*
* map_strict[]
*
* Valid bytes:
* - ASCII printable: 0x20–0x7E
* - ISO-8859-1 high printable: 0xA0–0xFF
* - Whitespace/control: TAB (0x09), LF (0x0A), CR (0x0D)
*
* Invalid bytes (binary indicators):
* - NULL byte (0x00)
* - C0 controls (0x01–0x08, 0x0B–0x0C, 0x0E–0x1F)
* - DEL (0x7F)
* - C1 controls (0x80–0x9F)
*/
static const uint8_t map_strict[256] = {
0,0,0,0,0,0,0,0,0,1,1,0,0,1,0,0, // 00
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 10
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 20
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 30
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 40
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 50
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 60
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0, // 70
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 80
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 90
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // A0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // B0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // C0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // D0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // E0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 // F0
};
/*
* map_loose[]
*
* Valid bytes:
* - ASCII printable characters: 0x20–0x7E
* - Whitespace/control characters: TAB (0x09), LF (0x0A), CR (0x0D)
* - High bytes: 0x80–0xFF
*
* Invalid bytes (binary indicators):
* - NULL byte: 0x00
* - C0 control codes: 0x01–0x08, 0x0B–0x0C, 0x0E–0x1F
* - DEL character: 0x7F
*/
static const uint8_t map_loose[256] = {
0,0,0,0,0,0,0,0,0,1,1,0,0,1,0,0, // 00
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 10
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 20
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 30
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 40
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 50
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 60
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 70
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 80
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 90
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // A0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // B0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // C0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // D0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // E0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 // F0
};
/*
* is_text_file()
*
* just plug in your own map[]...
*
* Returns:
* 1 - text
* 0 - binary indicator
* -1 - could not open
*/
int is_text_file(const char *path, const uint8_t map[256]) {
// now we follow symlinks...
struct stat st;
if (stat(path, &st) != 0) return -1; // can not access file
if (!S_ISREG(st.st_mode)) return -1; // reject: directories/devices/sockets
FILE *f = fopen(path, "rb");
if (!f) return -1; // could not open file
// 4KB: 4096, 8KB: 8192, 16KB: 16384, 32KB: 32768, 64KB: 65536
unsigned char buf[16384];
size_t n, i;
while ((n = fread(buf, 1, sizeof(buf), f)) > 0) {
for (i = 0; i < n; i++) {
if (!map[buf[i]]) {
fclose(f);
return 0; // not text (binary indicator detected)
}
}
}
fclose(f);
return 1; // probally text
}
// eof
--
:wq
Mike Sanders
[toc] | [prev] | [next] | [standalone]
| From | "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> |
|---|---|
| Date | 2025-12-12 15:33 -0800 |
| Message-ID | <10hi8nd$3fdu2$2@dont-email.me> |
| In reply to | #395798 |
On 12/12/2025 2:54 PM, Michael Sanders wrote: [...] > fclose(f); > return 1; // probally text > } define the probability? Say in 0...1? [...]
[toc] | [prev] | [next] | [standalone]
| From | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| Date | 2025-12-13 00:20 +0000 |
| Message-ID | <10hibh0$3gd1h$1@dont-email.me> |
| In reply to | #395799 |
On Fri, 12 Dec 2025 15:33:01 -0800, Chris M. Thomasson wrote: > On 12/12/2025 2:54 PM, Michael Sanders wrote: > [...] >> fclose(f); >> return 1; // probally text >> } > > define the probability? Say in 0...1? > > [...] Add it Chris & I'll roll it in =) Me? I'd go with steps of say, 10% just to make it human-friendly, but that's just me. -- :wq Mike Sanders
[toc] | [prev] | [next] | [standalone]
| From | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| Date | 2025-12-13 02:32 +0000 |
| Message-ID | <10hij8p$3j03j$1@dont-email.me> |
| In reply to | #395800 |
On Sat, 13 Dec 2025 00:20:48 -0000 (UTC), Michael Sanders wrote: > On Fri, 12 Dec 2025 15:33:01 -0800, Chris M. Thomasson wrote: > >> On 12/12/2025 2:54 PM, Michael Sanders wrote: >> [...] >>> fclose(f); >>> return 1; // probally text >>> } >> >> define the probability? Say in 0...1? >> >> [...] > > Add it Chris & I'll roll it in =) > > Me? I'd go with steps of say, 10% just to > make it human-friendly, but that's just me. just thinking out loud about probabilities... int is_text_file(const char *path, const uint8_t map[256], int probability) -- :wq Mike Sanders
[toc] | [prev] | [next] | [standalone]
| From | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| Date | 2025-12-16 00:26 +0000 |
| Message-ID | <10hq8vk$29cl8$1@dont-email.me> |
| In reply to | #395799 |
On Fri, 12 Dec 2025 15:33:01 -0800, Chris M. Thomasson wrote:
> define the probability? Say in 0...1?
Probabilities for Chris... Enjoy.
#include <stdio.h>
#include <stdint.h>
#include <sys/stat.h>
/*
* map_strict[]
*
* Valid bytes:
* - ASCII printable: 0x20–0x7E
* - ISO-8859-1 high printable: 0xA0–0xFF
* - Whitespace: TAB (0x09), LF (0x0A), CR (0x0D)
*
* Invalid bytes (binary indicators):
* - NULL byte (0x00)
* - C0 controls (0x01–0x08, 0x0B–0x0C, 0x0E–0x1F)
* - DEL (0x7F)
* - C1 controls (0x80–0x9F)
*/
static const uint8_t map_strict[256] = {
0,0,0,0,0,0,0,0,0,1,1,0,0,1,0,0, // 00
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 10
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 20
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 30
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 40
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 50
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 60
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0, // 70
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 80
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 90
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // A0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // B0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // C0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // D0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // E0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 // F0
};
/*
* map_loose[]
*
* Valid bytes:
* - ASCII printable characters: 0x20–0x7E
* - Whitespace: TAB (0x09), LF (0x0A), CR (0x0D)
* - High bytes: 0x80–0xFF
*
* Invalid bytes (binary indicators):
* - NULL byte: 0x00
* - C0 control codes: 0x01–0x08, 0x0B–0x0C, 0x0E–0x1F
* - DEL character: 0x7F
*/
static const uint8_t map_loose[256] = {
0,0,0,0,0,0,0,0,0,1,1,0,0,1,0,0, // 00
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 10
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 20
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 30
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 40
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 50
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 60
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 70
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 80
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 90
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // A0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // B0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // C0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // D0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // E0
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 // F0
};
/*
* is_text_file(const char *path, const uint8_t map[256], double probability)
*
* Determines whether a file is likely a text file driven by a byte map
* & probability threshold.
*
* Signature:
*
* path
* Pathname of the file to examine. The function follows symlinks
* & only operates on regular files.
*
* map[256]
* A 256-entry lookup table indexed by byte value (0–255).
* Each entry must be either:
* 1 -> byte is considered valid text
* 0 -> byte is considered invalid (binary indicator)
*
* probability
* Maximum allowed percentage (0.0 to 100.0) of invalid bytes.
* If the percentage of bytes for which map[byte] == 0 exceeds
* this value, the file is classified as non-text.
*
* Behavior:
* - Reads the file in binary mode.
* - Counts total bytes & 'bad' bytes (map[] == 0).
* - Returns 1 if bad byte percentage <= probability.
* - Returns 0 if bad byte percentage > probability.
* - Returns -1 on access/open errors or if the file is not regular.
* - Function exits early if impossible to satisfy probability threshold.
*
* Returns:
* 1 - text
* 0 - binary indicator
* -1 - could not open
*
* Example with 94.3% probability threshold:
* is_text_file("./example.txt", your_map[256], 94.3)
*
*/
int is_text_file(const char *path, const uint8_t map[256], double probability) {
struct stat st;
if (stat(path, &st) != 0) return -1; // can not access file
if (!S_ISREG(st.st_mode)) return -1; // reject: directories/devices/sockets
FILE *f = fopen(path, "rb"); if (!f) return -1; // could not open file
// 4KB: 4096, 8KB: 8192, 16KB: 16384, 32KB: 32768, 64KB: 65536
unsigned char buf[16384];
size_t n, i;
size_t total = 0;
size_t bad = 0;
while ((n = fread(buf, 1, sizeof(buf), f)) > 0) {
for (i = 0; i < n; i++) {
total++;
if (!map[buf[i]]) bad++;
if (((double)bad / total) * 100.0 > probability) {
fclose(f);
return 0; // not text
}
}
}
fclose(f);
if (total == 0) return 0; // empty file or total binary...
// final probability check
return (((double)bad / total) * 100.0 <= probability) ? 1 : 0;
}
--
:wq
Mike Sanders
[toc] | [prev] | [next] | [standalone]
| From | Bonita Montero <Bonita.Montero@gmail.com> |
|---|---|
| Date | 2025-12-16 17:24 +0100 |
| Message-ID | <10hs12r$2s3mh$2@raubtier-asyl.eternal-september.org> |
| In reply to | #395828 |
How long have you been dealing with that ? 10 days ?
I finished the AVX-512 version in C++ in two hours.
Am 16.12.2025 um 01:26 schrieb Michael Sanders:
> On Fri, 12 Dec 2025 15:33:01 -0800, Chris M. Thomasson wrote:
>
>> define the probability? Say in 0...1?
> Probabilities for Chris... Enjoy.
>
> #include <stdio.h>
> #include <stdint.h>
> #include <sys/stat.h>
>
> /*
> * map_strict[]
> *
> * Valid bytes:
> * - ASCII printable: 0x20–0x7E
> * - ISO-8859-1 high printable: 0xA0–0xFF
> * - Whitespace: TAB (0x09), LF (0x0A), CR (0x0D)
> *
> * Invalid bytes (binary indicators):
> * - NULL byte (0x00)
> * - C0 controls (0x01–0x08, 0x0B–0x0C, 0x0E–0x1F)
> * - DEL (0x7F)
> * - C1 controls (0x80–0x9F)
> */
>
> static const uint8_t map_strict[256] = {
> 0,0,0,0,0,0,0,0,0,1,1,0,0,1,0,0, // 00
> 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 10
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 20
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 30
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 40
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 50
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 60
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0, // 70
> 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 80
> 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 90
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // A0
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // B0
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // C0
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // D0
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // E0
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 // F0
> };
>
> /*
> * map_loose[]
> *
> * Valid bytes:
> * - ASCII printable characters: 0x20–0x7E
> * - Whitespace: TAB (0x09), LF (0x0A), CR (0x0D)
> * - High bytes: 0x80–0xFF
> *
> * Invalid bytes (binary indicators):
> * - NULL byte: 0x00
> * - C0 control codes: 0x01–0x08, 0x0B–0x0C, 0x0E–0x1F
> * - DEL character: 0x7F
> */
>
> static const uint8_t map_loose[256] = {
> 0,0,0,0,0,0,0,0,0,1,1,0,0,1,0,0, // 00
> 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 10
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 20
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 30
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 40
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 50
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 60
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 70
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 80
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // 90
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // A0
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // B0
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // C0
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // D0
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // E0
> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 // F0
> };
>
> /*
> * is_text_file(const char *path, const uint8_t map[256], double probability)
> *
> * Determines whether a file is likely a text file driven by a byte map
> * & probability threshold.
> *
> * Signature:
> *
> * path
> * Pathname of the file to examine. The function follows symlinks
> * & only operates on regular files.
> *
> * map[256]
> * A 256-entry lookup table indexed by byte value (0–255).
> * Each entry must be either:
> * 1 -> byte is considered valid text
> * 0 -> byte is considered invalid (binary indicator)
> *
> * probability
> * Maximum allowed percentage (0.0 to 100.0) of invalid bytes.
> * If the percentage of bytes for which map[byte] == 0 exceeds
> * this value, the file is classified as non-text.
> *
> * Behavior:
> * - Reads the file in binary mode.
> * - Counts total bytes & 'bad' bytes (map[] == 0).
> * - Returns 1 if bad byte percentage <= probability.
> * - Returns 0 if bad byte percentage > probability.
> * - Returns -1 on access/open errors or if the file is not regular.
> * - Function exits early if impossible to satisfy probability threshold.
> *
> * Returns:
> * 1 - text
> * 0 - binary indicator
> * -1 - could not open
> *
> * Example with 94.3% probability threshold:
> * is_text_file("./example.txt", your_map[256], 94.3)
> *
> */
>
> int is_text_file(const char *path, const uint8_t map[256], double probability) {
>
> struct stat st;
> if (stat(path, &st) != 0) return -1; // can not access file
> if (!S_ISREG(st.st_mode)) return -1; // reject: directories/devices/sockets
> FILE *f = fopen(path, "rb"); if (!f) return -1; // could not open file
>
> // 4KB: 4096, 8KB: 8192, 16KB: 16384, 32KB: 32768, 64KB: 65536
> unsigned char buf[16384];
> size_t n, i;
> size_t total = 0;
> size_t bad = 0;
>
> while ((n = fread(buf, 1, sizeof(buf), f)) > 0) {
> for (i = 0; i < n; i++) {
> total++;
> if (!map[buf[i]]) bad++;
> if (((double)bad / total) * 100.0 > probability) {
> fclose(f);
> return 0; // not text
> }
> }
> }
>
> fclose(f);
>
> if (total == 0) return 0; // empty file or total binary...
>
> // final probability check
> return (((double)bad / total) * 100.0 <= probability) ? 1 : 0;
> }
>
[toc] | [prev] | [next] | [standalone]
| From | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| Date | 2025-12-17 03:19 +0000 |
| Message-ID | <10ht7gr$37hrt$1@dont-email.me> |
| In reply to | #395831 |
On Tue, 16 Dec 2025 17:24:49 +0100, Bonita Montero wrote: > How long have you been dealing with that ? 10 days ? > I finished the AVX-512 version in C++ in two hours. As long as I want to. -- :wq Mike Sanders
[toc] | [prev] | [next] | [standalone]
| From | Bonita Montero <Bonita.Montero@gmail.com> |
|---|---|
| Date | 2025-12-17 07:57 +0100 |
| Message-ID | <10htk7g$3aaea$1@raubtier-asyl.eternal-september.org> |
| In reply to | #395833 |
Am 17.12.2025 um 04:19 schrieb Michael Sanders:
> On Tue, 16 Dec 2025 17:24:49 +0100, Bonita Montero wrote:
>
>> How long have you been dealing with that ? 10 days ?
>> I finished the AVX-512 version in C++ in two hours.
> As long as I want to.
>
Here, about 11 times faster than your solution:
bool asciiAllowedAvx( string_view buf, double prop )
{
char const
*pBegin = buf.data(),
*pEnd = pBegin + buf.size();
size_t
uBegin = (size_t)pBegin,
uEnd = (size_t)pEnd,
head = uBegin & 63,
tail = uEnd & 63;
__m512i
*p64Begin = (__m512i *)(uBegin - head),
*p64End = (__m512i *)((uEnd + 63) & -64);
span<__m512i const> range( p64Begin, p64End );
__m512i const
prnt = _mm512_set1_epi8( (char)0x20 ),
cr = _mm512_set1_epi8( (char)'\r' ),
lf = _mm512_set1_epi8( (char)'\n' ),
tab = _mm512_set1_epi8( (char)'\t' );
uint64_t mask = (uint64_t)-1ll << head;
ptrdiff_t nFits = 0;
auto cur = range.begin(), end = range.end();
auto doChunk = [&]()
{
__m512i chunk = _mm512_loadu_epi8( (void *)to_address( cur ) );
uint64_t
prntMask = _mm512_cmpge_epu8_mask( chunk, prnt ),
crMask = _mm512_cmpeq_epi8_mask( chunk, cr ),
lfMask = _mm512_cmpeq_epi8_mask( chunk, lf ),
tabMask = _mm512_cmpeq_epi8_mask( chunk, tab );
nFits += popcount( (prntMask | crMask | lfMask | tabMask) & mask );
};
for( ; cur != end - (bool)tail; ++cur, mask = -1ll )
doChunk();
if( tail )
{
mask &= ~((uint64_t)-1ll << tail);
doChunk();
}
return 1.0 - (double)nFits / (double)(ptrdiff_t)buf.size() > prop;
}
[toc] | [prev] | [next] | [standalone]
| From | Michael Sanders <porkchop@invalid.foo> |
|---|---|
| Date | 2025-12-17 19:35 +0000 |
| Message-ID | <10hv0mt$3pisc$1@dont-email.me> |
| In reply to | #395835 |
On Wed, 17 Dec 2025 07:57:42 +0100, Bonita Montero wrote:
> [...]
Sigh, here we go...
Everything in every thread I've read from you is faster, better,
& you finished in two hours & yet here you are.
*Very rich* Bontia. Thank you, but as others have pointed out:
This is a C newsgroup, not a C++ newsgroup.
static const char phrase[] = "I dont want or care about C++...";
printf("%s\n", phrase);
--
:wq
Mike Sanders
[toc] | [prev] | [next] | [standalone]
| From | Bonita Montero <Bonita.Montero@gmail.com> |
|---|---|
| Date | 2025-12-18 08:44 +0100 |
| Message-ID | <10i0baq$4pgj$1@raubtier-asyl.eternal-september.org> |
| In reply to | #395838 |
Am 17.12.2025 um 20:35 schrieb Michael Sanders:
> Sigh, here we go...
> Everything in every thread I've read from you is faster, better,
> & you finished in two hours & yet here you are.
> *Very rich* Bontia. Thank you, but as others have pointed out:
> This is a C newsgroup, not a C++ newsgroup.
That's not so important because it's about the general principle.
Writing it in C would only be slightly different.
> static const char phrase[] = "I dont want or care about C++...";
> printf("%s\n", phrase);
>
[toc] | [prev] | [next] | [standalone]
| From | bart <bc@freeuk.com> |
|---|---|
| Date | 2025-12-18 12:49 +0000 |
| Message-ID | <10i0t8p$a010$1@dont-email.me> |
| In reply to | #395839 |
On 18/12/2025 07:44, Bonita Montero wrote:
> Am 17.12.2025 um 20:35 schrieb Michael Sanders:
>> Sigh, here we go...
>> Everything in every thread I've read from you is faster, better,
>> & you finished in two hours & yet here you are.
>> *Very rich* Bontia. Thank you, but as others have pointed out:
>> This is a C newsgroup, not a C++ newsgroup.
> That's not so important because it's about the general principle.
> Writing it in C would only be slightly different.
So, why not post C versions? Or, I guess it would be extended C.
>> static const char phrase[] = "I dont want or care about C++...";
>> printf("%s\n", phrase);
>>
>
[toc] | [prev] | [next] | [standalone]
| From | Bonita Montero <Bonita.Montero@gmail.com> |
|---|---|
| Date | 2025-12-18 14:06 +0100 |
| Message-ID | <10i0u79$aa6d$1@raubtier-asyl.eternal-september.org> |
| In reply to | #395841 |
Am 18.12.2025 um 13:49 schrieb bart: > So, why not post C versions? Or, I guess it would be extended C. Becaue it's easier to write safe code in C++. F.e. I'm using a span of AVX2/AVX-512 words. While debugging I have bounds checking with that.
[toc] | [prev] | [next] | [standalone]
| From | gazelle@shell.xmission.com (Kenny McCormack) |
|---|---|
| Date | 2025-12-18 13:17 +0000 |
| Message-ID | <10i0utj$4ltf$1@news.xmission.com> |
| In reply to | #395842 |
In article <10i0u79$aa6d$1@raubtier-asyl.eternal-september.org>,
Bonita Montero <Bonita.Montero@gmail.com> wrote:
>Am 18.12.2025 um 13:49 schrieb bart:
>> So, why not post C versions? Or, I guess it would be extended C.
>Becaue it's easier to write safe code in C++.
Wouldn't it be easier still to just not post at all?
--
"If God wanted us to believe in him, he'd exist."
(Linda Smith on "10 Funniest Londoners", TimeOut, 23rd June, 2005.)
[toc] | [prev] | [next] | [standalone]
| From | Bonita Montero <Bonita.Montero@gmail.com> |
|---|---|
| Date | 2025-12-18 16:03 +0100 |
| Message-ID | <10i1525$cgr1$1@raubtier-asyl.eternal-september.org> |
| In reply to | #395843 |
Am 18.12.2025 um 14:17 schrieb Kenny McCormack: >> Becaue it's easier to write safe code in C++. > Wouldn't it be easier still to just not post at all? Some people here thought they could develop efficient code for that. I just wanted to show that this is possible a lot faster.
[toc] | [prev] | [next] | [standalone]
| From | Keith Thompson <Keith.S.Thompson+u@gmail.com> |
|---|---|
| Date | 2025-12-05 17:42 -0800 |
| Message-ID | <87fr9ofks9.fsf@example.invalid> |
| In reply to | #395686 |
Michael Sanders <porkchop@invalid.foo> writes:
> Am I close? Missing anything you'd consider to be (or not) needed?
There is no completely reliable way to do this, but you might be
able to make a reasonable guess. A binary file might happen to
contain only byte values that represent printable characters.
> <stdio.h>
>
> /*
> * Checks if a file is likely a binary by examining its content
> * for NULL bytes (0x00) or unusual control characters.
> * Returns 0 if text, 1 if binary or file open failure.
> */
Please use the term "null bytes", not "NULL bytes". NULL is a standard
macro that expands to a null pointer constant.
> int is_binary_file(const char *path) {
> FILE *f = fopen(path, "rb");
> if (!f) return 1; // cannot open file, treat as error/fail check
It seems odd to say that a file is assumed to be binary if you can't
open it. I suggest having the function return more than two distinct
values:
- File seems to be binary
- File seems to be text
- Could be either
- Something went wrong
An enum is probably a good choice.
> unsigned char buf[65536];
> size_t n, i;
>
> while ((n = fread(buf, 1, sizeof(buf), f)) > 0) {
Since you're only looking at individual characters, you might as well
read one character at a time. The stdio functions will buffer the input
for you, so there won't be much loss of performance.
> for (i = 0; i < n; i++) {
> unsigned char c = buf[i];
>
> // 1. check for the NULL byte (strong indicator of binary
> data)
"null byte", not "NULL byte".
> if (c == 0x00) {
> fclose(f);
> return 1; // IS binary
> }
>
> // 2. check for C0 control codes (0x01-0x1F), excluding known
> // text formatting characters: 0x09 (Tab), 0x0A (LF), 0x0D (CR)
> if (c < 0x20) {
> if (c != 0x09 && c != 0x0A && c != 0x0D) {
This test will detect '\0' bytes, making your first check redundant.
> fclose(f);
> return 1; // IS binary (contains unexpected control code)
> }
You're assuming an ASCII-based character set, which is very
probably a safe assumption. But I'd suggest replacing most of
the hex constants with character constants. Aside from being more
portable (realistically EBCDIC systems are the only case where it
will matter), it makes the code more readable. And things like
UTF-8 and UTF-16 make things a lot more complicated.
0x00 -> '\0'
0x20 -> ' '
0x09 -> '\t'
0x0A -> '\n'
0x0D -> '\r'
> }
> }
> }
>
> fclose(f);
fclose(f) can fail. That's not likely, but you should check.
> return 0; // NOT binary
> }
You treat an empty file as text. That's not entirely unreasonable,
but you should at least document it.
You assume that a binary file is one that contains any byte values
in the range 0..31 other than '\t', '\n', and '\r'. So a "text"
file can't contain formfeed characters (debatable), but it can
contain DEL characters and anything above 127.
For Latin-1, values from 0xa0 to 0xff are printable (0xa0 is
NO-BREAK SPACE, so that might be debatable). For UTF-8, bytes with
values 0x80 and higher can be valid, but only in certain contexts.
And so on.
Depending on how far you want to get into it, distinguishing between
text and binary files is anywhere from difficult to literally
impossible.
Take a look at the "file" command.
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
[toc] | [prev] | [next] | [standalone]
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2025-12-06 17:37 +0000 |
| Message-ID | <bRZYQ.2361$8WR2.310@fx46.iad> |
| In reply to | #395688 |
Keith Thompson <Keith.S.Thompson+u@gmail.com> writes: >Michael Sanders <porkchop@invalid.foo> writes: >> Am I close? Missing anything you'd consider to be (or not) needed? > >There is no completely reliable way to do this, but you might be >able to make a reasonable guess. A binary file might happen to >contain only byte values that represent printable characters. > >> <stdio.h> >> >> /* >> * Checks if a file is likely a binary by examining its content >> * for NULL bytes (0x00) or unusual control characters. >> * Returns 0 if text, 1 if binary or file open failure. >> */ > >Please use the term "null bytes", not "NULL bytes". NULL is a standard >macro that expands to a null pointer constant. The proper term IMO is 'NUL' byte as defined by ASCII. Some older operating systems actually stored the file type in metadata (like the unix inode). The Burroughs MCP filesystems included a file-type field in the metadata for a file; the CANDE editor would use this to determine the programming language (and the associated language formatting rules a la COBOL or FORTRAN vis-a-vis column assignments for the sequence number, program verbs, etc.
[toc] | [prev] | [next] | [standalone]
Page 2 of 6 — ← Prev page 1 [2] 3 4 5 6 Next page →
Back to top | Article view | comp.lang.c
csiph-web