Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.internet.services.video.youtube > #388

Re: Creating new files of indeterminate sort?

Cross-posted to 2 groups.

Show all headers | View raw

On Wed, 7/30/2025 9:24 AM, Philip Herlihy wrote:
> In article <106b8gk$2rau6$1@dont-email.me>, address@is.invalid says...
>>
>> Knuttle,
>>
>>> It is amazing what you can do with the old Dos Batch file commands.
>>
> ...
>> That makes me remember this story :
>>
>> https://thedailywtf.com/articles/Classic-WTF-The-Indexer
>>
> 
> You could do even more with Unix shell-scripting - presumably still 
> availablein Linux, and something similar's available with PowerShell 
> (never learned either as it happens).
> 
> Years ago (mid- 90s) the head of my department called me into his 
> office.  Had a project for me, likely to last 6m or so.  He described it 
> - a way of grooming a live feed so that stats could be displayed on one 
> of these new-fangled "web pages".  Realisation dawned as he spoke.  I 
> asked if I could come back to him in an hour's time.  He was puzzled, 
> but agreed.
> 
> An hour later I explained that I'd assembled a series of shell-script 
> fragments that I'd been gradually accumulating over the last year into a 
> utility that would do exactly what he wanted, crucially using the 
> relational "JOIN" script command.  It would need a little tidying up in 
> terms of the exact display, but it was working.  Right now, at my desk 
> upstairs.
> 
> He just didn't get it.  He said he would "drop by" and take a look, but 
> he never did.  And a week later, one of my colleagues was given the same 
> project.  He was still trying to write C++ code six months later.  Yet 
> within an hour of being asked, I'd had a working prototype running on my 
> screen.  Didn't do me any good, actually.  That was meant to be a plum 
> project, and I was overlooked after that.  (I left.)
> 
> In another role, I'd come in one morning to find my then boss sizzling 
> over his keyboard.  He'd been given a large (!) file of data including a 
> "code" for a location in a vast network.  Unfortunately the other data 
> that was to be used had the code formatted very slightly differently - 
> the human eye could make the translation trivially, but matching it 
> electronically was impossible, it seemed.  He'd come in just after dawn 
> and had been working in a text editor manually changing the codes for 
> two hours by the time I arrived.  The progress indicator still showed 
> 0%.
> 
> "Would you like me to do that for you?" I asked.  He practically kicked 
> his chair down the hall.  I got him to explain the exact naming 
> convention, and spent ten minutes studying both files of data, and 
> writing a shell "pipeline" using "SED", the only utility capable both of 
> "regular expressions" and handling a file of arbitrary size.  I closed 
> his edit session (cheekily telling it not to bother saving changes) and 
> ran my script.  It ran for ten minutes, and fixed every single code.
> 
> Then there was the project based on an object-oriented database (which 
> didn't seem to work after a month's hard work on it) which I replaced 
> with a shell-script I'd written in two hours for a customer demo which 
> had been sprung on me at 24 hours' notice.  My then boss subsequently 
> went off into rhetorical flights about how this COULD NOT HAVE BEEN 
> ACHIEVED in just one month without using "advanced programming 
> techniques" like the latest object-oriented database.  I only told him 
> afterwards.
> 
> DOS scripting never had the power or fluency of Unix shell-scripting, 
> but although I find the "look" of PowerShell rather off-putting I think 
> it very likely that PS could do all this - it's just a shame I've never 
> had any reason to learn it.  But if you're a Linux user, check to see if 
> SED and AWK and JOIN (etc) are still available.  If so, you're sitting 
> on some real data processing power there.
> 
> --
> Phil, London
> 

AWK is what I use for text processing. I have the AWK book, from the inventors
of the language, printed in 1988. But the book is not necessary, because
of the PDF manual that comes with GAWK (which came later in time). You can
make system calls from there, which gives you some elements of scripting,
with perhaps a bit easier syntax.

This one has Windows line endings, and is version 3.1.6 .

   https://gnuwin32.sourceforge.net/packages/gawk.htm

If I switch to Bash shell, the Linux there has Gawk 4, but you
need a two line stanza to fix the line endings (because it has
Linux line endings). On only one occasion, I had the windows 3.1.6
version crash, while processing a 10GB text file. I took it over
to the Gawk 4 one, and the script ran without a problem.

The doc link would get you the PDF file (after you unpack
the ZIP that the site delivers).

   https://gnuwin32.sourceforge.net/downlinks/gawk-doc-zip.php

SED is more powerful, in that you can handle some pretty nasty
messes in files with it. But then the syntax is also a handful.

AWK has the limitation it's not a UTF-8 thing, it works with
ANSI text most directly. You have to be more careful on
modern Windows, with your Notepad and the saving modes,
to not make disaster-materials for your AWK run.

   Paul

Back to comp.internet.services.video.youtube | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: Creating new files of indeterminate sort? "J. P. Gilliver" <G6JPG@255soft.uk> - 2025-07-28 18:18 +0100
  Re: Creating new files of indeterminate sort? "J. P. Gilliver" <G6JPG@255soft.uk> - 2025-07-28 19:10 +0100
    Re: Creating new files of indeterminate sort? "R.Wieser" <address@is.invalid> - 2025-07-28 22:18 +0200
      Re: Creating new files of indeterminate sort? "J. P. Gilliver" <G6JPG@255soft.uk> - 2025-07-29 01:37 +0100
        Re: Creating new files of indeterminate sort? "R.Wieser" <address@is.invalid> - 2025-07-29 09:10 +0200
          Re: Creating new files of indeterminate sort? "J. P. Gilliver" <G6JPG@255soft.uk> - 2025-07-29 18:46 +0100
            Re: Creating new files of indeterminate sort? knuttle <keith_nuttle@yahoo.com> - 2025-07-29 13:58 -0400
            Re: Creating new files of indeterminate sort? "R.Wieser" <address@is.invalid> - 2025-07-29 20:21 +0200
              Re: Creating new files of indeterminate sort? knuttle <keith_nuttle@yahoo.com> - 2025-07-29 15:20 -0400
                Re: Creating new files of indeterminate sort? "R.Wieser" <address@is.invalid> - 2025-07-29 21:47 +0200
                Re: Creating new files of indeterminate sort? "J. P. Gilliver" <G6JPG@255soft.uk> - 2025-07-29 22:38 +0100
                Re: Creating new files of indeterminate sort? "R.Wieser" <address@is.invalid> - 2025-07-30 08:57 +0200
                Re: Creating new files of indeterminate sort? Paul <nospam@needed.invalid> - 2025-07-29 22:02 -0400
                Re: Creating new files of indeterminate sort? "R.Wieser" <address@is.invalid> - 2025-07-30 09:54 +0200
                Re: Creating new files of indeterminate sort? Philip Herlihy <nothing@invalid.com> - 2025-07-30 14:24 +0100
                Re: Creating new files of indeterminate sort? Paul <nospam@needed.invalid> - 2025-07-30 09:55 -0400
                Re: Creating new files of indeterminate sort? knuttle <keith_nuttle@yahoo.com> - 2025-07-30 13:44 -0400
                Re: Creating new files of indeterminate sort? Paul <nospam@needed.invalid> - 2025-07-30 23:33 -0400
        Re: Creating new files of indeterminate sort? "R.Wieser" <address@is.invalid> - 2025-07-29 10:03 +0200
          Re: Creating new files of indeterminate sort? "R.Wieser" <address@is.invalid> - 2025-07-29 10:15 +0200
          Re: Creating new files of indeterminate sort? "J. P. Gilliver" <G6JPG@255soft.uk> - 2025-07-29 22:51 +0100
            Re: Creating new files of indeterminate sort? "R.Wieser" <address@is.invalid> - 2025-07-30 09:42 +0200
              Re: Creating new files of indeterminate sort? Paul <nospam@needed.invalid> - 2025-07-30 10:02 -0400
                Re: Creating new files of indeterminate sort? "R.Wieser" <address@is.invalid> - 2025-07-30 17:04 +0200
              Re: Creating new files of indeterminate sort? "J. P. Gilliver" <G6JPG@255soft.uk> - 2025-07-30 21:49 +0100
            Re: Creating new files of indeterminate sort? Mark Lloyd <not.email@all.invalid> - 2025-07-30 18:15 +0000
              Re: Creating new files of indeterminate sort? Paul <nospam@needed.invalid> - 2025-07-30 23:37 -0400

csiph-web