Re: Contents of file changed while application is reading it

Subject	Re: Contents of file changed while application is reading it
Newsgroups	comp.os.linux.misc
References	(11 earlier) <ttn9gg$279gm$46@dont-email.me> <2GqdnVP81dcMgZ35nZ2dnZfqnPednZ2d@earthlink.com> <ttpr01$279gm$50@dont-email.me> <dYudnfNen9mG55z5nZ2dnZfqn_ednZ2d@earthlink.com> <ttsejp$279gm$64@dont-email.me>
From	"28A.I873" <28A.I873@noabzba.net>
Organization	mitotic resell
Date	2023-03-04 01:12 -0500
Message-ID	<qPadnQSthu_CfJ_5nZ2dnZfqn_ednZ2d@earthlink.com> (permalink)

Show all headers | View raw

On 3/3/23 4:28 AM, The Natural Philosopher wrote:
> On 03/03/2023 04:39, 28A.I873 wrote:
>> On 3/2/23 4:41 AM, The Natural Philosopher wrote:
>>> On 02/03/2023 03:48, 28A.I873 wrote:
>>>> On 3/1/23 5:30 AM, The Natural Philosopher wrote:
>>>>> On 01/03/2023 06:32, 28A.I873 wrote:
>>>>>> "Concurrency" is a PAIN IN THE ASS at the programming
>>>>>>    level !
>>>>>
>>>>> Which is why things like multi-process/multi-user operating system 
>>>>> developed things like semaphores, messages, queues and database 
>>>>> engines.
>>>>
>>>>    Yep, lots of tricks (often proprietary to the particular app)
>>>>
>>>>    "Database engines" are just a more user-transparent approach
>>>>    to the semaphores/messages/queues/locks/etc. They all still
>>>>    DO it, it's just a bit "under the hood".
>>>>
>>>>    If you need to write your own DB, even if quite simple, then
>>>>    YOU have to ensure at least file-level, preferably record-level,
>>>>    concurrency protections.
>>> Yup.  TBH that is pretty much all the functionality I expected from 
>>> MYSql. It for sure is pretty crap at executing *complex* SQL.
>>>
>>> I had written a C program and debugged it and got the answer I wanted 
>>> before the SQL query it duplicated had finished...
>>
>>    SQL *can* be VERY clunky (and the syntax can start to
>>    look like Martian). I had an old DOS-era DB that
>>    implemented a fair SQL. However if you went to the
>>    lesser-documented underlying, mostly 'C', routines
>>    you could speed it up by 50 times easily for big
>>    complex search/sorts and such.
>>
>>    IMHO, a "standard" language that quickly becomes
>>    incomprehensible to all but the most dedicated
>>    hacks is a POOR standard. Yea, you can write
>>    horrible 'C' ... but you don't HAVE to (and the
>>    horrors aren't any faster/smaller either). With
>>    SQL you kinda HAVE to write horrors, nested
>>    inner/outer joins are the least of it. I see why
>>    lots moved to Java to do the hard work. STILL
>>    more pretty and understandable than SQL "paragraphs"
>>    (sometimes "novellas").
>>
> Exactly. What I wanted to do was normalise some data that was extremely 
> un-normalised. It was in fact a correlation between political eras - 
> parishes boroughs counties and so on, and UK postcodes.
> 
> Developing the syntax took several days. Then I started it running. 
> There was no logging possible on its progress.
> 
> So I took a subset of the data and ran it on that. It took ages to 
> discover the two bugs.
> 
> I fixed those and ran it again. This time it looked like - assuming 
> linear extrapolation  on the data set size - , it would run for several 
> days.
> 
> 
> So I put on an old Thinking Cap I had lying around, and worked out how 
> to do it in C,  using custom designed data structures, and logging how 
> fast it was going as well onto the screen.
> 
> A couple of hours later all the bugs seemed yo be out and it ran in 
> about 2 hours. There were a couple of odd enties due IIRC to apostrophes 
> in the data (John O' Groats)
> 
> But those were fixable manually.
> 
>>    ANYway, I tend to avoid making 'complex' databases
>>    now. More pain than gain. 99.9% of what any outfit
>>    really NEEDS a DB for can be implemented using
>>    SIMPLE methods and structure. For that MySQL or
>>    numerous others are perfectly good. Now it you're
>>    the DOD or DOE or HS or something, then, well ...
> 
> Exactly so.  What I discovered is that building a database that is fully 
> normalised is not hard, and it works really well, but one must eschew 
> complex SQL queries on performance grounds.
> 
> Conceptually use of a high level language to do a 'for each in this 
> table, select from that table' worked far better than some sort of 
> complex join.
> 
> And I also discovered the PHP is utter crap with large data sets. My PHP 
> now calls a C program to access the database and present the selected 
> data as an output stream.

   PHP *can* do almost anything. However HOW to do 'almost
   anything' can be a long long search.

> In the context of concurrency, I also ran into a weird PHP buffering 
> issue. . In that an uploaded file to a php equipped web server is 
> available as a temporary entity, which must be converted to an explicit 
> file if you want to keep it or access it. Which I duly did before 
> invoking an SQL command to 'load data file' . Which always failed to 
> work. Because PHP didn't actually write the file until it exited.
> 
> I cannot  remember what sketchy sort of kludge I eventually used., It 
> may well have been a copy file command that finally resulted in it being 
> written to the OS where the  SQL daemon could read it.

   Always a few irregularities that don't fit the paradigm.
   However if they're FEW you can manually tweak 'em.

   John O' Groats ... isn't that the bitter end of the UK
   mainland ? I've toured it on Earth ... seems like an almost
   ideal place (to ME) to build a little retirement cottage  :-)

   But UK taxes are too high and all the pols these days seem
   to be out of a Python sketch .........

   I had some HUGE old databases - some in PICK-ish MV format
   and others flat-file ... that I needed to preserve for
   posterity. 10+ years of detailed field observations in
   each. Alas it WAS possible to over-ride the official
   choice of responses, and some managed to put stuff from
   one category into another. Would up with about 300
   "anomalies". Fortunately MOST weren't TOO awful - so
   I fixed them in MySQL by hand. Do check out "DBeaver"
   interface ... simple but effective.

   Ah ... an interesting technical question about 'C' ...
   assume you're looking for the position of a given
   character in a string .......

     for(x=0;x<strlen(TheStr);x++)
     {
       if(TheStr[x]==TheChar) return(x);
     }
     return(-1);

  is a pretty common approach. HOWEVER, the QUESTION,
  is "strlen(TheStr)" calculated over and over again
  for every iteration ??? That seems SLOW, esp if
  TheStr is BIG.

  Of late I've been using :

    for(x=0;TheStr[x] != 0;x++)
    {
      if(TheStr[x]==TheChar) return(x);
    }
    return(-1);

  instead of using strlen().

  Works just as well, at least in fairly simple cases,
  but seems much faster than running strlen() over and
  over a zillion times.

  I can envision a few circumstances where TheStr gets
  seriously mutated and strlen() is probably best to
  deal with its new length - but for these straight-up
  sorts of things ....

  And no, I tend to avoid pointer solutions because they
  are not nearly so clear (and often have MANY more
  little sub-steps). The compiler turns the ordinary
  index numbers INTO pointers/pointer-math anyhow ... so
  may as well be CLEAR & OBVIOUS.

  Came across a web page the other day about 'strstr()'.
  The first advice code was about three lines long. Then
  some other hotshot said "Oh ! You should be using
  POINTERS !" and offered the equiv - about *30* lines
  long and almost incomprehensible at first glance  :-)

  Hey, I need code *I* can understand one, five, ten,
  years down the line !

Back to comp.os.linux.misc | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Contents of file changed while application is reading it Clark Smith <noaddress@nowhere.net> - 2023-02-26 18:52 +0000
  Re: Contents of file changed while application is reading it Rich <rich@example.invalid> - 2023-02-26 19:20 +0000
  Re: Contents of file changed while application is reading it Robert Heller <heller@deepsoft.com> - 2023-02-26 21:45 +0000
    Re: Contents of file changed while application is reading it "Carlos E.R." <robin_listas@es.invalid> - 2023-02-26 23:05 +0100
      Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-02-26 23:38 +0000
    Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-02-26 23:32 +0000
      Re: Contents of file changed while application is reading it "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2023-02-26 19:06 -0500
        Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-02-27 00:26 +0000
          Re: Contents of file changed while application is reading it "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2023-02-26 20:48 -0500
          Re: Contents of file changed while application is reading it Richard Kettlewell <invalid@invalid.invalid> - 2023-02-27 09:03 +0000
            Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-02-27 09:58 +0000
              Re: Contents of file changed while application is reading it "Carlos E.R." <robin_listas@es.invalid> - 2023-02-27 11:16 +0100
              Re: Contents of file changed while application is reading it Richard Kettlewell <invalid@invalid.invalid> - 2023-02-27 15:20 +0000
        Re: Contents of file changed while application is reading it "Carlos E.R." <robin_listas@es.invalid> - 2023-02-27 10:34 +0100
          Re: Contents of file changed while application is reading it Richard Kettlewell <invalid@invalid.invalid> - 2023-02-27 09:49 +0000
            Re: Contents of file changed while application is reading it "Carlos E.R." <robin_listas@es.invalid> - 2023-02-27 11:13 +0100
          Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-02-27 10:02 +0000
            Re: Contents of file changed while application is reading it "Carlos E.R." <robin_listas@es.invalid> - 2023-02-27 11:13 +0100
              Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-02-27 10:17 +0000
                Re: Contents of file changed while application is reading it "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2023-02-27 12:27 -0500
                Re: Contents of file changed while application is reading it "Carlos E.R." <robin_listas@es.invalid> - 2023-02-27 19:19 +0100
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-02-27 21:46 -0500
                Re: Contents of file changed while application is reading it "Carlos E.R." <robin_listas@es.invalid> - 2023-02-28 13:33 +0100
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-02-28 10:49 -0500
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-02-28 16:21 +0000
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-03-01 01:05 -0500
                Re: Contents of file changed while application is reading it Charlie Gibbs <cgibbs@kltpzyxm.invalid> - 2023-03-01 21:57 +0000
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-03-01 22:53 -0500
                Re: Contents of file changed while application is reading it Charlie Gibbs <cgibbs@kltpzyxm.invalid> - 2023-03-02 19:06 +0000
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-03-02 23:08 -0500
                Re: Contents of file changed while application is reading it Charlie Gibbs <cgibbs@kltpzyxm.invalid> - 2023-03-03 05:23 +0000
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-03 09:36 +0000
                Re: Contents of file changed while application is reading it "Carlos E.R." <robin_listas@es.invalid> - 2023-03-03 11:19 +0100
                Re: Contents of file changed while application is reading it Charlie Gibbs <cgibbs@kltpzyxm.invalid> - 2023-03-03 15:58 +0000
                Re: Contents of file changed while application is reading it "Carlos E. R." <robin_listas@es.invalid> - 2023-03-03 18:47 +0100
                Re: Contents of file changed while application is reading it Rich <rich@example.invalid> - 2023-03-04 16:24 +0000
                Re: Contents of file changed while application is reading it "Carlos E.R." <robin_listas@es.invalid> - 2023-03-04 18:40 +0100
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-03 18:44 +0000
                Re: Contents of file changed while application is reading it Charlie Gibbs <cgibbs@kltpzyxm.invalid> - 2023-03-04 01:25 +0000
                Re: Contents of file changed while application is reading it Rich <rich@example.invalid> - 2023-03-04 16:21 +0000
                Re: Contents of file changed while application is reading it "28B.I874" <28B.I874@noabzba.net> - 2023-03-04 23:55 -0500
                Re: Contents of file changed while application is reading it Charlie Gibbs <cgibbs@kltpzyxm.invalid> - 2023-03-05 05:35 +0000
                Re: Contents of file changed while application is reading it Allodoxaphobia <trepidation@example.net> - 2023-03-05 13:48 +0000
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-05 14:01 +0000
                Re: Contents of file changed while application is reading it Charlie Gibbs <cgibbs@kltpzyxm.invalid> - 2023-03-05 19:16 +0000
                Re: Contents of file changed while application is reading it "28B.I874" <28B.I874@noabzba.net> - 2023-03-06 00:43 -0500
                Re: Contents of file changed while application is reading it "28B.I874" <28B.I874@noabzba.net> - 2023-03-06 00:42 -0500
                Re: Contents of file changed while application is reading it "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2023-02-28 12:01 -0500
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-03-01 01:17 -0500
                Re: Contents of file changed while application is reading it Robert Riches <spamtrap42@jacob21819.net> - 2023-02-28 04:13 +0000
                Re: Contents of file changed while application is reading it "Carlos E.R." <robin_listas@es.invalid> - 2023-02-28 13:16 +0100
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-03-01 01:32 -0500
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-01 10:30 +0000
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-03-01 22:48 -0500
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-02 09:41 +0000
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-03-02 23:39 -0500
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-03 09:28 +0000
                Re: Contents of file changed while application is reading it "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2023-03-03 12:16 -0500
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-03-04 01:36 -0500
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-04 07:15 +0000
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-03-04 02:47 -0500
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-03-04 01:12 -0500
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-04 06:51 +0000
                Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-03-04 03:32 -0500
                Re: Contents of file changed while application is reading it Richard Kettlewell <invalid@invalid.invalid> - 2023-03-04 09:54 +0000
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-04 10:09 +0000
                Re: Contents of file changed while application is reading it Richard Kettlewell <invalid@invalid.invalid> - 2023-03-04 11:22 +0000
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-04 11:58 +0000
                Re: Contents of file changed while application is reading it Richard Kettlewell <invalid@invalid.invalid> - 2023-03-05 16:43 +0000
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-06 09:20 +0000
                Re: Contents of file changed while application is reading it "28B.I874" <28B.I874@noabzba.net> - 2023-03-04 20:24 -0500
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-05 13:40 +0000
                Re: Contents of file changed while application is reading it Dan Espen <dan1espen@gmail.com> - 2023-03-05 11:29 -0500
                Re: Contents of file changed while application is reading it "28B.I874" <28B.I874@noabzba.net> - 2023-03-06 01:21 -0500
                Re: Contents of file changed while application is reading it "28B.I874" <28B.I874@noabzba.net> - 2023-03-06 01:06 -0500
                Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-03-06 09:28 +0000
                Re: Contents of file changed while application is reading it "28B.I874" <28B.I874@noabzba.net> - 2023-03-06 08:54 -0500
  Re: Contents of file changed while application is reading it Dan Espen <dan1espen@gmail.com> - 2023-02-26 17:35 -0500
    Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-02-26 23:39 +0000
      Re: Contents of file changed while application is reading it "28A.I873" <28A.I873@noabzba.net> - 2023-02-26 22:42 -0500
    Re: Contents of file changed while application is reading it "Carlos E.R." <robin_listas@es.invalid> - 2023-02-27 10:45 +0100
  Re: Contents of file changed while application is reading it The Natural Philosopher <tnp@invalid.invalid> - 2023-02-26 23:31 +0000

csiph-web