Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.shell > #795 > unrolled thread

variable-length strings

Back to article view | Back to comp.unix.shell

Contents

  variable-length strings Uno <Uno@example.invalid> - 2011-06-01 03:36 -0600
    Re: variable-length strings ccc31807 <cartercc@gmail.com> - 2011-06-01 07:13 -0700
      Re: variable-length strings Uno <Uno@example.invalid> - 2011-06-02 10:54 -0600
    Re: variable-length strings "George Mpouras" <nospam.gravitalsun@hotmail.com.nospam> - 2011-06-01 17:01 +0300
    Re: variable-length strings "Peter J. Holzer" <hjp-usenet2@hjp.at> - 2011-06-02 15:33 +0200
    Re: variable-length strings "Peter J. Holzer" <hjp-usenet2@hjp.at> - 2011-06-02 15:44 +0200
      Re: variable-length strings Willem <willem@toad.stack.nl> - 2011-06-02 14:19 +0000
        cmp (was: variable-length strings) "Peter J. Holzer" <hjp-usenet2@hjp.at> - 2011-06-02 18:15 +0200
        Re: variable-length strings Uno <Uno@example.invalid> - 2011-06-03 23:21 -0600
          Re: variable-length strings "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2011-06-04 02:43 -0400
      Re: variable-length strings bonomi@host122.r-bonomi.com (Robert Bonomi) - 2011-06-03 11:08 -0500
        Re: variable-length strings Uno <Uno@example.invalid> - 2011-06-04 00:03 -0600
          Re: variable-length strings pacman@kosh.dhis.org (Alan Curry) - 2011-06-04 20:18 +0000
            Re: variable-length strings Uno <Uno@example.invalid> - 2011-06-07 23:00 -0600
        cmp (was: variable-length strings) "Peter J. Holzer" <hjp-usenet2@hjp.at> - 2011-06-04 10:29 +0200
          Re: cmp Uno <Uno@example.invalid> - 2011-06-07 22:38 -0600
            Re: cmp pacman@kosh.dhis.org (Alan Curry) - 2011-06-08 05:22 +0000
              Re: cmp Uno <Uno@example.invalid> - 2011-06-09 16:10 -0600
      Re: variable-length strings Uno <Uno@example.invalid> - 2011-06-03 22:58 -0600
        bdiff (was: variable-length strings) "Peter J. Holzer" <hjp-usenet2@hjp.at> - 2011-06-04 10:52 +0200
          Re: bdiff Uno <Uno@example.invalid> - 2011-06-08 21:59 -0600
            Re: bdiff Ian Collins <ian-news@hotmail.com> - 2011-06-09 16:08 +1200
              Re: bdiff Uno <Uno@example.invalid> - 2011-06-08 22:20 -0600
                Re: bdiff Uno <Uno@example.invalid> - 2011-06-09 02:57 -0600
                  Re: bdiff Janis Papanagnou <janis_papanagnou@hotmail.com> - 2011-06-09 11:10 +0200
                  Re: bdiff bonomi@host122.r-bonomi.com (Robert Bonomi) - 2011-06-19 06:10 -0500
            Re: bdiff Eric Sosman <esosman@ieee-dot-org.invalid> - 2011-06-09 07:13 -0400
              Re: bdiff Uno <Uno@example.invalid> - 2011-06-09 13:30 -0600
      Re: variable-length strings Uno <Uno@example.invalid> - 2011-06-03 23:11 -0600
        Re: variable-length strings "Peter J. Holzer" <hjp-usenet2@hjp.at> - 2011-06-04 10:55 +0200

Page 1 of 2  [1] 2  Next page →

#795 — variable-length strings

Hello from the land of enchantment,

I begin a crosspost herewith that I hope to make relevant to the 
newsgroups listed. It will take some time to get to there from here, but 
the ambition is to make it relatively lean with the topicality of each 
ng addressed and respected.  I am OP; if that's a problem for you, I'd 
like you to move on graciously.

It begins with perl, which I am currently reading.  So that's gonna be 
q1). There will be more questions in the same format, and I would like 
to make you aware of that as you consider an editorial choice.  OP 
suggests editorial choices that derive from the questions, that is, are 
trimmed with respect to q.

$ pwd
/media/KINGSTON
$ perl marni3.pl
death
$ cat marni3.pl
#!/usr/bin/perl -w
use Net::FTP;
my $domain = 'www.merrillpjensen.com';
my $username = 'u61210220';
my $password = '';
my $file = 'jac1.avi';


$ftp = Net::FTP->new($domain)    or die "Can't connect: $@\n";
$ftp->login($username, $password)       or die "Couldn't login\n";

$ftp->cwd('/wsb6326330301/resources/') or die "death $@\n";
$ftp->put($file) or die "put failed ", $ftp->message;

$

q1)  Why did I die?

I had a very successful day.  I could show it, except that I'm unable to 
communicate well with the internet.  This is usually the case when I try 
to extend my ubuntu capability.  I have something that hangs that 
wouldn't under a more firmly-established identity.

q2) [I imagine that I have a distinct question for each listed ng]  In 
testing, I've noticed a difference in file sizes between what I upload 
with ftp and what I download.  The files are 60 megs, conceived by 
apple, and I can't show what I'm saying until I get ftp working.

There's no way I can look at these files with a hex editor and my eyes. 
  Doesn't unix have a built-in way to compare them?

q3)

a)  Is there some newsgroup out there that considers web design, and not 
the kind where you have to give them money for their opinion.  If I paid 
$40 for usenet advice, it might be worse than what a hooker on central 
might do; it's given by people who want to make $40 on usenet.

alt.web.css.embedded.object.rus ?

q4)  and then the good people of c.l.c.  It looks like your tone has 
come a long way.  Down the road I'd like to consider how to pass 
information which is perl input to fortran output using the iso c binding.

Cheers,
-- 
Uno

[toc] | [next] | [standalone]

#798

On Jun 1, 5:36 am, Uno <U...@example.invalid> wrote:
> q1)  Why did I die?

The simplest place to start is with your command line ftp program. If
you can connect and transfer files with your command line ftp, then IN
THEORY you should be able to put the same series of commands in a
batch file (i.e., a Perl script) and run them. If you can do that,
then you have all the functionality you neet./


> q2) [I imagine that I have a distinct question for each listed ng]  In
> There's no way I can look at these files with a hex editor and my eyes.
>   Doesn't unix have a built-in way to compare them?

On Unix/Linux, use diff. If you are comfortable with vi/vim, it also
has a diff capability.

> q3)
>
> a)  Is there some newsgroup out there that considers web design, and not
> the kind where you have to give them money for their opinion.  If I paid
> $40 for usenet advice, it might be worse than what a hooker on central
> might do; it's given by people who want to make $40 on usenet.

Sorry, but you need to rephrase your question, as this does not make
sense to me.

> q4)  and then the good people of c.l.c.  It looks like your tone has
> come a long way.  Down the road I'd like to consider how to pass
> information which is perl input to fortran output using the iso c binding.

Data is data. You can use Perl to manipulate and format data. In fact,
PERL (in the opinion of some, anyway) stands for Practical Extraction
and Reporting Language.

CC.

[toc] | [prev] | [next] | [standalone]

#819

On 06/01/2011 08:13 AM, ccc31807 wrote:
> On Jun 1, 5:36 am, Uno<U...@example.invalid>  wrote:
>> q1)  Why did I die?
>
> The simplest place to start is with your command line ftp program. If
> you can connect and transfer files with your command line ftp, then IN
> THEORY you should be able to put the same series of commands in a
> batch file (i.e., a Perl script) and run them. If you can do that,
> then you have all the functionality you neet./

I needed a fresh day and fresh eyes, and t set Debug=> 2 to see that the 
wsb "web site builder" number was wrong.  That's a separate directory 
that my host uses for when they build the website.  I want to adios this 
thing, so I called them, and they have you change an administrative 
setting so that a browser will be directed to the index.html in the root 
menu as opposed to in the wsb folder.

So now this /wsb.../ thing is a relic for me to play with and destroy. 
Why didn't this work:

[sorry can only paste as quotation:]

> $ perl marni5.pl
> Net::FTP>>> Net::FTP(2.77)
> Net::FTP>>>   Exporter(5.63)
> Net::FTP>>>   Net::Cmd(2.29)
> Net::FTP>>>   IO::Socket::INET(1.31)
> Net::FTP>>>     IO::Socket(1.31)
> Net::FTP>>>       IO::Handle(1.28)
> Net::FTP=GLOB(0xa07a7d8)<<< 220 FTP Server ready.
> Net::FTP=GLOB(0xa07a7d8)>>> USER u61210220
> Net::FTP=GLOB(0xa07a7d8)<<< 331 Password required for u61210220
> Net::FTP=GLOB(0xa07a7d8)>>> PASS ....
> Net::FTP=GLOB(0xa07a7d8)<<< 230 User u61210220 logged in
> Net::FTP=GLOB(0xa07a7d8)>>> RMD /wsb6121022001/
> Net::FTP=GLOB(0xa07a7d8)<<< 550 /wsb6121022001/: Directory not empty
> Net::FTP=GLOB(0xa07a7d8)>>> PORT 192,168,0,64,142,193
> Net::FTP=GLOB(0xa07a7d8)<<< 200 PORT command successful
> Net::FTP=GLOB(0xa07a7d8)>>> NLST /wsb6121022001/
> Net::FTP=GLOB(0xa07a7d8)<<< 150 Opening ASCII mode data connection for file list
> Net::FTP=GLOB(0xa07a7d8)<<< 226 Transfer complete
> Net::FTP=GLOB(0xa07a7d8)>>> DELE /wsb6121022001/.
> Net::FTP=GLOB(0xa07a7d8)<<< 550 /wsb6121022001/.: Is a directory
> Net::FTP=GLOB(0xa07a7d8)>>> RMD /wsb6121022001/.
> Net::FTP=GLOB(0xa07a7d8)<<< 550 /wsb6121022001/.: Directory not empty
> Net::FTP=GLOB(0xa07a7d8)>>> PORT 192,168,0,64,186,213
> Net::FTP=GLOB(0xa07a7d8)<<< 200 PORT command successful
> Net::FTP=GLOB(0xa07a7d8)>>> NLST /wsb6121022001/.
> Net::FTP=GLOB(0xa07a7d8)<<< 150 Opening ASCII mode data connection for file list
> Net::FTP=GLOB(0xa07a7d8)<<< 226 Transfer complete

...

it just keeps on telling me Directory not empty
> ^C
> $ cat marni5.pl
> #!/usr/bin/perl -w
> use Net::FTP;
> my $domain = 'www.merrillpjensen.com';
> my $username = 'u61210220';
> my $password = '';
>
> $ftp = Net::FTP->new($domain, Debug => 2)    or die "Can't connect: $@\n";
> $ftp->login($username, $password)       or die "Couldn't login\n";
>
> my $recurse = 1;
> $ftp->rmdir('/wsb6121022001/', $recurse);
> #$ftp->cwd('/wsb6121022001/') or die "death $@\n";
>
> my @file = $ftp->ls();
>
> foreach $file (@file){
> print "$file\n";
> }
> $

Cpan says that RECURSE has to be true, and I thought a good guess for 
"true" in a syntax that is a child of C would be one.

>
>> q2) [I imagine that I have a distinct question for each listed ng]  In
>> There's no way I can look at these files with a hex editor and my eyes.
>>    Doesn't unix have a built-in way to compare them?
>
> On Unix/Linux, use diff. If you are comfortable with vi/vim, it also
> has a diff capability.

Thx, ccc, I'll do that.  Here's what's vexing me right now.

http://www.merrillpjensen.com/images/jac1.bmp

The only difference between these files, is that the smaller one went on 
a round on the internet.  It was sent up like this:

> #!/usr/bin/perl -w
> use strict;
> use Net::FTP;
> my $domain = 'www.merrillpjensen.com';
> my $username = 'u61210220';
> my $password = '';
> my $file = 'jac1.bmp';
> my $dir = '/images/';
>
> my $ftp = Net::FTP->new($domain, Debug =>2)    or die "Can't connect: $@\n";
> $ftp->login($username, $password)       or die "Couldn't login\n";
> $ftp->mkdir($dir) or die "death: $@\n";
> $ftp->cwd($dir) or die "death: $@\n";
> $ftp->put($file) or die "put failed ", $ftp->message;

And downloaded using google chrome.  Is this typical, because if my 
experience is, then I don't really see how any large file gets 
transferred faithfully.

Gotta go pull down some legal tender.  Adios.
-- 
Uno

[toc] | [prev] | [next] | [standalone]

#799

try to change the connection line to

$ftp = Net::FTP->new($domain, Debug=>1, Passive=>1) 

[toc] | [prev] | [next] | [standalone]

#809

["Followup-To:" header set to comp.lang.perl.misc.]
On 2011-06-01 09:36, Uno <Uno@example.invalid> wrote:
> $ perl marni3.pl
> death
> $ cat marni3.pl
> #!/usr/bin/perl -w
> use Net::FTP;
> my $domain = 'www.merrillpjensen.com';
> my $username = 'u61210220';
> my $password = '';
> my $file = 'jac1.avi';
>
>
> $ftp = Net::FTP->new($domain)    or die "Can't connect: $@\n";
> $ftp->login($username, $password)       or die "Couldn't login\n";
>
> $ftp->cwd('/wsb6326330301/resources/') or die "death $@\n";
> $ftp->put($file) or die "put failed ", $ftp->message;
>
> $
>
> q1)  Why did I die?

Simple answer: Because you couldn't change the current working directory
to '/wsb6326330301/resources/'. 

As to why that didn't work, the Net::FTP module would tell you if you
asked it. Read the synopsis of perldoc Net::FTP for a few examples on
printing proper error messages.

	hp

[toc] | [prev] | [next] | [standalone]

#811

["Followup-To:" header set to comp.unix.shell.]
On 2011-06-01 09:36, Uno <Uno@example.invalid> wrote:
> q2) [I imagine that I have a distinct question for each listed ng]  In 
> testing, I've noticed a difference in file sizes between what I upload 
> with ftp and what I download.  The files are 60 megs, conceived by 
> apple, and I can't show what I'm saying until I get ftp working.
>
> There's no way I can look at these files with a hex editor and my eyes. 
>   Doesn't unix have a built-in way to compare them?

There's cmp, but it only shows you where the first difference is.

There's diff, but it is intended for text files.

You could convert both files with od or hd and diff those. But that
doesn't work well.

I think Emacs has a binary-diff mode, but I'm from the vi fraction.

So, back in the mid 1990's I wrote bdiff (binary diff):
http://www.hjp.at/programs/bdiff/

I guess I should dust it off, write a man-page and try to get it added
to some linux distribution. But I very rarely need it myself.

	hp

[toc] | [prev] | [next] | [standalone]

#812

Peter J. Holzer wrote:
) ["Followup-To:" header set to comp.unix.shell.]
) On 2011-06-01 09:36, Uno <Uno@example.invalid> wrote:
)> There's no way I can look at these files with a hex editor and my eyes. 
)>   Doesn't unix have a built-in way to compare them?
)
) There's cmp, but it only shows you where the first difference is.

cmp -l shows all differences, as does cmp -x


SaSW, Willem
-- 
Disclaimer: I am in no way responsible for any of the statements
            made in the above text. For all I know I might be
            drugged or something..
            No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT

[toc] | [prev] | [next] | [standalone]

#817 — cmp (was: variable-length strings)

On 2011-06-02 14:19, Willem <willem@toad.stack.nl> wrote:
> Peter J. Holzer wrote:
> ) ["Followup-To:" header set to comp.unix.shell.]
> ) On 2011-06-01 09:36, Uno <Uno@example.invalid> wrote:
> )> There's no way I can look at these files with a hex editor and my eyes. 
> )>   Doesn't unix have a built-in way to compare them?
> )
> ) There's cmp, but it only shows you where the first difference is.
>
> cmp -l shows all differences, as does cmp -x

I'm not sure whether the -l option existed in 1994. In any case it
doesn't resynchronize, so if your files differ by a single inserted or
deleted byte, cmp -l will report all the bytes after that change as
different.

Neither GNU cmp nor the POSIX standard know a "-x" option.

	hp

[toc] | [prev] | [next] | [standalone]

#835

On 06/02/2011 08:19 AM, Willem wrote:
> Peter J. Holzer wrote:
> ) ["Followup-To:" header set to comp.unix.shell.]
> ) On 2011-06-01 09:36, Uno<Uno@example.invalid>  wrote:
> )>  There's no way I can look at these files with a hex editor and my eyes.
> )>    Doesn't unix have a built-in way to compare them?
> )
> ) There's cmp, but it only shows you where the first difference is.
>
> cmp -l shows all differences, as does cmp -x
>

The first output is with cmp -l.  By the time it's in the 200,000's, 
almost every one differs

  289458   5 102
  289459 333 200
  289460 342 133
  289461  15 254
  289462 153 120
  289463 161 151
  289464 373  37
  289465 304 122
  289466 331 103
  289467  43 130
  289468   1 321
  289469 372 264
  289470 203 317
  289471 374 107
  289472 351 255
  289473 376 200
^Ccmp before.avi after2.avi
before.avi after2.avi differ: byte 5, line 1
$

So, if byte 5 were simply omitted, the rest of the file wouldn't match 
except coincidentally.

Since corrupted media is so often the outcome I got, I wonder whether 
the transfer protocol is UDP for this type of download.
-- 
Uno

[toc] | [prev] | [next] | [standalone]

#837

On Sat, 04 Jun 2011 01:21:16 -0400, Uno <Uno@example.invalid> wrote:

> Since corrupted media is so often the outcome I got, I wonder whether
> the transfer protocol is UDP for this type of download.

Set the transfer mode to binary, instead of ascii.

Regards, Dave Hodgins

-- 
Change nomail.afraid.org to ody.ca to reply by email.
(nomail.afraid.org has been set up specifically for
use in usenet. Feel free to use it yourself.)

[toc] | [prev] | [next] | [standalone]

#827

In article <slrniuf4u1.dhq.hjp-usenet2@hrunkner.hjp.at>,
Peter J. Holzer <hjp-usenet2@hjp.at> wrote:
>["Followup-To:" header set to comp.unix.shell.]
>On 2011-06-01 09:36, Uno <Uno@example.invalid> wrote:
>> q2) [I imagine that I have a distinct question for each listed ng]  In 
>> testing, I've noticed a difference in file sizes between what I upload 
>> with ftp and what I download.  The files are 60 megs, conceived by 
>> apple, and I can't show what I'm saying until I get ftp working.
>>
>> There's no way I can look at these files with a hex editor and my eyes. 
>>   Doesn't unix have a built-in way to compare them?
>
>There's cmp, but it only shows you where the first difference is.

Spoken by someone who hasn't/can't read the manpage.   see the -l option.

[toc] | [prev] | [next] | [standalone]

#836

On 06/03/2011 10:08 AM, Robert Bonomi wrote:

> [snip] see the -l option.

$ cmp -l -n 50  before.avi after2.avi
  5 204 150
  6 251 364
  7 112  66
  8   0   2
17 314 166
18  22  42
33  65 126
37   0 351
38 300  73
39  22  10
46   0  11
47   1   0
49 170 301
50   0  15
$

It looks like it gets back on track for a bit.  Can you speculate what 
happened to this download?
-- 
Uno

[toc] | [prev] | [next] | [standalone]

#859

In article <94u05hF6b1U1@mid.individual.net>, Uno  <Uno@example.invalid> wrote:
>On 06/03/2011 10:08 AM, Robert Bonomi wrote:
>
>> [snip] see the -l option.
>
>$ cmp -l -n 50  before.avi after2.avi
>  5 204 150
>  6 251 364
>  7 112  66
>  8   0   2

In an AVI file, the first 4 bytes are "RIFF" and the next 4 bytes are the
size of the payload, which should be the rest of the file. It's a
little-endian format. Combining the bytes:

dc<<<'8i 204 251 112 0 Ai 256*+256*+256*+p'
4893060
dc<<<'8i 150 364 66 2 Ai 256*+256*+256*+p'
37155944

the first file has a declared payload size of 4893060, so it should have been
a file of about 4893068 bytes. Sometimes they have a junk segment on the end
that isn't counted in the RIFF header but those are small. The other file
should be about 37155952 bytes long, more than 7 times the size of the first
file.

The differences themselves don't look like any kind of simple corruption. The
insertion or deletion of extra '\r' (13) bytes as in FTP ASCII mode wouldn't
have done this. It's not just a couple of bits flipped either.

After the RIFF size, the next 8 bytes would normally be "AVI ", signifying
the type of the root node, and "LIST" indicating that the next node is a
list. (In the RIFF format, there are LIST nodes which are like directories
and non-LIST nodes are like files...)

And the cmp shows no differences there. As expected, all AVI files have the
same stuff there.

>17 314 166
>18  22  42

After "LIST" will be the size of the list node. I expect this to be the
"hdrl" (header list) node, which contains the movie metadata (frame rate,
codecs, etc.) This is another 4-byte little endian number. I think we can
assume that bytes 19 and 20 (the upper half of the hdrl size) are 0, since
the metadata easily fits in less than 64K.

The first file has hdrl size=4812, the second has hdrl size=8822.

dc<<<'8i 314 22 0 0 Ai 256*+256*+256*+p'
4812
dc<<<'8i 166 42 0 0 Ai 256*+256*+256*+p'
8822

The next 4 bytes (21 through 24 in the cmp output) would be "hdrl". And they
match. After that would come the name of the hdrl node's first child. That's
probably going to be "avih", explaining why bytes 25 through 28 match. Bytes
29 through 32 would be the size of the avih node, which is always going to be
56 bytes, so they match too.

Starting at byte 33 we'll get the main AVI header. Bytes 33-36 are the field
dwMicroSecPerFrame. This time it's not reasonable to guess that all the bytes
not shown by cmp were 0.

>33  65 126

These bytes (in hex: 0x35 and 0x56) could reasonably be the low parts of
0x8235 and 0x8256, representing frame rates of:

dc<<<'1000000 16i8235 2k/p'
30.00
dc<<<'1000000 16i8256 2k/p'
29.97

30fps and 29.97fps, both of which are commonly occurring frame rates.

>37   0 351
>38 300  73
>39  22  10

Bytes 37-40 are the field dwMaxBytesPerSec. Assuming the high byte is 0, the
2 files have values of 1228800 and 539625. I don't know if this field is
useful, since it seems to be common for creators to put a 0 here and for
players to ignore it.

Bytes 41-44 are the field dwPaddingGranularity. All 0, I guess.

>46   0  11
>47   1   0

Bytes 45-48 are dwFlags. The first file has AVIF_WASCAPTUREFILE (0x10000) and
the second file has AVIF_ISINTERLEAVED|AVIF_TRUSTCKTYPE (0x900). These look
reasonable (although I can't figure out what AVIF_WASCAPTUREFILE means even
after reading Microsoft's explanation of it).

Bytes 49-52 are dwTotalFrames. We only have the low 2 bytes here:

>49 170 301
>50   0  15
>$

But it's plausible that the upper bytes were 0 anyway. In that case, the
first file has 120 frames and the second file has 3521 frames. In the first
case, 120 frames matches up very well with a size of 4893060 bytes, a rate of
1228800 bytes per second, and 30 frames per second. 4893060/1228800*30=119.45

The second one doesn't match up so easily. It contains audio (you can tell
that from the AVIF_ISINTERLEAVED flag) which counts toward dwMaxBytesPerSec
but not dwTotalFrames. The bytes per second of the video stream should be 
37155944 bytes * 29.97 frames/sec / 3521 frames = 316263 bytes/sec. Its
declared rate is 539625 bytes/sec. If the audio takes up the missing space,
it would be about 42% of the file. That could be correct if the audio is
uncompressed.

Or maybe dwMaxBytesPerSec just wasn't useful for this purpose. It's not
called "average bytes per sec" after all.

>
>It looks like it gets back on track for a bit.  Can you speculate what 
>happened to this download?

It gets back on track because the AVI header has a lot of unused space after
the interesting fields, so there's a big spread of 0's to match up. They
probably have some more mismatches around byte 4840, where the first file's
hdrl chunk ends.

Since they both look like valid AVI files, with many differing fields but
none that look invalid, I speculate that they're 2 different movies, and even
though one or both of them may contain errors, neither one is a corrupted
copy of the other. The differences between them are just too non-random to be
any kind of accidental corruption.

If you want to compare AVIs, better options than cmp are:

1. play them! side by side if necessary.

2. /usr/bin/file file1.avi file2.avi

3. (like /usr/bin/file but better)
alias mi='mplayer -noconsolecontrols -identify -frames 0 -vo null -ao null'
mi file1.avi 2>&1 | grep '^ID' > dump1
mi file2.avi 2>&1 | grep '^ID' > dump2
diff -u dump1 dump2

4. xdelta delta file1.avi file2.avi deltafile ; ls -l deltafile
xdelta records a complete set of instructions for constructing file2 from
file1. If the delta file is very small, that means there's not much
difference between them, and one could be a corrupted copy of the other. If
the delta file is nearly as big as file2, they files just didn't have much in
common.

-- 
Alan Curry

[toc] | [prev] | [next] | [standalone]

#872

On 06/04/2011 02:18 PM, Alan Curry wrote:
> In article<94u05hF6b1U1@mid.individual.net>, Uno<Uno@example.invalid>  wrote:
>> On 06/03/2011 10:08 AM, Robert Bonomi wrote:
>>
>>> [snip] see the -l option.
>>
>> $ cmp -l -n 50  before.avi after2.avi
>>   5 204 150
>>   6 251 364
>>   7 112  66
>>   8   0   2
>
> In an AVI file, the first 4 bytes are "RIFF" and the next 4 bytes are the
> size of the payload, which should be the rest of the file. It's a
> little-endian format. Combining the bytes:
>
> dc<<<'8i 204 251 112 0 Ai 256*+256*+256*+p'
> 4893060
> dc<<<'8i 150 364 66 2 Ai 256*+256*+256*+p'
> 37155944
>
> the first file has a declared payload size of 4893060, so it should have been
> a file of about 4893068 bytes. Sometimes they have a junk segment on the end
> that isn't counted in the RIFF header but those are small. The other file
> should be about 37155952 bytes long, more than 7 times the size of the first
> file.

$ cmp -l -n 50  before.avi after2.avi
  5 204 150
  6 251 364
  7 112  66
  8   0   2
17 314 166
18  22  42
33  65 126
37   0 351
38 300  73
39  22  10
46   0  11
47   1   0
49 170 301
50   0  15
$ ls -l

-rw-r--r--  1 dan dan  37155580 2011-05-30 16:54 after2.avi
-rw-r--r--  1 dan dan   4892982 2011-06-01 17:13 before.avi

Thanks for your generous response, Alan, and I don't want you to be mad 
because I have to reveal that your calculations were correct, and I was 
basically comparing two files that never had anything to do with each 
other except that they ended in .avi and together lay in a pool of my 
failures.

Downthread I have a better isolation of this problem.  It's one where 
I'm back and forth to windows, and I hope you'll understand that this is 
precisely the type of f* up that results from this OS schizophrenia.
>
> The differences themselves don't look like any kind of simple corruption. The
> insertion or deletion of extra '\r' (13) bytes as in FTP ASCII mode wouldn't
> have done this. It's not just a couple of bits flipped either.
>
> After the RIFF size, the next 8 bytes would normally be "AVI ", signifying
> the type of the root node, and "LIST" indicating that the next node is a
> list. (In the RIFF format, there are LIST nodes which are like directories
> and non-LIST nodes are like files...)
>
> And the cmp shows no differences there. As expected, all AVI files have the
> same stuff there.
>
>> 17 314 166
>> 18  22  42
>
> After "LIST" will be the size of the list node. I expect this to be the
> "hdrl" (header list) node, which contains the movie metadata (frame rate,
> codecs, etc.) This is another 4-byte little endian number. I think we can
> assume that bytes 19 and 20 (the upper half of the hdrl size) are 0, since
> the metadata easily fits in less than 64K.
>
> The first file has hdrl size=4812, the second has hdrl size=8822.
>
> dc<<<'8i 314 22 0 0 Ai 256*+256*+256*+p'
> 4812
> dc<<<'8i 166 42 0 0 Ai 256*+256*+256*+p'
> 8822

Holy crap, I can do it, too:

$ dc<<<'8i 314 22 0 0 Ai 256*+256*+256*+p'
4812
$ dc<<<'8i 166 42 0 0 Ai 256*+256*+256*+p'
8822
$ man dc
$

>
> The next 4 bytes (21 through 24 in the cmp output) would be "hdrl". And they
> match. After that would come the name of the hdrl node's first child. That's
> probably going to be "avih", explaining why bytes 25 through 28 match. Bytes
> 29 through 32 would be the size of the avih node, which is always going to be
> 56 bytes, so they match too.
>
> Starting at byte 33 we'll get the main AVI header. Bytes 33-36 are the field
> dwMicroSecPerFrame. This time it's not reasonable to guess that all the bytes
> not shown by cmp were 0.
>
>> 33  65 126
>
> These bytes (in hex: 0x35 and 0x56) could reasonably be the low parts of
> 0x8235 and 0x8256, representing frame rates of:
>
> dc<<<'1000000 16i8235 2k/p'
> 30.00
> dc<<<'1000000 16i8256 2k/p'
> 29.97
>
> 30fps and 29.97fps, both of which are commonly occurring frame rates.
>
>> 37   0 351
>> 38 300  73
>> 39  22  10
>
> Bytes 37-40 are the field dwMaxBytesPerSec. Assuming the high byte is 0, the
> 2 files have values of 1228800 and 539625. I don't know if this field is
> useful, since it seems to be common for creators to put a 0 here and for
> players to ignore it.
>
> Bytes 41-44 are the field dwPaddingGranularity. All 0, I guess.
>
>> 46   0  11
>> 47   1   0
>
> Bytes 45-48 are dwFlags. The first file has AVIF_WASCAPTUREFILE (0x10000) and
> the second file has AVIF_ISINTERLEAVED|AVIF_TRUSTCKTYPE (0x900). These look
> reasonable (although I can't figure out what AVIF_WASCAPTUREFILE means even
> after reading Microsoft's explanation of it).
>
> Bytes 49-52 are dwTotalFrames. We only have the low 2 bytes here:
>
>> 49 170 301
>> 50   0  15
>> $
>
> But it's plausible that the upper bytes were 0 anyway. In that case, the
> first file has 120 frames and the second file has 3521 frames. In the first
> case, 120 frames matches up very well with a size of 4893060 bytes, a rate of
> 1228800 bytes per second, and 30 frames per second. 4893060/1228800*30=119.45
>
> The second one doesn't match up so easily. It contains audio (you can tell
> that from the AVIF_ISINTERLEAVED flag) which counts toward dwMaxBytesPerSec
> but not dwTotalFrames. The bytes per second of the video stream should be
> 37155944 bytes * 29.97 frames/sec / 3521 frames = 316263 bytes/sec. Its
> declared rate is 539625 bytes/sec. If the audio takes up the missing space,
> it would be about 42% of the file. That could be correct if the audio is
> uncompressed.
>
> Or maybe dwMaxBytesPerSec just wasn't useful for this purpose. It's not
> called "average bytes per sec" after all.
>
>>
>> It looks like it gets back on track for a bit.  Can you speculate what
>> happened to this download?
>
> It gets back on track because the AVI header has a lot of unused space after
> the interesting fields, so there's a big spread of 0's to match up. They
> probably have some more mismatches around byte 4840, where the first file's
> hdrl chunk ends.
>
> Since they both look like valid AVI files, with many differing fields but
> none that look invalid, I speculate that they're 2 different movies, and even
> though one or both of them may contain errors, neither one is a corrupted
> copy of the other. The differences between them are just too non-random to be
> any kind of accidental corruption.

Of course, you're right.
>
> If you want to compare AVIs, better options than cmp are:
>
> 1. play them! side by side if necessary.

Windows is always telling me that the files have been corrupted.
>
> 2. /usr/bin/file file1.avi file2.avi
>
> 3. (like /usr/bin/file but better)
> alias mi='mplayer -noconsolecontrols -identify -frames 0 -vo null -ao null'
> mi file1.avi 2>&1 | grep '^ID'>  dump1
> mi file2.avi 2>&1 | grep '^ID'>  dump2
> diff -u dump1 dump2
>
> 4. xdelta delta file1.avi file2.avi deltafile ; ls -l deltafile
> xdelta records a complete set of instructions for constructing file2 from
> file1. If the delta file is very small, that means there's not much
> difference between them, and one could be a corrupted copy of the other. If
> the delta file is nearly as big as file2, they files just didn't have much in
> common.
>

I'll consider these when I have opportunity.  Cheers,
-- 
Uno

[toc] | [prev] | [next] | [standalone]

#838 — cmp (was: variable-length strings)

On 2011-06-03 16:08, Robert Bonomi <bonomi@host122.r-bonomi.com> wrote:
> In article <slrniuf4u1.dhq.hjp-usenet2@hrunkner.hjp.at>,
> Peter J. Holzer <hjp-usenet2@hjp.at> wrote:
>>On 2011-06-01 09:36, Uno <Uno@example.invalid> wrote:
[binary files]
>>>   Doesn't unix have a built-in way to compare them?
>>
>>There's cmp, but it only shows you where the first difference is.
>
> Spoken by someone who hasn't/can't read the manpage.   see the -l option.

See <slrniufdoj.dhq.hjp-usenet2@hrunkner.hjp.at> (written almost 24
hours before your posting).

	hp

[toc] | [prev] | [next] | [standalone]

#871 — Re: cmp

On 06/04/2011 02:29 AM, Peter J. Holzer wrote:

> See<slrniufdoj.dhq.hjp-usenet2@hrunkner.hjp.at>  (written almost 24
> hours before your posting).

I have it better-isolated now:

[sorry, can only paste as quotation]
> $ ls -l

> -rw-r--r--  1 dan dan  19573258 2011-06-07 22:14 downloaded1.wmv
> -rw-r--r--  1 dan dan  19573712 2011-06-05 17:32 shoulder.wmv
> -rw-r--r--  1 dan dan       511 2011-06-07 22:07 upload1.pl

> shoulder.wmv downloaded1.wmv differ: byte 196502, line 1409
> $ cmp -l -n 196560 shoulder.wmv downloaded1.wmv
> 196502  15  12
> 196503  12 172
> 196504 172 271
> 196505 271 161
> 196506 161 123
> 196507 123 152
> 196508 152 225
> 196509 225 256
> 196510 256 115
> 196511 115 145
> 196513 145 102
> 196514 102 237
> 196515 237   7
> 196516   7 372
> 196517 372 301
> 196518 301 375
> 196519 375  64
> 196520  64 103
> 196521 103 206
> 196522 206  47
> 196523  47  67
> 196524  67 242
> 196525 242  50
> 196526  50  21
> 196527  21 246
> 196528 246 250
> 196529 250 154
> 196530 154   0
> 196531   0 300
> 196532 300 223
> 196533 223 332
> 196534 332 375
> 196535 375 134
> 196536 134  70
> 196537  70  55
> 196538  55 315
> 196539 315  20
> 196540  20  41
> 196541  41  40
> 196542  40  37
> 196543  37 330
> 196544 330  31
> 196545  31   3
> 196546   3 157
> 196547 157 300
> 196548 300  52
> 196549  52 346
> 196550 346 263
> 196551 263  40
> 196552  40 143
> 196554 143 222
> 196555 222 353
> 196556 353 101
> 196557 101  72
> 196558  72 245
> 196559 245  13
> 196560  13 116
> $

Can someone speculate what happened to the 196,502nd byte?
> $ cat upload1.pl
> #!/usr/bin/perl -w
> use strict;
> use Net::FTP;
> my $domain = 'www.merrillpjensen.com';
> my $username = 'u61210220';
> my $password = '';
> my $file = 'shoulder.wmv';
> my $file2 = 'downloaded1.wmv';
>
> my $ftp = Net::FTP->new($domain, Debug=>1, Passive=>1)    or die "Can't connect: $@\n";
> $ftp->login($username, $password)       or die "Couldn't login\n";
>
> $ftp->cwd('/vids/') or die "death $@\n";
> $ftp->put($file) or die "put failed ", $ftp->message;
> $ftp->get($file, $file2) or die "put failed ", $ftp->message;
>
> $
-- 
Uno

[toc] | [prev] | [next] | [standalone]

#873 — Re: cmp

In article <958cmkFs5mU1@mid.individual.net>, Uno  <Uno@example.invalid> wrote:
>
>> $ cmp -l -n 196560 shoulder.wmv downloaded1.wmv
>> 196502  15  12
>> 196503  12 172
>> 196504 172 271
>> 196505 271 161
>> 196506 161 123
>> 196507 123 152
[...]
>
>Can someone speculate what happened to the 196,502nd byte?

Octal 15 (ASCII CR) inserted before octal 12 (ASCII LF). Classical text-mode
FTP corruption. Use binary mode.

-- 
Alan Curry

[toc] | [prev] | [next] | [standalone]

#897 — Re: cmp

On 06/07/2011 11:22 PM, Alan Curry wrote:
> In article<958cmkFs5mU1@mid.individual.net>, Uno<Uno@example.invalid>  wrote:
>>
>>> $ cmp -l -n 196560 shoulder.wmv downloaded1.wmv
>>> 196502  15  12
>>> 196503  12 172
>>> 196504 172 271
>>> 196505 271 161
>>> 196506 161 123
>>> 196507 123 152
> [...]
>>
>> Can someone speculate what happened to the 196,502nd byte?
>
> Octal 15 (ASCII CR) inserted before octal 12 (ASCII LF). Classical text-mode
> FTP corruption. Use binary mode.
>

$ perl upload1.pl
Net::FTP>>> Net::FTP(2.77)
Net::FTP>>>   Exporter(5.63)
Net::FTP>>>   Net::Cmd(2.29)
Net::FTP>>>   IO::Socket::INET(1.31)
Net::FTP>>>     IO::Socket(1.31)
Net::FTP>>>       IO::Handle(1.28)
Net::FTP=GLOB(0x9a13968)<<< 220 FTP Server ready.
Net::FTP=GLOB(0x9a13968)>>> USER u61210220
Net::FTP=GLOB(0x9a13968)<<< 331 Password required for u61210220
Net::FTP=GLOB(0x9a13968)>>> PASS ....
Net::FTP=GLOB(0x9a13968)<<< 230 User u61210220 logged in
Net::FTP=GLOB(0x9a13968)>>> TYPE I
Net::FTP=GLOB(0x9a13968)<<< 200 Type set to I
Net::FTP=GLOB(0x9a13968)>>> CWD /vids/
Net::FTP=GLOB(0x9a13968)<<< 250 CWD command successful
Net::FTP=GLOB(0x9a13968)>>> ALLO 19573712
Net::FTP=GLOB(0x9a13968)<<< 202 No storage allocation necessary
Net::FTP=GLOB(0x9a13968)>>> PASV
Net::FTP=GLOB(0x9a13968)<<< 227 Entering Passive Mode 
(74,208,244,112,229,99).
Net::FTP=GLOB(0x9a13968)>>> STOR shoulder.wmv
Net::FTP=GLOB(0x9a13968)<<< 150 Opening BINARY mode data connection for 
shoulder.wmv
Net::FTP=GLOB(0x9a13968)<<< 226 Transfer complete
Net::FTP=GLOB(0x9a13968)>>> PASV
Net::FTP=GLOB(0x9a13968)<<< 227 Entering Passive Mode 
(74,208,244,112,218,171).
Net::FTP=GLOB(0x9a13968)>>> RETR shoulder.wmv
Net::FTP=GLOB(0x9a13968)<<< 150 Opening BINARY mode data connection for 
shoulder.wmv (19573712 bytes)
Net::FTP=GLOB(0x9a13968)<<< 226 Transfer complete
$ cat upload1.pl
#!/usr/bin/perl -w
use strict;
use Net::FTP;
my $domain = 'www.merrillpjensen.com';
my $username = 'u61210220';
my $password = '';
my $file = 'shoulder.wmv';
my $file2 = 'downloaded2.wmv';

my $ftp = Net::FTP->new($domain, Debug=>1, Passive=>1)    or die "Can't 
connect: $@\n";
$ftp->login($username, $password)       or die "Couldn't login\n";
$ftp->binary();
$ftp->cwd('/vids/') or die "death $@\n";
$ftp->put($file) or die "put failed ";
$ftp->get($file, $file2) or die "get failed ";
$

and, was it faithful?

$ cmp -l shoulder.wmv downloaded2.wmv

Woo-hoo.  Thx, alan.  (You'll be subjected to my piano self-study if you 
look at it.  I made the breakthrough of figuring out where middle C is 
on both clefs.)
-- 
Uno

[toc] | [prev] | [next] | [standalone]

#833

On 06/02/2011 07:44 AM, Peter J. Holzer wrote:
> ["Followup-To:" header set to comp.unix.shell.]
> On 2011-06-01 09:36, Uno<Uno@example.invalid>  wrote:
>> q2) [I imagine that I have a distinct question for each listed ng]  In
>> testing, I've noticed a difference in file sizes between what I upload
>> with ftp and what I download.  The files are 60 megs, conceived by
>> apple, and I can't show what I'm saying until I get ftp working.
>>
>> There's no way I can look at these files with a hex editor and my eyes.
>>    Doesn't unix have a built-in way to compare them?
>
> There's cmp, but it only shows you where the first difference is.
>
> There's diff, but it is intended for text files.
>
> You could convert both files with od or hd and diff those. But that
> doesn't work well.
>
> I think Emacs has a binary-diff mode, but I'm from the vi fraction.
>
> So, back in the mid 1990's I wrote bdiff (binary diff):
> http://www.hjp.at/programs/bdiff/
>
> I guess I should dust it off, write a man-page and try to get it added
> to some linux distribution. But I very rarely need it myself.
>
> 	hp
>

Thx, peter, I want to get that running, but I'm mired in the type of the 
thing that a guy gets stuck in, when it's still his first year of a 
stable *nix dual-booting system.

My current difficulty is typical for me in that I start trying to 
wrangle something with perl, and pretty soon it shifts into a problem 
with figuring out more unix.

So I downloaded your software, extracted it on a thumbdrive, compiled
it, told ubuntu to run it, and it told me I don't have permission.

$ sudo chmod 777 bdiff
$ ls -l

-rw-r--r-- 1 dan dan    12414 2011-06-02 16:29 bdiff

That was my last try with that executable.  So I tried it again.

$ pwd
/media/KINGSTON/bdiff-1.0
$ cc -Wall -Wextra bdiff.c -o bdiff2
bdiff.c: In function ‘readfile’:
bdiff.c:98: warning: comparison between signed and unsigned integer 
expressions
bdiff.c: In function ‘findmatch’:
bdiff.c:227: warning: comparison between signed and unsigned integer 
expressions
bdiff.c:233: warning: comparison between signed and unsigned integer 
expressions
bdiff.c: In function ‘main’:
bdiff.c:296: warning: comparison between signed and unsigned integer 
expressions
bdiff.c:342: warning: comparison between signed and unsigned integer 
expressions
bdiff.c:350: warning: comparison between signed and unsigned integer 
expressions
$ ./bdiff2
bash: ./bdiff2: Permission denied
$ ls -l
total 123472
-rw-r--r-- 1 dan dan 37155580 2011-05-30 16:54 after2.avi
-rw-r--r-- 1 dan dan 37155580 2011-05-30 16:13 after3.avi
-rw-r--r-- 1 dan dan  4892982 2011-06-01 18:13 after.avi
-rw-r--r-- 1 dan dan     5289 1998-12-12 12:04 ascfile-1
-rw-r--r-- 1 dan dan     5428 1998-12-12 12:04 ascfile-2
-rw-r--r-- 1 dan dan    12414 2011-06-02 16:29 bdiff
-rw-r--r-- 1 dan dan    34907 2011-06-02 16:21 bdiff-1.0.tar.gz
-rw-r--r-- 1 dan dan    12414 2011-06-03 22:45 bdiff2
-rw-r--r-- 1 dan dan    10267 2011-06-03 22:45 bdiff.c
-rw-r--r-- 1 dan dan  4892982 2011-06-01 17:13 before.avi
-rw-r--r-- 1 dan dan    38623 1998-12-12 12:05 binfile-1
-rw-r--r-- 1 dan dan    39037 1998-12-12 12:05 binfile-2
-rw-r--r-- 1 dan dan      148 1998-12-12 12:05 dotest
-rw-r--r-- 1 dan dan 37155580 2011-05-30 17:54 
hydrogenalternativefromfossils.avi
-rw-r--r-- 1 dan dan  4893068 2011-03-29 16:17 jac1.avi
-rw-r--r-- 1 dan dan      136 1998-12-12 12:04 Makefile
$ chmod 777 bdiff2
$ ./bdiff2
bash: ./bdiff2: Permission denied
$ ls -l
total 123472
-rw-r--r-- 1 dan dan 37155580 2011-05-30 16:54 after2.avi
-rw-r--r-- 1 dan dan 37155580 2011-05-30 16:13 after3.avi
-rw-r--r-- 1 dan dan  4892982 2011-06-01 18:13 after.avi
-rw-r--r-- 1 dan dan     5289 1998-12-12 12:04 ascfile-1
-rw-r--r-- 1 dan dan     5428 1998-12-12 12:04 ascfile-2
-rw-r--r-- 1 dan dan    12414 2011-06-02 16:29 bdiff
-rw-r--r-- 1 dan dan    34907 2011-06-02 16:21 bdiff-1.0.tar.gz
-rw-r--r-- 1 dan dan    12414 2011-06-03 22:45 bdiff2
-rw-r--r-- 1 dan dan    10267 2011-06-03 22:45 bdiff.c
-rw-r--r-- 1 dan dan  4892982 2011-06-01 17:13 before.avi
-rw-r--r-- 1 dan dan    38623 1998-12-12 12:05 binfile-1
-rw-r--r-- 1 dan dan    39037 1998-12-12 12:05 binfile-2
-rw-r--r-- 1 dan dan      148 1998-12-12 12:05 dotest
-rw-r--r-- 1 dan dan 37155580 2011-05-30 17:54 
hydrogenalternativefromfossils.avi
-rw-r--r-- 1 dan dan  4893068 2011-03-29 16:17 jac1.avi
-rw-r--r-- 1 dan dan      136 1998-12-12 12:04 Makefile
$

What am I missing?

Shit's on fire down here in the US SW.  Arizona has a 400,000 acre fire 
that is 0% contained.  I just wish they would keep all their Arizonaness 
to themselves.  The smoke hurts my eyes.  Tja.
-- 
Uno

[toc] | [prev] | [next] | [standalone]

#839 — bdiff (was: variable-length strings)

On 2011-06-04 04:58, Uno <Uno@example.invalid> wrote:
> On 06/02/2011 07:44 AM, Peter J. Holzer wrote:
>> So, back in the mid 1990's I wrote bdiff (binary diff):
>> http://www.hjp.at/programs/bdiff/
>>
>> I guess I should dust it off, write a man-page and try to get it added
>> to some linux distribution. But I very rarely need it myself.
[...]
> So I downloaded your software, extracted it on a thumbdrive, compiled
> it, told ubuntu to run it, and it told me I don't have permission.
>
> $ sudo chmod 777 bdiff
> $ ls -l
>
> -rw-r--r-- 1 dan dan    12414 2011-06-02 16:29 bdiff
>
> That was my last try with that executable.  So I tried it again.
>
> $ pwd
> /media/KINGSTON/bdiff-1.0
 ^^^^^^^^^^^^^^^^

What kind of file system is this and how is it mounted?

> $ cc -Wall -Wextra bdiff.c -o bdiff2
> bdiff.c: In function ‘readfile’:
> bdiff.c:98: warning: comparison between signed and unsigned integer 
> expressions

The warnings are harmless, AFAICS, and partially misleading (comparison
between size_t and -1 is well-defined as long as sizeof(size_t) >=
sizeof(int), which is pretty much guaranteed).

	hp

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.unix.shell

csiph-web