Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.shell > #14009 > unrolled thread

Comparing Binary Files

Back to article view | Back to comp.unix.shell

Contents

  Comparing Binary Files lawrencedo99@gmail.com - 2016-10-02 21:52 -0700
    Re: Comparing Binary Files Janis Papanagnou <janis_papanagnou@hotmail.com> - 2016-10-03 07:25 +0200
      Re: Comparing Binary Files lawrencedo99@gmail.com - 2016-10-02 22:48 -0700
        Re: Comparing Binary Files Janis Papanagnou <janis_papanagnou@hotmail.com> - 2016-10-03 09:12 +0200
          Re: Comparing Binary Files lawrencedo99@gmail.com - 2016-10-05 00:18 -0700
      Re: Comparing Binary Files Ivan Shmakov <ivan@siamics.net> - 2016-10-04 18:02 +0000
        Re: Comparing Binary Files lawrencedo99@gmail.com - 2016-10-04 11:32 -0700
        Re: Comparing Binary Files Janis Papanagnou <janis_papanagnou@hotmail.com> - 2016-10-05 10:01 +0200
          Re: Comparing Binary Files Ivan Shmakov <ivan@siamics.net> - 2016-10-07 12:30 +0000
    Re: Comparing Binary Files Rakesh Sharma <sharma__r@hotmail.com> - 2016-10-04 03:03 -0700
    Re: Comparing Binary Files Michael Paoli <michael1cat@yahoo.com> - 2016-10-06 01:32 -0700
      Re: Comparing Binary Files lawrencedo99@gmail.com - 2016-10-06 14:42 -0700

#14009 — Comparing Binary Files

Instead of, for example

    diff /dev/dvd disc.iso

which will only say “binary files differ” and nothing more, I use

    diff -u <(xxd /dev/dvd) <(xxd disc.iso)

to compare hex dumps of the files instead.

This works great--for example, showing me that the only difference is a few extra sectors of zero padding at the end of the disc. However, it seems to use a lot of memory, even when no differences are found...

[toc] | [next] | [standalone]

#14012

On 03.10.2016 06:52, lawrencedo99@gmail.com wrote:
> Instead of, for example
> 
>     diff /dev/dvd disc.iso
> 
> which will only say “binary files differ” and nothing more, I use
> 
>     diff -u <(xxd /dev/dvd) <(xxd disc.iso)
> 
> to compare hex dumps of the files instead.
> 
> This works great--for example, showing me that the only difference is a few
> extra sectors of zero padding at the end of the disc.

That does not seem to be a satisfying solution in the general case; even
simple byte offsets by a few additional bytes at the beginning of one file
will show the whole files differing. In that case a “binary files differ”
message would be preferable. For the case of changed bytes or additional
data at the end of one file it's okay.

> However, it seems to use a lot of memory, even when no differences are found...

Why? You are creating just to more processes and the shell establishes two
pipes for the diff. I would not expect "a lot" memory demand for that.

Janis

[toc] | [prev] | [next] | [standalone]

#14014

On Monday, October 3, 2016 at 6:25:45 PM UTC+13, Janis Papanagnou wrote:
>
> On 03.10.2016 06:52, Lawrence D’Oliveiro wrote:
>
>>     diff -u <(xxd /dev/dvd) <(xxd disc.iso)
> 
> ... even simple byte offsets by a few additional bytes at the beginning of
> one file will show the whole files differing.

Won’t happen in this case.

[toc] | [prev] | [next] | [standalone]

#14015

On 03.10.2016 07:48, lawrencedo99@gmail.com wrote:
> On Monday, October 3, 2016 at 6:25:45 PM UTC+13, Janis Papanagnou wrote:
>>
>> On 03.10.2016 06:52, Lawrence D’Oliveiro wrote:
>>
>>>     diff -u <(xxd /dev/dvd) <(xxd disc.iso)
>>
>> ... even simple byte offsets by a few additional bytes at the beginning of
>> one file will show the whole files differing.
> 
> Won’t happen in this case.

Your posting sounded like a general suggestion, and could be perceived as
that by unaware readers.

Janis

[toc] | [prev] | [next] | [standalone]

#14030

On Monday, October 3, 2016 at 8:12:54 PM UTC+13, Janis Papanagnou wrote:
> Your posting sounded like a general suggestion ...

Hmm ... how would you make it general?

Seems to me you’d have to have one byte per line:

    diff -u <(xxd -c1 file1 | cut -c11,12) <(xxd -c1 file2 | cut -c11,12)

and watch out for the memory-hogging...

[toc] | [prev] | [next] | [standalone]

#14026

>>>>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>>>> On 03.10.2016 06:52, lawrencedo99@gmail.com wrote:

[...]

 >> diff -u <(xxd /dev/dvd) <(xxd disc.iso)

 >> to compare hex dumps of the files instead.

 >> This works great--for example, showing me that the only difference
 >> is a few extra sectors of zero padding at the end of the disc.

 > That does not seem to be a satisfying solution in the general case;
 > even simple byte offsets by a few additional bytes at the beginning
 > of one file will show the whole files differing.  In that case a
 > "binary files differ" message would be preferable.  For the case of
 > changed bytes or additional data at the end of one file it's okay.

	As would be $ cmp -l, I suppose.

 >> However, it seems to use a lot of memory, even when no differences
 >> are found...

 > Why?  You are creating just to more processes and the shell
 > establishes two pipes for the diff.  I would not expect "a lot"
 > memory demand for that.

	Two reasons I can readily think of:

	* xxd(1) can easily turn a 4.5 GB DVD+R image into a 20+ GB
	  hexdump;

	* upon seeing that the files are indeed "binary", diff(1) may
	  refrain from running a memory-costly LCS algorithm and resort
	  to "exit (1) on the first byte that differs; exit (0) if none"
	  behavior that costs virtually no memory at all.

-- 
FSF associate member #7257  58F8 0F47 53F5 2EB2 F6A5  8916 3013 B6A0 230E 334A

[toc] | [prev] | [next] | [standalone]

#14027

On Wednesday, October 5, 2016 at 7:03:03 AM UTC+13, Ivan Shmakov wrote:
> 	* upon seeing that the files are indeed "binary", diff(1) may
> 	  refrain from running a memory-costly LCS algorithm ...

I wonder why it needs to do that before actually seeing the first difference...

[toc] | [prev] | [next] | [standalone]

#14031

On 04.10.2016 20:02, Ivan Shmakov wrote:
>>>>>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>>>>> On 03.10.2016 06:52, lawrencedo99@gmail.com wrote:
> [...]
> 
>  >> However, it seems to use a lot of memory, even when no differences
>  >> are found...
> 
>  > Why?  You are creating just to more processes and the shell
>  > establishes two pipes for the diff.  I would not expect "a lot"
>  > memory demand for that.
> 
> 	Two reasons I can readily think of:
> 
> 	* xxd(1) can easily turn a 4.5 GB DVD+R image into a 20+ GB
> 	  hexdump;

And where is the demand of "lot of memory" in that process? - There isn't,
because you have only the constant size IPC buffers - there are no bulky
files created, nor would the diff command need to increase it's internally
used buffers, since all it sees is file handles.

(The increase in processing time would have been a different statement.)

Janis

> 
> 	* upon seeing that the files are indeed "binary", diff(1) may
> 	  refrain from running a memory-costly LCS algorithm and resort
> 	  to "exit (1) on the first byte that differs; exit (0) if none"
> 	  behavior that costs virtually no memory at all.
> 

[toc] | [prev] | [next] | [standalone]

#14052

>>>>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>>>> On 04.10.2016 20:02, Ivan Shmakov wrote:
>>>>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>>>> On 03.10.2016 06:52, lawrencedo99@gmail.com wrote:

 >>>> However, it seems to use a lot of memory, even when no differences
 >>>> are found...

 >>> Why?  You are creating just to more processes and the shell
 >>> establishes two pipes for the diff.  I would not expect "a lot"
 >>> memory demand for that.

 >> Two reasons I can readily think of:

 >> * xxd(1) can easily turn a 4.5 GB DVD+R image into a 20+ GB hexdump;

 > And where is the demand of "lot of memory" in that process?

	In the 'diff' part of the pipeline.

	But if the real question is "why the longest common subsequence
	algorithm, as implemented by diff(1), requires more memory to
	compare larger inputs", then I'm afraid that being unfamiliar
	with both the algorithm and (the insides of) the implementation,
	I cannot answer that.

	(Note the "maxresident" values in the transcript below.) 

$ time diff -u \
      -- <(seq -f %07.f 0 99999) <(seq -f %07.f 0 99999 | sed -e 12345d) 
--- /dev/fd/63  2016-10-07 12:27:54.594595945 +0000
+++ /dev/fd/62  2016-10-07 12:27:54.598595754 +0000
@@ -12342,7 +12342,6 @@
 0012341
 0012342
 0012343
-0012344
 0012345
 0012346
 0012347
Command exited with non-zero status 1
0.00user 0.00system 0:00.25elapsed 3%CPU (0avgtext+0avgdata 2536maxresident)k
0inputs+0outputs (0major+693minor)pagefaults 0swaps
$ time diff -u -- \
      <(seq -f %07.f 0 999999) <(seq -f %07.f 0 999999 | sed -e 123456d) 
--- /dev/fd/63  2016-10-07 12:24:35.740077089 +0000
+++ /dev/fd/62  2016-10-07 12:24:35.744076899 +0000
@@ -123453,7 +123453,6 @@
 0123452
 0123453
 0123454
-0123455
 0123456
 0123457
 0123458
Command exited with non-zero status 1
0.02user 0.03system 0:02.84elapsed 2%CPU (0avgtext+0avgdata 16600maxresident)k
0inputs+0outputs (0major+4208minor)pagefaults 0swaps
$ command time diff -u -- \
      <(seq -f %07.f 0 9999999) <(seq -f %07.f 0 9999999 | sed -e 1234567d) 
--- /dev/fd/63  2016-10-07 12:23:28.371276047 +0000
+++ /dev/fd/62  2016-10-07 12:23:28.375275857 +0000
@@ -1234564,7 +1234564,6 @@
 1234563
 1234564
 1234565
-1234566
 1234567
 1234568
 1234569
Command exited with non-zero status 1
0.20user 0.34system 0:30.85elapsed 1%CPU (0avgtext+0avgdata 157224maxresident)k
250inputs+0outputs (5major+39359minor)pagefaults 0swaps
$ 

[...]

-- 
FSF associate member #7257  58F8 0F47 53F5 2EB2 F6A5  8916 3013 B6A0 230E 334A

[toc] | [prev] | [next] | [standalone]

#14022

On Monday, 3 October 2016 10:22:18 UTC+5:30, lawren...@gmail.com  wrote:
> Instead of, for example
> 
>     diff /dev/dvd disc.iso
> 
> which will only say “binary files differ” and nothing more, I use
> 
>     diff -u <(xxd /dev/dvd) <(xxd disc.iso)
> 
> to compare hex dumps of the files instead.
> 
> This works great--for example, showing me that the only difference is a few extra sectors of zero padding at the end of the disc. However, it seems to use a lot of memory, even when no differences are found...


did you try the "tkdiff" tool

[toc] | [prev] | [next] | [standalone]

#14044

On Sunday, October 2, 2016 at 9:52:18 PM UTC-7, lawren...@gmail.com wrote:
> Instead of, for example
>     diff /dev/dvd disc.iso
> 
> which will only say “binary files differ” and nothing more, I use
>     diff -u <(xxd /dev/dvd) <(xxd disc.iso)
> to compare hex dumps of the files instead.
> 
> This works great--for example, showing me that the only difference is a few extra sectors of zero padding at the end of the disc. However, it seems to use a lot of memory, even when no differences are found...

Of course that's resource intensive, as diff will compare the full contents
of the hex dumps of each.

There's the highly standard Unix utility cmp(1).
A couple quick examples:
$ echo short > short
$ (cat short; 2>>/dev/null dd if=/dev/zero bs=200 count=1) > longer
$ cmp short longer
cmp: EOF on short
$ (cat short /dev/zero) | cmp - longer
cmp: EOF on longer
$ echo nope | cmp - short
- short differ: char 1, line 1
$ 

By default, cmp(1) only reads the files until it detects a difference.
And no need to convert format - e.g. to hex, nor to read beyond the
first difference detected.  So, much more efficient.

[toc] | [prev] | [next] | [standalone]

#14046

On Thursday, October 6, 2016 at 9:32:40 PM UTC+13, Michael Paoli wrote:
> ... diff will compare the full contents of the hex dumps of each.

But why should it use up memory buffering up sections that are identical?

[toc] | [prev] | [standalone]

Back to top | Article view | comp.unix.shell

csiph-web