Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.shell > #14052

Re: Comparing Binary Files

Show all headers | View raw

>>>>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>>>> On 04.10.2016 20:02, Ivan Shmakov wrote:
>>>>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>>>> On 03.10.2016 06:52, lawrencedo99@gmail.com wrote:

 >>>> However, it seems to use a lot of memory, even when no differences
 >>>> are found...

 >>> Why?  You are creating just to more processes and the shell
 >>> establishes two pipes for the diff.  I would not expect "a lot"
 >>> memory demand for that.

 >> Two reasons I can readily think of:

 >> * xxd(1) can easily turn a 4.5 GB DVD+R image into a 20+ GB hexdump;

 > And where is the demand of "lot of memory" in that process?

	In the 'diff' part of the pipeline.

	But if the real question is "why the longest common subsequence
	algorithm, as implemented by diff(1), requires more memory to
	compare larger inputs", then I'm afraid that being unfamiliar
	with both the algorithm and (the insides of) the implementation,
	I cannot answer that.

	(Note the "maxresident" values in the transcript below.) 

$ time diff -u \
      -- <(seq -f %07.f 0 99999) <(seq -f %07.f 0 99999 | sed -e 12345d) 
--- /dev/fd/63  2016-10-07 12:27:54.594595945 +0000
+++ /dev/fd/62  2016-10-07 12:27:54.598595754 +0000
@@ -12342,7 +12342,6 @@
 0012341
 0012342
 0012343
-0012344
 0012345
 0012346
 0012347
Command exited with non-zero status 1
0.00user 0.00system 0:00.25elapsed 3%CPU (0avgtext+0avgdata 2536maxresident)k
0inputs+0outputs (0major+693minor)pagefaults 0swaps
$ time diff -u -- \
      <(seq -f %07.f 0 999999) <(seq -f %07.f 0 999999 | sed -e 123456d) 
--- /dev/fd/63  2016-10-07 12:24:35.740077089 +0000
+++ /dev/fd/62  2016-10-07 12:24:35.744076899 +0000
@@ -123453,7 +123453,6 @@
 0123452
 0123453
 0123454
-0123455
 0123456
 0123457
 0123458
Command exited with non-zero status 1
0.02user 0.03system 0:02.84elapsed 2%CPU (0avgtext+0avgdata 16600maxresident)k
0inputs+0outputs (0major+4208minor)pagefaults 0swaps
$ command time diff -u -- \
      <(seq -f %07.f 0 9999999) <(seq -f %07.f 0 9999999 | sed -e 1234567d) 
--- /dev/fd/63  2016-10-07 12:23:28.371276047 +0000
+++ /dev/fd/62  2016-10-07 12:23:28.375275857 +0000
@@ -1234564,7 +1234564,6 @@
 1234563
 1234564
 1234565
-1234566
 1234567
 1234568
 1234569
Command exited with non-zero status 1
0.20user 0.34system 0:30.85elapsed 1%CPU (0avgtext+0avgdata 157224maxresident)k
250inputs+0outputs (5major+39359minor)pagefaults 0swaps
$ 

[...]

-- 
FSF associate member #7257  58F8 0F47 53F5 2EB2 F6A5  8916 3013 B6A0 230E 334A

Back to comp.unix.shell | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Comparing Binary Files lawrencedo99@gmail.com - 2016-10-02 21:52 -0700
  Re: Comparing Binary Files Janis Papanagnou <janis_papanagnou@hotmail.com> - 2016-10-03 07:25 +0200
    Re: Comparing Binary Files lawrencedo99@gmail.com - 2016-10-02 22:48 -0700
      Re: Comparing Binary Files Janis Papanagnou <janis_papanagnou@hotmail.com> - 2016-10-03 09:12 +0200
        Re: Comparing Binary Files lawrencedo99@gmail.com - 2016-10-05 00:18 -0700
    Re: Comparing Binary Files Ivan Shmakov <ivan@siamics.net> - 2016-10-04 18:02 +0000
      Re: Comparing Binary Files lawrencedo99@gmail.com - 2016-10-04 11:32 -0700
      Re: Comparing Binary Files Janis Papanagnou <janis_papanagnou@hotmail.com> - 2016-10-05 10:01 +0200
        Re: Comparing Binary Files Ivan Shmakov <ivan@siamics.net> - 2016-10-07 12:30 +0000
  Re: Comparing Binary Files Rakesh Sharma <sharma__r@hotmail.com> - 2016-10-04 03:03 -0700
  Re: Comparing Binary Files Michael Paoli <michael1cat@yahoo.com> - 2016-10-06 01:32 -0700
    Re: Comparing Binary Files lawrencedo99@gmail.com - 2016-10-06 14:42 -0700

csiph-web