Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #201239 > unrolled thread

any program that search for same files?

Back to article view | Back to linux.debian.user

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  any program that search for same files? Long Wind <longwind2@yahoo.com> - 2018-10-15 00:10 +0200
    Re: any program that search for same files? The Wanderer <wanderer@fastmail.fm> - 2018-10-15 00:20 +0200
      Re: any program that search for same files? Håkon Alstadheim <hakon@alstadheim.priv.no> - 2018-10-15 00:40 +0200
        Re: any program that search for same files? Long Wind <longwind2@yahoo.com> - 2018-10-15 01:00 +0200
          Re: any program that search for same files? The Wanderer <wanderer@fastmail.fm> - 2018-10-15 01:40 +0200
            Re: any program that search for same files? Long Wind <longwind2@yahoo.com> - 2018-10-15 02:20 +0200
        Re: any program that search for same files? Long Wind <longwind2@yahoo.com> - 2018-10-15 01:10 +0200
    Re: any program that search for same files? Erik Christiansen <dvalin@internode.on.net> - 2018-10-15 03:20 +0200
    Re: any program that search for same files? Andrew McGlashan <andrew.mcglashan@affinityvision.com.au> - 2018-10-15 17:20 +0200
      Re: any program that search for same files? Long Wind <longwind2@yahoo.com> - 2018-10-16 10:00 +0200
    Re: any program that search for same files? David Christensen <dpchrist@holgerdanske.com> - 2018-10-17 06:20 +0200
      Re: any program that search for same files? Long Wind <longwind2@yahoo.com> - 2018-10-17 13:20 +0200

#201239 — any program that search for same files?

[Multipart message — attachments visible in raw view] — view raw

given two directories, the program can print files that are in both directories

to make it easy, if file name and size are same, then they are same

i've to admit my memory is poor, if good, who need such program?

i'm about to write it in java, it can be completed in a few hours
but i think there might be simple solution
 

[toc] | [next] | [standalone]

#201240

[Multipart message — attachments visible in raw view] — view raw

On 2018-10-14 at 18:06, Long Wind wrote:

> given two directories, the program can print files that are in both
> directories
> 
> to make it easy, if file name and size are same, then they are same
> 
> i've to admit my memory is poor, if good, who need such program?
> 
> i'm about to write it in java, it can be completed in a few hours but
> i think there might be simple solution

Sounds like a perfect case for rdfind.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#201242

Den 15. okt. 2018 00:19, skrev The Wanderer:
> On 2018-10-14 at 18:06, Long Wind wrote:
>
>> given two directories, the program can print files that are in both
>> directories
>>
>> to make it easy, if file name and size are same, then they are same
>>
>> i've to admit my memory is poor, if good, who need such program?
>>
>> i'm about to write it in java, it can be completed in a few hours but
>> i think there might be simple solution
> Sounds like a perfect case for rdfind.
>
fdupes or jdupes fit the bill.

[toc] | [prev] | [next] | [standalone]

#201243

[Multipart message — attachments visible in raw view] — view raw

 
according to rdfind manual:

EXAMPLES       Search for duplicate files in home directory and a backup directory:
              rdfind ~ /mnt/backup

i run "rdfind /mnt/money/doc /mnt/play"
to my alarm, it delete files:
Now scanning "/mnt/money/doc", found 925 files.
Now scanning "/mnt/play", found 30527 files.
Now have 31452 files in total.
Removed 0 files due to nonunique device and inode.
Now removing files with zero size from list...removed 1 files
Total size is 30245034596 bytes or 28 Gib
Now sorting on size:removed 18727 files due to unique sizes from list.12724 files left.
Now eliminating candidates based on first bytes:^C
    On Monday, October 15, 2018, 6:39:40 AM GMT+8, Håkon Alstadheim <hakon@alstadheim.priv.no> wrote:  
 
 

Den 15. okt. 2018 00:19, skrev The Wanderer:
> On 2018-10-14 at 18:06, Long Wind wrote:
>
>> given two directories, the program can print files that are in both
>> directories
>>
>> to make it easy, if file name and size are same, then they are same
>>
>> i've to admit my memory is poor, if good, who need such program?
>>
>> i'm about to write it in java, it can be completed in a few hours but
>> i think there might be simple solution
> Sounds like a perfect case for rdfind.
>
fdupes or jdupes fit the bill.

  

[toc] | [prev] | [next] | [standalone]

#201245

[Multipart message — attachments visible in raw view] — view raw

On 2018-10-14 at 18:56, Long Wind wrote:

> according to rdfind manual:
> 
> EXAMPLES       Search for duplicate files in home directory and a backup directory:
>               rdfind ~ /mnt/backup
> 
> i run "rdfind /mnt/money/doc /mnt/play"
> to my alarm, it delete files:

Where do you get the idea that it was deleting files from?

> Now scanning "/mnt/money/doc", found 925 files.
> Now scanning "/mnt/play", found 30527 files.
> Now have 31452 files in total.
> Removed 0 files due to nonunique device and inode.
> Now removing files with zero size from list...removed 1 files

This means that it found a file with zero size, so it dropped that file
from the list. It doesn't mean that it deleted the file from the disk.

> Total size is 30245034596 bytes or 28 Gib
> Now sorting on size:removed 18727 files due to unique sizes from list.12724 files left.

This means that it dropped files from the list because there's no other
file in the list with the exact same file size. It doesn't mean that it
deleted any files from the disk.

> Now eliminating candidates based on first bytes:^C

This means that it's dropping files from the list because their first
few bytes are different. It doesn't mean that it's deleting files from
the disk.


I have used rdfind myself. It does not delete files unless you pass an
option telling it to.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#201247

[Multipart message — attachments visible in raw view] — view raw

Sorry! i've misunderstood rdfind messages

Thank The Wanderer for your detailed explaination!
 

    On Monday, October 15, 2018 7:39 AM, The Wanderer <wanderer@fastmail.fm> wrote:
 

 On 2018-10-14 at 18:56, Long Wind wrote:

> according to rdfind manual:
> 
> EXAMPLES      Search for duplicate files in home directory and a backup directory:
>              rdfind ~ /mnt/backup
> 
> i run "rdfind /mnt/money/doc /mnt/play"
> to my alarm, it delete files:

Where do you get the idea that it was deleting files from?

> Now scanning "/mnt/money/doc", found 925 files.
> Now scanning "/mnt/play", found 30527 files.
> Now have 31452 files in total.
> Removed 0 files due to nonunique device and inode.
> Now removing files with zero size from list...removed 1 files

This means that it found a file with zero size, so it dropped that file
from the list. It doesn't mean that it deleted the file from the disk.

> Total size is 30245034596 bytes or 28 Gib
> Now sorting on size:removed 18727 files due to unique sizes from list.12724 files left.

This means that it dropped files from the list because there's no other
file in the list with the exact same file size. It doesn't mean that it
deleted any files from the disk.

> Now eliminating candidates based on first bytes:^C

This means that it's dropping files from the list because their first
few bytes are different. It doesn't mean that it's deleting files from
the disk.


I have used rdfind myself. It does not delete files unless you pass an
option telling it to.

-- 
  The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.        -- George Bernard Shaw


   

[toc] | [prev] | [next] | [standalone]

#201244

[Multipart message — attachments visible in raw view] — view raw

Thank Håkon Alstadheim!I've just installed fdupes.
 

    On Monday, October 15, 2018 6:39 AM, Håkon Alstadheim <hakon@alstadheim.priv.no> wrote:
 

 

Den 15. okt. 2018 00:19, skrev The Wanderer:
> On 2018-10-14 at 18:06, Long Wind wrote:
>
>> given two directories, the program can print files that are in both
>> directories
>>
>> to make it easy, if file name and size are same, then they are same
>>
>> i've to admit my memory is poor, if good, who need such program?
>>
>> i'm about to write it in java, it can be completed in a few hours but
>> i think there might be simple solution
> Sounds like a perfect case for rdfind.
>
fdupes or jdupes fit the bill.



   

[toc] | [prev] | [next] | [standalone]

#201248

On 14.10.18 22:06, Long Wind wrote:
> given two directories, the program can print files that are in both directories
> 
> to make it easy, if file name and size are same, then they are same
> 
> i've to admit my memory is poor, if good, who need such program?
> 
> i'm about to write it in java, it can be completed in a few hours
> but i think there might be simple solution
>  

What I use with full satisfaction is just "diff -qr dir1 dir2", though
that does actually diff the files. So it suits well for double checking
a backup to flash stick. N.B. My paranoia began after one stick showed
corrupted bytes _without_ any change in file size. I.e. size is no guide.

My favourite was once the dircmp script, from AT&T, back in 1988. Google
shows others have sought it since, but I've not seen it on Linux.

Erik

[toc] | [prev] | [next] | [standalone]

#201265

Hi,

On 15/10/18 09:06, Long Wind wrote:
> given two directories, the program can print files that are in both
> directories
> 
> to make it easy, if file name and size are same, then they are same
> 
> i've to admit my memory is poor, if good, who need such program?
> 
> i'm about to write it in java, it can be completed in a few hours
> but i think there might be simple solution

diff -s

Cheers
A

[toc] | [prev] | [next] | [standalone]

#201290

[Multipart message — attachments visible in raw view] — view raw

it seems all those packages compare contents of filesbut my definition of same file is same name and sizeand should run much fast
and they seem to search all directories for same filesand this is different from my requirement
my ideal program should take two directories as argumentand print files that are in both directoriesi don't care same files within one directory
my ideal program can also take one directory as argumentand print same file in it
if more than two directories are given, 
then first directory is searched, all files that are also in other directories are printed
it seems i have to write it in javais there users' interest for such program?

 

    On Monday, October 15, 2018 11:15 PM, Andrew McGlashan <andrew.mcglashan@affinityvision.com.au> wrote:
 

 Hi,

On 15/10/18 09:06, Long Wind wrote:
> given two directories, the program can print files that are in both
> directories
> 
> to make it easy, if file name and size are same, then they are same
> 
> i've to admit my memory is poor, if good, who need such program?
> 
> i'm about to write it in java, it can be completed in a few hours
> but i think there might be simple solution

diff -s

Cheers
A



   

[toc] | [prev] | [next] | [standalone]

#201348

On 10/14/18 3:06 PM, Long Wind wrote:
> given two directories, the program can print files that are in both 
> directories
> 
> to make it easy, if file name and size are same, then they are same
> 
> i've to admit my memory is poor, if good, who need such program?
> 
> i'm about to write it in java, it can be completed in a few hours but
> i think there might be simple solution


On 10/14/18 3:19 PM, The Wanderer wrote:
> Sounds like a perfect case for rdfind.


On 10/14/18 3:39 PM, Håkon Alstadheim wrote:
> fdupes or jdupes fit the bill.


On 10/14/18 6:11 PM, Erik Christiansen wrote:
> What I use with full satisfaction is just "diff -qr dir1 dir2"...


On 10/15/18 8:14 AM, Andrew McGlashan wrote:
> diff -s


On 10/16/18 12:53 AM, Long Wind wrote:
> it seems all those packages compare contents of files but my 
> definition of same file is same name and size and should run much 
> fast
> 
> and they seem to search all directories for same files and this is 
> different from my requirement
> 
> my ideal program should take two directories as argument and print 
> files that are in both directories
> 
> i don't care same files within one directory
> 
> my ideal program can also take one directory as argument and print 
> same file in it if more than two directories are given, then first 
> directory is searched, all files that are also in other directories 
> are printed
> 
> it seems i have to write it in java
> 
> is there users' interest for such program?


It sounds like you want a script to meet your specific needs.


It is my belief that writing a general-purpose file system metadata
search utility is a non-trivial task.  Take a look at find(1) to get an
idea of the scope of the most obvious related utility:

     https://www.gnu.org/software/findutils/


Here is a Perl 5 script I wrote years ago to compare the modification 
times of same-named files in two directory trees:

     https://sourceforge.net/projects/dirdiff/

It could be modified to meet your requirements.


I suggest that you start at the small end -- writing scripts to meet 
specific needs:

1.  Pick a scripting language.

2.  Define the requirements of the script.

3.  Design, construct, test, debug, and document the script.

4.  Integrate the script into your system administration processes.


Bonuses include:

1.  Use a version control system.

2.  Use an issue tracking system.


While you could use Java, understand that a Java runtime environment 
must be installed for Java apps to work.  sh, bash, perl, and/or python 
are supported OOTB by many OS distributions, which makes it easier to 
run your scripts inside fresh install, rescue, live, and other 
restricted environments.


David

[toc] | [prev] | [next] | [standalone]

#201367

[Multipart message — attachments visible in raw view] — view raw

Thank David for your encouragement!
i have little experience with scripti prefer to use java
i have read reference guide for debian packagingi'm afraid i can't become package developer
maybe the only user of my program is meanyway i write a java method today 
it search all files in directory d and store them in Vector v1;

 public findsame(File d)
   {String a[],s; File f,p; int i; Vector v1=new Vector();     Vector v2=new Vector(); v2.addElement(d);
    while(true)
     {if(v2.size()==0) break; 
      f=(File)v2.lastElement(); v2.remove(v2.size()-1);
      a=f.list(); for(i=0;i<a.length;i++) 
       {s=f.getAbsolutePath()+File.separatorChar+a[i];
        p=new File(s); if(p.isFile()) v1.addElement(s);
        if(p.isDirectory())    v2.addElement(p);
       }
     }  
   }


 

    On Wednesday, October 17, 2018 12:12 PM, David Christensen <dpchrist@holgerdanske.com> wrote:
 

 On 10/14/18 3:06 PM, Long Wind wrote:
> given two directories, the program can print files that are in both 
> directories
> 
> to make it easy, if file name and size are same, then they are same
> 
> i've to admit my memory is poor, if good, who need such program?
> 
> i'm about to write it in java, it can be completed in a few hours but
> i think there might be simple solution


On 10/14/18 3:19 PM, The Wanderer wrote:
> Sounds like a perfect case for rdfind.


On 10/14/18 3:39 PM, Håkon Alstadheim wrote:
> fdupes or jdupes fit the bill.


On 10/14/18 6:11 PM, Erik Christiansen wrote:
> What I use with full satisfaction is just "diff -qr dir1 dir2"...


On 10/15/18 8:14 AM, Andrew McGlashan wrote:
> diff -s


On 10/16/18 12:53 AM, Long Wind wrote:
> it seems all those packages compare contents of files but my 
> definition of same file is same name and size and should run much 
> fast
> 
> and they seem to search all directories for same files and this is 
> different from my requirement
> 
> my ideal program should take two directories as argument and print 
> files that are in both directories
> 
> i don't care same files within one directory
> 
> my ideal program can also take one directory as argument and print 
> same file in it if more than two directories are given, then first 
> directory is searched, all files that are also in other directories 
> are printed
> 
> it seems i have to write it in java
> 
> is there users' interest for such program?


It sounds like you want a script to meet your specific needs.


It is my belief that writing a general-purpose file system metadata
search utility is a non-trivial task.  Take a look at find(1) to get an
idea of the scope of the most obvious related utility:

    https://www.gnu.org/software/findutils/


Here is a Perl 5 script I wrote years ago to compare the modification 
times of same-named files in two directory trees:

    https://sourceforge.net/projects/dirdiff/

It could be modified to meet your requirements.


I suggest that you start at the small end -- writing scripts to meet 
specific needs:

1.  Pick a scripting language.

2.  Define the requirements of the script.

3.  Design, construct, test, debug, and document the script.

4.  Integrate the script into your system administration processes.


Bonuses include:

1.  Use a version control system.

2.  Use an issue tracking system.


While you could use Java, understand that a Java runtime environment 
must be installed for Java apps to work.  sh, bash, perl, and/or python 
are supported OOTB by many OS distributions, which makes it easier to 
run your scripts inside fresh install, rescue, live, and other 
restricted environments.


David



   

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web