Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #10706 > unrolled thread

Hardlink sub-directories and files

Back to article view | Back to comp.lang.python

Contents

  Hardlink sub-directories and files loial <jldunn2000@gmail.com> - 2011-08-02 02:32 -0700
    Re: Hardlink sub-directories and files Peter Otten <__peter__@web.de> - 2011-08-02 12:01 +0200
    Re: Hardlink sub-directories and files Thomas Jollans <t@jollybox.de> - 2011-08-02 12:13 +0200
    Re: Hardlink sub-directories and files Tim Chase <python.list@tim.thechases.com> - 2011-08-02 06:17 -0500
    Re: Hardlink sub-directories and files Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2011-08-02 21:46 -0700
    Re: Hardlink sub-directories and files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2011-08-03 12:02 +0530
      Re: Hardlink sub-directories and files Grant Edwards <invalid@invalid.invalid> - 2011-08-03 14:22 +0000
        Re: Hardlink sub-directories and files Thomas Jollans <t@jollybox.de> - 2011-08-03 17:08 +0200
        Re: Hardlink sub-directories and files Ned Deily <nad@acm.org> - 2011-08-03 12:57 -0700
    Re: Hardlink sub-directories and files Nobody <nobody@nowhere.com> - 2011-08-03 08:04 +0100
    Re: Hardlink sub-directories and files Thomas Jollans <t@jollybox.de> - 2011-08-03 11:47 +0200
    Re: Hardlink sub-directories and files Thomas Jollans <t@jollybox.de> - 2011-08-03 20:49 +0200
    Re: Hardlink sub-directories and files Thomas Jollans <t@jollybox.de> - 2011-08-03 23:54 +0200
    Re: Hardlink sub-directories and files Alexander Gattin <xrgtn@yandex.ru> - 2011-08-08 10:07 +0300

#10706 — Hardlink sub-directories and files

I am trying to hardlink all files in a directory structure using
os.link.

This works fine for files, but the directory also contains sub-
directories (which themselves contain files and sub-directories).
However I do not think it is possible to hard link directories ?

So presumably I would need to do a mkdir for each sub-directory
encountered?
Or is there an easier way to hardlink everything in a directory
structure?.

The requirement is for hard links, not symbolic links

[toc] | [next] | [standalone]

#10708

loial wrote:

> I am trying to hardlink all files in a directory structure using
> os.link.
> 
> This works fine for files, but the directory also contains sub-
> directories (which themselves contain files and sub-directories).
> However I do not think it is possible to hard link directories ?
> 
> So presumably I would need to do a mkdir for each sub-directory
> encountered?
> Or is there an easier way to hardlink everything in a directory
> structure?.
> 
> The requirement is for hard links, not symbolic links

You cannot make hardlinks for directories, that's a restriction imposed by 
the operating system.

Look for shutil.copytree() for a template of what you are trying to do; you 
might even monkepatch it

# untested
copy2 = shutil.copy2
shutil.copy2 = os.link
try:
    shutil.copytree(source, dest)
finally:
    shutil.copy2 = copy2

if you are OK with a quick-and-dirty solution.

[toc] | [prev] | [next] | [standalone]

#10709

On 02/08/11 11:32, loial wrote:
> I am trying to hardlink all files in a directory structure using
> os.link.
> 
> This works fine for files, but the directory also contains sub-
> directories (which themselves contain files and sub-directories).
> However I do not think it is possible to hard link directories ?
> 
> So presumably I would need to do a mkdir for each sub-directory
> encountered?
> Or is there an easier way to hardlink everything in a directory
> structure?.
> 
> The requirement is for hard links, not symbolic links
> 

Yes, you have to mkdir everything. However, there is an easier way:

subprocess.Popen(['cp','-Rl','target','link'])

This is assuming that you're only supporting Unices with a working cp
program, but as you're using hard links, that's quite a safe bet, I
should think.

- Thomas

[toc] | [prev] | [next] | [standalone]

#10725

On 08/02/2011 04:32 AM, loial wrote:
> I am trying to hardlink all files in a directory structure using
> os.link.
> Or is there an easier way to hardlink everything in a directory
> structure?.
>
> The requirement is for hard links, not symbolic links

While Peter & Thomas gave good answers, also be aware that 
hard-links can't cross mount-points (an OS limitation).  So if 
you have something mounted under the directory you're trying to 
hard-link-copy, attempting to create a hard-link will fail for 
things within that mount.

-tkc

[toc] | [prev] | [next] | [standalone]

#10783

On Tue, 2 Aug 2011 18:59:59 -0700, Dan Stromberg <drsalists@gmail.com>
declaimed the following in gmane.comp.python.general:

> On Tue, Aug 2, 2011 at 3:13 AM, Thomas Jollans <t@jollybox.de> wrote:
> 
> > On 02/08/11 11:32, loial wrote:
> > > I am trying to hardlink all files in a directory structure using
> > > os.link.
> > >
> > > However I do not think it is possible to hard link directories ?
> >
> 
> That is pretty true.  I've heard of hardlinked directories on Solaris, but
> that's kind of an exception to the general rule.
>
	It's been 15 years, but I think one could create them on the Amiga.
Softlinks where the tricky one to use, since they were just data files
with full path to the linked file as the data -- applications basically
had to check if it was a softlink, if so, open/read/close then open the
linked path.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#10784

On Wed, Aug 3, 2011 at 7:29 AM, Dan Stromberg <drsalists@gmail.com> wrote:
>
> On Tue, Aug 2, 2011 at 3:13 AM, Thomas Jollans <t@jollybox.de> wrote:
>>
>> On 02/08/11 11:32, loial wrote:
>> > I am trying to hardlink all files in a directory structure using
>> > os.link.
>> >
>> > However I do not think it is possible to hard link directories ?
>
> That is pretty true.  I've heard of hardlinked directories on Solaris, but
> that's kind of an exception to the general rule.
>

In APUE, Richard Stevens says only root could do this, if it is
supported by the system at all.  In a footnote, he additionally
mentions he screwed up his filesystem by creating a loop of hardlinked
directories while writing that section of the book.

I suppose it is a good thing systems don't allow that now.

-- 
regards,
kushal

[toc] | [prev] | [next] | [standalone]

#10807

On 2011-08-03, Kushal Kumaran <kushal.kumaran+python@gmail.com> wrote:
> On Wed, Aug 3, 2011 at 7:29 AM, Dan Stromberg <drsalists@gmail.com> wrote:
>>
>> On Tue, Aug 2, 2011 at 3:13 AM, Thomas Jollans <t@jollybox.de> wrote:
>>>
>>> On 02/08/11 11:32, loial wrote:
>>> > I am trying to hardlink all files in a directory structure using
>>> > os.link.
>>> >
>>> > However I do not think it is possible to hard link directories ?
>>
>> That is pretty true.?? I've heard of hardlinked directories on Solaris, but
>> that's kind of an exception to the general rule.
>
> In APUE, Richard Stevens says only root could do this,

Yep, in early versions of Solaris root could hard-link directories.
I did it once, and it's not something one did a second time.  fsck
couldn't deal with it and pretty much fell over.  IIRC, the only way
to recover was to clear several inodes manually and then let fsck
salvage things.

> if it is supported by the system at all.  In a footnote, he
> additionally mentions he screwed up his filesystem by creating a loop
> of hardlinked directories while writing that section of the book.

That sounds about right.

> I suppose it is a good thing systems don't allow that now.

It wouldn't be a problem, except there are some important places in
Unix where it is assume that filesystems are trees.  Hard linking
directories causes that assumption to be false.

-- 
Grant Edwards               grant.b.edwards        Yow! Somewhere in DOWNTOWN
                                  at               BURBANK a prostitute is
                              gmail.com            OVERCOOKING a LAMB CHOP!!

[toc] | [prev] | [next] | [standalone]

#10808

On 03/08/11 16:22, Grant Edwards wrote:
> On 2011-08-03, Kushal Kumaran <kushal.kumaran+python@gmail.com> wrote:
>> I suppose it is a good thing systems don't allow that now.
> 
> It wouldn't be a problem, except there are some important places in
> Unix where it is assume that filesystems are trees.  Hard linking
> directories causes that assumption to be false.
> 

It is generally assumed that ".." is clearly defined, i.e. that every
directory has a parent. With hard linked directories, this would break down.

[toc] | [prev] | [next] | [standalone]

#10825

In article <j1blk1$gvo$1@reader1.panix.com>,
 Grant Edwards <invalid@invalid.invalid> wrote:
> On 2011-08-03, Kushal Kumaran <kushal.kumaran+python@gmail.com> wrote:
> > On Wed, Aug 3, 2011 at 7:29 AM, Dan Stromberg <drsalists@gmail.com> wrote:
> >>
> >> On Tue, Aug 2, 2011 at 3:13 AM, Thomas Jollans <t@jollybox.de> wrote:
> >>>
> >>> On 02/08/11 11:32, loial wrote:
> >>> > I am trying to hardlink all files in a directory structure using
> >>> > os.link.
> >>> >
> >>> > However I do not think it is possible to hard link directories ?
> >>
> >> That is pretty true.?? I've heard of hardlinked directories on Solaris, but
> >> that's kind of an exception to the general rule.
> >
> > In APUE, Richard Stevens says only root could do this,
> 
> Yep, in early versions of Solaris root could hard-link directories.
> I did it once, and it's not something one did a second time.  fsck
> couldn't deal with it and pretty much fell over.  IIRC, the only way
> to recover was to clear several inodes manually and then let fsck
> salvage things.
> 
> > if it is supported by the system at all.  In a footnote, he
> > additionally mentions he screwed up his filesystem by creating a loop
> > of hardlinked directories while writing that section of the book.
> 
> That sounds about right.
> 
> > I suppose it is a good thing systems don't allow that now.
> 
> It wouldn't be a problem, except there are some important places in
> Unix where it is assume that filesystems are trees.  Hard linking
> directories causes that assumption to be false.

FWIW, Apple implemented directory hard links for HFS+ file systems in 
Mac OS X 10.5 specifically to make the Time Machine incremental backup 
system work efficiently.  The hard directory links are, of course, not 
meant to be used by the casual user.  The section "Directory Hard Links" 
in this blog entry from Amit Singh explains what restrictions are 
enforced on directory hard links to prevent cycles:

http://osxbook.com/blog/2008/11/09/hfsdebug-40-and-new-hfs-features/

-- 
 Ned Deily,
 nad@acm.org

[toc] | [prev] | [next] | [standalone]

#10786

On Tue, 02 Aug 2011 02:32:54 -0700, loial wrote:

> However I do not think it is possible to hard link directories ?

Modern Unices disallow hard links to directories, as it makes the
directory "tree" not a tree, so anything which performs a recursive walk
must explicitly check for cycles to avoid infnite recursion.

Older systems which allowed hard links to directories required root
privilege to do so.

[toc] | [prev] | [next] | [standalone]

#10795

On 03/08/11 03:59, Dan Stromberg wrote:
> 
> On Tue, Aug 2, 2011 at 3:13 AM, Thomas Jollans <t@jollybox.de
> <mailto:t@jollybox.de>> wrote:
> 
>     On 02/08/11 11:32, loial wrote:
>     > I am trying to hardlink all files in a directory structure using
>     > os.link.
>     >
>     > However I do not think it is possible to hard link directories ?
> 
> 
> That is pretty true.  I've heard of hardlinked directories on Solaris,
> but that's kind of an exception to the general rule.
>  
> 
>     > So presumably I would need to do a mkdir for each sub-directory
>     > encountered?
>     > Or is there an easier way to hardlink everything in a directory
>     > structure?.
>     >
>     > The requirement is for hard links, not symbolic links
>     >
> 
>     Yes, you have to mkdir everything. However, there is an easier way:
> 
>     subprocess.Popen(['cp','-Rl','target','link'])
> 
>     This is assuming that you're only supporting Unices with a working cp
>     program, but as you're using hard links, that's quite a safe bet, I
>     should think.
> 
> 
> A little more portable way:
> 
> $ cd from; find . -print | cpio -pdlv ../to
> cpio: ./b linked to ../to/./b
> ../to/./b
> cpio: ./a linked to ../to/./a
> ../to/./a
> cpio: ./c linked to ../to/./c
> ../to/./c
> ../to/./d
> cpio: ./d/1 linked to ../to/./d/1
> ../to/./d/1
> cpio: ./d/2 linked to ../to/./d/2
> ../to/./d/2
> cpio: ./d/3 linked to ../to/./d/3
> ../to/./d/3
> 0 blocks
> 
> However, you could do it without a shell command (IOW in pure python)
> using os.path.walk().

Is it more portable? I don't actually have cpio installed on this
system. Which implementations of cp don't implement -R and -l? Of
course, the best way is probably implementing this in Python, either
with os.path.walk, or with a monkey-patched shutil.copytree, as Peter
suggested.

Thomas

[toc] | [prev] | [next] | [standalone]

#10822

On 03/08/11 18:29, Dan Stromberg wrote:
> 
> On Wed, Aug 3, 2011 at 2:47 AM, Thomas Jollans <t@jollybox.de
> <mailto:t@jollybox.de>> wrote:
> 
>     Is it more portable? I don't actually have cpio installed on this
>     system.
> 
> 
> Interesting.  Of course, it's probably readily available to you.  What
> *ix are you seeing that doesn't include cpio by default?

Arch Linux - the base install is quite minimal. I just discovered that I
have a program called bsdcpio which is used by mkinitcpio (and possibly
other system scripts); no need for the GNU cpio. Curious.

>  
> 
>     Which implementations of cp don't implement -R and -l?
> 
> 
> Probably most of them, except GNU and newer BSD.

Okay. While GNU libc manuals usually document how portable functions are
in detail, that's not true for the GNU coreutils manuals.

Thomas

[toc] | [prev] | [next] | [standalone]

#10828

On 03/08/11 23:25, Dan Stromberg wrote:
> 
> On Wed, Aug 3, 2011 at 11:49 AM, Thomas Jollans <t@jollybox.de
> <mailto:t@jollybox.de>> wrote:
> 
> 
>     > Interesting.  Of course, it's probably readily available to you.  What
>     > *ix are you seeing that doesn't include cpio by default?
> 
>     Arch Linux - the base install is quite minimal. I just discovered that I
>     have a program called bsdcpio which is used by mkinitcpio (and possibly
>     other system scripts); no need for the GNU cpio. Curious.
> 
> 
> I guess that makes some sense.  If you want to really strip down an
> install, removing cpio is a good candidate since it duplicates what's in
> tar, and tar is more popular - especially for interactive use.
> 
>>     Which implementations of cp don't implement -R and -l?
>>
>>
>> Probably most of them, except GNU and newer BSD.
> 
>     Okay. While GNU libc manuals usually document how portable functions are
>     in detail, that's not true for the GNU coreutils manuals.
> 
> 
> I don't think cpio is in GNU coreutils.  Also, I think GNU cpio is a
> reimplementation, not the original.

Indeed. But cp is in the coreutils, and that was what we were talking about.

As for GNU cpio, that's simply what /usr/bin/cpio, if present, is
expected to be on a GNU/Linux system.

> 
> cpio's been around since PWB/Unix, which sits between 6th Edition Unix
> and 7th Edition.  It should be in just about everything, unless a
> vendor/distributor got pretty zealous about cutting duplicate utilities.
> 

[toc] | [prev] | [next] | [standalone]

#11027

Hello,

On Tue, Aug 02, 2011 at 02:32:54AM -0700, loial
wrote:
> This works fine for files, but the directory
> also contains sub- directories (which themselves
> contain files and sub-directories).  However I
> do not think it is possible to hard link
> directories ?

On some Unices it is, as I heard. But in general,
it's not always possible to even hardlink to an
ordinary file:
1) across filesystem boundary
2) on GRSEC system, if you don't own the original
   file

-- 
WBR,
xrgtn

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web