Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #243942 > unrolled thread

Re: restructure folders

Back to article view | Back to linux.debian.user

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: restructure folders Emanuel Berg <moasenwood@zoho.eu> - 2022-01-13 05:50 +0100
    Re: restructure folders "sp007@caiway.net" <sp007@caiway.net> - 2022-01-13 06:10 +0100
      Re: restructure folders Emanuel Berg <moasenwood@zoho.eu> - 2022-01-13 06:20 +0100
      Re: restructure folders Charles Curley <charlescurley@charlescurley.com> - 2022-01-13 07:00 +0100
        Re: restructure folders "sp007@caiway.net" <sp007@caiway.net> - 2022-01-13 08:10 +0100
          Re: restructure folders "sp007@caiway.net" <sp007@caiway.net> - 2022-01-13 08:40 +0100
            Re: restructure folders "sp007@caiway.net" <sp007@caiway.net> - 2022-01-13 10:30 +0100
              Re: restructure folders Greg Wooledge <greg@wooledge.org> - 2022-01-13 13:30 +0100
                Re: restructure folders Jonathan Dowland <jon+debian-user@dow.land> - 2022-01-14 16:50 +0100
              Re: restructure folders David Wright <deblis@lionunicorn.co.uk> - 2022-01-13 17:10 +0100

#243942 — Re: restructure folders

sp007@caiway.net wrote:

> I have this foldertree tree:
>
> /dir1/dirA/dirB
> /dir2/dir*
>
> I want the foldertree to become:
>
> /dirA/DirB
> /dir*
>
> Is this possible on the commandline?
> Any idea is welcome

Shell programming?

-- 
underground experts united
https://dataswamp.org/~incal

[toc] | [next] | [standalone]

#243945

On Thu, 13 Jan 2022 05:41:49 +0100
Emanuel Berg <moasenwood@zoho.eu> wrote:

> sp007@caiway.net wrote:
> 
> > I have this foldertree tree:
> >
> > /dir1/dirA/dirB
> > /dir2/dir*
> >
> > I want the foldertree to become:
> >
> > /dirA/DirB
> > /dir*
> >
> > Is this possible on the commandline?
> > Any idea is welcome
> 
> Shell programming?
> 

I found a python script to extract debian packages, but every package
is extracted in a directory named after the packagefile, so this
structure is formed:

/abi-dumper_1.2-1_all.deb/usr/bin
/abi-dumper_1.2-1_all.deb/usr/share/doc/abi-dumper
..
..
/abigail-doc_1.8-1_all.deb/usr/share/doc
/abigail-doc_1.8-1_all.deb/usr/share/man
..
..
/zzuf_0.15-1+b1_amd64.deb/usr/bin

it is about 60.000 different parent folders

the python script (a langauge I can not read):

import argparse
import os
import glob
import ntpath
import subprocess
import stat
import time
import datetime

ELF_HEADER = b"\x7fELF"
TIME_FORMAT = "%H:%M:%S, %9A, %d-%m-%Y"

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('root', help='the root directory from where to
search for all the .deb files') parser.add_argument('dest', help='the
destination directory to which the debs will be extracted')
parser.add_argument('--del_nonelfs', default=True, help='Delete all
non-elf files after extracting a .deb') args = parser.parse_args()
root_dir = args.root dest_dir_root = args.dest
    del_nonelfs = args.del_nonelfs
    print("Extracting all debs from {} to {}".format(root_dir,
dest_dir_root)) print("Deleting non-elf files: {}".format(del_nonelfs))
    break_counter = 0
    break_limit = -1

    global_start_time = time.localtime()
    print("Extraction started on: {}".format(time.strftime(TIME_FORMAT,
    global_start_time))) full_deb_file_names = set()
    # Recursively searching from the root directory
    print("Searching for .deb files:")
    for r, d, f in os.walk(root_dir):
        # Searching the directory for all .deb files
        for full_deb_file_name in glob.glob(os.path.join(r, '*.deb')):
            full_deb_file_names.add(full_deb_file_name)
            print("deb files found:
    {}".format(len(full_deb_file_names)), end = '\r', flush=True)
    break_counter += 1 if break_limit >= 0 and break_counter >=
    break_limit: break
        if break_limit >= 0 and break_counter >= break_limit:
            break
    print("")
    total = len(full_deb_file_names)
    for i, full_deb_file_name in enumerate(full_deb_file_names):
            deb_file_name = ntpath.basename(full_deb_file_name)
            dest_dir = os.path.join(dest_dir_root, deb_file_name)
            os.mkdir(dest_dir)
            print("\r{}".format(' ' * os.get_terminal_size()[0]),
    end='\r', flush=True) print("({} / {}) Extracting {}".format(i + 1,
    total, full_deb_file_name), end='', flush=True)
    extract_deb(full_deb_file_name, deb_file_name, dest_dir) if
    del_nonelfs: print("\r{}".format(' ' * os.get_terminal_size()[0]),
    end='\r', flush=True) print("({} / {}) Deleting non-elfs
    {}".format(i + 1, total, dest_dir), end='', flush=True)
    delete_non_elfs(dest_dir) print("")
    global_end_time = time.localtime()
    print("Finished on: {}".format(time.strftime(TIME_FORMAT,
    global_end_time))) global_run_time =
    format_time_delta(global_start_time, global_end_time) print("Total
    time: {}".format(global_run_time))

def extract_deb(deb_file_path, deb_file_name, dest_dir):

    # TODO dpkg assumes the user has this installed. Maybe it would be
    better to extract manually? Wikipedia tells how this file should be
    structured: https://en.wikipedia.org/wiki/Deb_(file_format) p =
    subprocess.Popen(['dpkg', '-x', deb_file_path, dest_dir],
    cwd=dest_dir) p.wait() # Giving myself read and write permissions
    to all files and read/write/traverse permissions to directories
    (because apparently that's necessary) for root, dirs, files in
    os.walk(dest_dir): for d in dirs: full_dir_path =
    os.path.join(root, d) if not os.path.islink(full_dir_path):
    os.chmod(os.path.join(root, d), stat.S_IREAD | stat.S_IWRITE |
    stat.S_IEXEC) for f in files: full_file_path = os.path.join(root, f)
            if not os.path.islink(full_file_path):
                os.chmod(os.path.join(root, f), stat.S_IREAD |
    stat.S_IWRITE)

def delete_non_elfs(root_dir):
     for r, d, f in os.walk(root_dir):
        for file_name in f:
            full_file_name = os.path.join(r, file_name)
            # We don't want to delete something that may be somewhere
else. if not os.path.islink(full_file_name) and not
is_elf_file(full_file_name): os.remove(full_file_name)


def is_elf_file(full_file_name):
    with open(full_file_name, 'rb') as fd:
        header = fd.read(4)
        if header == ELF_HEADER:
            return True
    return False

def format_time_delta(start_time, end_time, short=False):
        start_time_datetime =
datetime.datetime.fromtimestamp(time.mktime(start_time))
end_time_datetime =
datetime.datetime.fromtimestamp(time.mktime(end_time))
time_delta_datetime = end_time_datetime - start_time_datetime seconds =
int(time_delta_datetime.seconds) days, seconds = divmod(seconds, 86400)
hours, seconds = divmod(seconds, 3600) minutes, seconds =
divmod(seconds, 60) if short:
                return "{0}:{1:02d}:{2:02d}:{3:02d}".format(days,
hours, minutes, seconds) else:
                return "{0} days, {1} hours, {2:02d} minutes and
{3:02d} seconds.".format(days, hours, minutes, seconds)

if __name__ == "__main__":
    main()

EOF

[toc] | [prev] | [next] | [standalone]

#243946

sp007@caiway.net wrote:

> the python script (a langauge I can not read)

Here is what you should improve then ...

Or use a language that you can read (and write).

-- 
underground experts united
https://dataswamp.org/~incal

[toc] | [prev] | [next] | [standalone]

#243947

On Thu, 13 Jan 2022 06:00:12 +0100
"sp007@caiway.net" <sp007@caiway.net> wrote:

> I found a python script to extract debian packages, but every package
> is extracted in a directory named after the packagefile, so this
> structure is formed:

What are you trying to do? From your limited selection of files, it
appears that your python program extracts packages as they are normally
extracted, except it places them in a directory named for the package.
Why? And what's wrong with putting files where the Debian package would
put them?

If you want to know what files a given package puts where, look into
apt-file. No need to actually extract anything.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[toc] | [prev] | [next] | [standalone]

#243950

On Wed, 12 Jan 2022 22:53:01 -0700
Charles Curley <charlescurley@charlescurley.com> wrote:

> On Thu, 13 Jan 2022 06:00:12 +0100
> "sp007@caiway.net" <sp007@caiway.net> wrote:
> 
> > I found a python script to extract debian packages, but every
> > package is extracted in a directory named after the packagefile, so
> > this structure is formed:
> 
> What are you trying to do? 

I want to extract all man pages and all docs from all packages. I do
not want to install the packages. Then make nice search interfaces with
dwww and recoll.

This is to be used in areas without internet access. I have a local
repository. 

[toc] | [prev] | [next] | [standalone]

#243952

On Thu, 13 Jan 2022 07:54:38 +0100
"sp007@caiway.net" <sp007@caiway.net> wrote:

> On Wed, 12 Jan 2022 22:53:01 -0700
> Charles Curley <charlescurley@charlescurley.com> wrote:
> 
> > On Thu, 13 Jan 2022 06:00:12 +0100
> > "sp007@caiway.net" <sp007@caiway.net> wrote:
> > 
> > > I found a python script to extract debian packages, but every
> > > package is extracted in a directory named after the packagefile,
> > > so this structure is formed:
> > 
> > What are you trying to do? 
> 
> I want to extract all man pages and all docs from all packages. I do
> not want to install the packages. Then make nice search interfaces
> with dwww and recoll.
> 
> This is to be used in areas without internet access. I have a local
> repository. 
> 
> 

I found a solution:

I made a chroot with debootstrap, minimal install
then I move all *deb from the repository inside a temp directory in the
chroot

then I can dpkg --unpack *.deb let the packages get unpacked in the
system, without installing

then I can copy the /usr/share/man etc. directories to the proper
place.


Thanks for the ideas!

[toc] | [prev] | [next] | [standalone]

#243955

> > > What are you trying to do? 
> > 
> > I want to extract all man pages and all docs from all packages. I do
> > not want to install the packages. Then make nice search interfaces
> > with dwww and recoll.
> > 
> > This is to be used in areas without internet access. I have a local
> > repository. 
> > 
> > 
> 
> I found a solution:
> 
> I made a chroot with debootstrap, minimal install
> then I move all *deb from the repository inside a temp directory in
> the chroot
> 
> then I can dpkg --unpack *.deb let the packages get unpacked in the
> system, without installing
> 
> then I can copy the /usr/share/man etc. directories to the proper
> place.
> 
> 
> Thanks for the ideas!
> 

No this is no good solution
I must install all packages and this seems an impossible task
Too much questions I have to answer

[toc] | [prev] | [next] | [standalone]

#243959

On Thu, Jan 13, 2022 at 10:26:55AM +0100, sp007@caiway.net wrote:
> > > I want to extract all man pages and all docs from all packages. I do
> > > not want to install the packages. Then make nice search interfaces
> > > with dwww and recoll.

First of all, this already exists:

https://manpages.debian.org/

Second, you need to understand that two or more packages may contain
different implementations of the same command, with the same name, and
the same named man page.  Even if you bypass the package installation and
just operate on the raw files, package A may contain the same pathname
as package B, and so one of them is going to overwrite the other.

That's why manpages.d.o has the package name in the URIs that it
generates.  E.g. if you look up the man page for "parallel", you may
get <https://manpages.debian.org/bullseye/parallel/parallel.1.en.html>
which is the page for parallel(1) from the parallel package, or you may
end up at <https://manpages.debian.org/bullseye/moreutils/parallel.1.en.html>
which is the page for parallel(1) from the moreutils package.

So, if you're reinventing manpages.d.o for fun, make sure you take that
into account.

[toc] | [prev] | [next] | [standalone]

#243987

On Thu, Jan 13, 2022 at 07:23:14AM -0500, Greg Wooledge wrote:
>First of all, this already exists:
>
>https://manpages.debian.org/

Alas OP mentioned (not in the top message) that this is for an offline
system.

However manpages.d.o mentions it is generated using this software:

     https://github.com/Debian/debiman

OP could give that a try.

Another option would be to contact the manpages.d.o maintainers and
discuss mirroring (or at least downloading) all of the generated data.
See <https://manpages.debian.org/faq.html> for who to get in touch with.

-- 
Please do not CC me for listmail.

👱🏻	Jonathan Dowland
✎	 jmtd@debian.org
🔗	https://jmtd.net

[toc] | [prev] | [next] | [standalone]

#243964

On Thu 13 Jan 2022 at 10:26:55 (+0100), sp007@caiway.net wrote:
> 
> > > > What are you trying to do? 
> > > 
> > > I want to extract all man pages and all docs from all packages. I do
> > > not want to install the packages. Then make nice search interfaces
> > > with dwww and recoll.
> > > 
> > > This is to be used in areas without internet access. I have a local
> > > repository. 

I'm not sure what you mean by "all packages". Surely you're not going
to download the entire Debian repository just to pick up the man pages
when they're already available on the web. Many in your collection
would seem destined to languish for ever (or until Debian moves on a
suite) without ever being consulted.

It would seem more efficient to process the man pages for the packages
already installed on the system, and provide your local copies through
a wrapper that notifies you of any that need providing as and when
they are demanded.

> > I found a solution:
> > 
> > I made a chroot with debootstrap, minimal install
> > then I move all *deb from the repository inside a temp directory in
> > the chroot
> > 
> > then I can dpkg --unpack *.deb let the packages get unpacked in the
> > system, without installing
> > 
> > then I can copy the /usr/share/man etc. directories to the proper
> > place.
> > 
> > Thanks for the ideas!
> 
> No this is no good solution
> I must install all packages and this seems an impossible task
> Too much questions I have to answer

There's nothing more to a .deb file than two .tar.xz archives
inside an .ar archive, viz:

$ ls -Glg iwd_1.14-3_amd64.deb 
-rw-r----- 1 561296 May  6  2021 iwd_1.14-3_amd64.deb

$ ar vt iwd_1.14-3_amd64.deb 
rw-r--r-- 0/0      4 May  6 02:06 2021 debian-binary
rw-r--r-- 0/0   1900 May  6 02:06 2021 control.tar.xz
rw-r--r-- 0/0 559204 May  6 02:06 2021 data.tar.xz

$ ar vx iwd_1.14-3_amd64.deb 
x - debian-binary
x - control.tar.xz
x - data.tar.xz

$ tar tvf control.tar.xz 
drwxr-xr-x root/root         0 2021-05-06 02:06 ./
-rw-r--r-- root/root       877 2021-05-06 02:06 ./control
-rw-r--r-- root/root      1298 2021-05-06 02:06 ./md5sums
-rwxr-xr-x root/root       778 2021-05-06 02:06 ./postinst
-rwxr-xr-x root/root       644 2021-05-06 02:06 ./postrm
-rwxr-xr-x root/root       219 2021-05-06 02:06 ./prerm

$ tar tvf data.tar.xz | grep '/man'
drwxr-xr-x root/root         0 2021-05-06 02:06 ./usr/share/man/
drwxr-xr-x root/root         0 2021-05-06 02:06 ./usr/share/man/man1/
-rw-r--r-- root/root       898 2021-05-06 02:06 ./usr/share/man/man1/iwctl.1.gz
-rw-r--r-- root/root       578 2021-05-06 02:06 ./usr/share/man/man1/iwmon.1.gz
drwxr-xr-x root/root         0 2021-05-06 02:06 ./usr/share/man/man5/
-rw-r--r-- root/root      2139 2021-05-06 02:06 ./usr/share/man/man5/iwd.ap.5.gz
-rw-r--r-- root/root      4264 2021-05-06 02:06 ./usr/share/man/man5/iwd.config.5.gz
-rw-r--r-- root/root      5745 2021-05-06 02:06 ./usr/share/man/man5/iwd.network.5.gz
drwxr-xr-x root/root         0 2021-05-06 02:06 ./usr/share/man/man7/
-rw-r--r-- root/root       723 2021-05-06 02:06 ./usr/share/man/man7/iwd.debug.7.gz
drwxr-xr-x root/root         0 2021-05-06 02:06 ./usr/share/man/man8/
-rw-r--r-- root/root      1148 2021-05-06 02:06 ./usr/share/man/man8/iwd.8.gz
$ 

That's a bullseye for iwd being inspected on a buster machine.
No need to install anything or run specifically "Debian" commands.

Cheers,
David.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web