Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #275877 > unrolled thread

repository setup and apt-file usage

Back to article view | Back to linux.debian.user

Contents

  repository setup and apt-file usage Frank Guthausen <fg.debian@shimps.de> - 2024-12-18 15:40 +0100
    Re: repository setup and apt-file usage Tim Woodall <debianuser@woodall.me.uk> - 2024-12-18 20:00 +0100
      Re: repository setup and apt-file usage Frank Guthausen <fg.debian@shimps.de> - 2024-12-18 20:20 +0100
        Re: repository setup and apt-file usage Frank Guthausen <fg.debian@shimps.de> - 2024-12-18 20:50 +0100
        Re: repository setup and apt-file usage Tim Woodall <debianuser@woodall.me.uk> - 2024-12-18 21:00 +0100
          Re: repository setup and apt-file usage Frank Guthausen <fg.debian@shimps.de> - 2024-12-18 23:40 +0100
    Re: repository setup and apt-file usage Frank Jezzer <etphonehomefrance@gmail.com> - 2024-12-25 17:50 +0100
      Re: repository setup and apt-file usage dulev <dulev@lugons.org> - 2024-12-26 10:30 +0100
      Re: repository setup and apt-file usage dulev <dulev@lugons.org> - 2024-12-26 10:30 +0100

#275877 — repository setup and apt-file usage

[Multipart message — attachments visible in raw view] — view raw

Hello.

I try to setup a repository with a similar structure as Debian's
original, just smaller. In the Debian repository, e.g. bookworm main[1]
there are a lot Contents-*.gz files. My repository is created with
reprepro following the wiki[2]. But I cannot find information about the
missing Contents-* files. What is the intended workflow to create them?

The files of my packages are not included in the database of apt-file,
the mandatory ``apt-file update'' does not help to find files or show
files in my packages.

I guess the problems are related.

What can I do to solve those issues?

 [1] https://ftp.debian.org/debian/dists/bookworm/main/
 [2] https://wiki.debian.org/DebianRepository/SetupWithReprepro

-- 
kind regards
Frank

[toc] | [next] | [standalone]

#275894

On Wed, 18 Dec 2024, Frank Guthausen wrote:

> Hello.
>
> I try to setup a repository with a similar structure as Debian's
> original, just smaller. In the Debian repository, e.g. bookworm main[1]
> there are a lot Contents-*.gz files. My repository is created with
> reprepro following the wiki[2]. But I cannot find information about the
> missing Contents-* files. What is the intended workflow to create them?
>
> The files of my packages are not included in the database of apt-file,
> the mandatory ``apt-file update'' does not help to find files or show
> files in my packages.
>
> I guess the problems are related.
>
> What can I do to solve those issues?
>
> [1] https://ftp.debian.org/debian/dists/bookworm/main/
> [2] https://wiki.debian.org/DebianRepository/SetupWithReprepro
>

I use apt-ftp-archive to do this so I can't help you with reprepo but
the first thing I'd check is whether the website looks reasonable.

I'm pretty sure the contents files aren't necessary for a working repo
(and I think apt-ftp-archive at least limits how much delta there is to
these files so they don't necessarily generate straight away)

I'm not sure which point is going wrong so below I've put the steps of
browsing the repo that needs to be available - if this structure all
exists but the Packages file is empty then it's presumably a reprepo problem.

So I have this sources file:

Types: deb
URIs: http://aptmirror.home.woodall.me.uk/local
Suites: bookworm
Components: main

when I browse to that website (not publicly available) I then see two
directories /local/dists/ and /local/pool/
(I've dropped the http://apt...uk from all of the links and just
included the path)

/local/dists/bookworm/InRelease will need to have lines something like:
  342869da630a775fe229d4bc2ebe09ff           246670 main/binary-amd64/Packages
  82a2348876c6d58709fe6c928f05ba07            56864 main/binary-amd64/Packages.xz

And then those files need to exist here in my case (relative to the
InRelease file):
/local/dists/bookworm/main/binary-amd64/


In the Packages file you will to see the packages listed, here's one
example from mine:

Package: apt-cacher-ng
Architecture: amd64
Version: 3.7.5-1+~tjw12r1
Priority: net
Section: optional
Maintainer: Eduard Bloch <blade@debian.org>
Installed-Size: 1833
Pre-Depends: dpkg (>= 1.15.6)
Depends: libbz2-1.0, libc-ares2 (>= 1.16.0), libc6 (>= 2.34), libevent-2.1-7 (>= 2.1.8-stable), libevent-pthreads-2.1-7 (>= 2.1.8-stable), libfuse2 (>= 2.6), libgcc-s1 (>= 3.0), liblzma5 (>= 5.1.1alpha+20120614), libssl3 (>= 3.0.0), libstdc++6 (>= 12), libsystemd0 (>= 210), libwrap0 (>= 7.6-4~), zlib1g (>= 1:1.2.0), debconf (>= 0.5) | debconf-2.0, adduser, sysvinit-utils (>= 3.05-4~) | lsb-base (>> 3.0-6)
Recommends: ca-certificates
Suggests: doc-base, avahi-daemon
Conflicts: logrotate (<< 3.8.0)
Filename: pool/main/a/apt-cacher-ng/apt-cacher-ng_3.7.5-1+~tjw12r1_amd64.deb
Size: 684648
MD5sum: 1e07511c47939f7deed5f9f1b4a20dfb
SHA1: 187491c9490c4449b17c9d6acd9c3afcd380e340
SHA256: 4748419918c5093623df26158c5c2fab4a7fb64c220c3869b6d64636c04c97cf
SHA512: 5c45eeb0e6f1a4597252e085bfcd481a95732e622b454b3082ebd2abf42723454f425c8fdff73efd7d3ac066969701502fea7c46c16546e8f8c5b8459c22bee4
Homepage: http://www.unix-ag.uni-kl.de/~bloch/acng/
Description: caching proxy server for software repositories
  Apt-Cacher NG is a caching proxy for downloading packages from Debian-style
  software repositories (or possibly from other types).
  .
  The main principle is that a central machine hosts the proxy for a local
  network, and clients configure their APT setup to download through it.
  Apt-Cacher NG keeps a copy of all useful data that passes through it, and when
  a similar request is made, the cached copy of the data is delivered without
  being re-downloaded.
  .
  Apt-Cacher NG has been designed from scratch as a replacement for
  apt-cacher, but with a focus on maximizing throughput with low system
  resource requirements. It can also be used as replacement for apt-proxy and
  approx with no need to modify clients' sources.list files.

and then you see that Filename: line in there, that file needs to exist
too which I see when I browse to here:
/local/pool/main/a/apt-cacher-ng/


HTH.

Tim.

[toc] | [prev] | [next] | [standalone]

#275896

[Multipart message — attachments visible in raw view] — view raw

On Wed, 18 Dec 2024 18:51:58 +0000 (GMT)
Tim Woodall <debianuser@woodall.me.uk> wrote:
> On Wed, 18 Dec 2024, Frank Guthausen wrote:
> >
> > The files of my packages are not included in the database of
> > apt-file, the mandatory ``apt-file update'' does not help to find
> > files or show files in my packages.
> >
> > I guess the problems are related.
> 
> I'm pretty sure the contents files aren't necessary for a working repo
> [...]
> 
> I'm not sure which point is going wrong [...]

The repository itself works fine. But I cannot use apt-file to show or
find content of packages. And all the Contents-*.gz files are missing.

I don't know any tool to generate them, and I don't know the workflow.
In Debian bookworm Release file[1] those files are included as visible
within e.g. the first 25 lines of this file.

 [1] https://ftp.debian.org/debian/dists/bookworm/Release

-- 
kind regards
Frank

[toc] | [prev] | [next] | [standalone]

#275899

[Multipart message — attachments visible in raw view] — view raw

On Wed, 18 Dec 2024 20:13:20 +0100
Frank Guthausen <fg.debian@shimps.de> wrote:
> 
> The repository itself works fine. But I cannot use apt-file to show or
> find content of packages. And all the Contents-*.gz files are missing.

Peter Pentchev posted the solution in debian-devel[1]. It's a
configuration of reprepro, conf/distributions file needs a line
containing "Contents:" only. Now the files are generated and apt-file
can deal with the content of the packages.

 [1] https://lists.debian.org/debian-devel/2024/12/msg00384.html

-- 
kind regards
Frank

[toc] | [prev] | [next] | [standalone]

#275901

On Wed, 18 Dec 2024, Frank Guthausen wrote:

> On Wed, 18 Dec 2024 18:51:58 +0000 (GMT)
> Tim Woodall <debianuser@woodall.me.uk> wrote:
>> On Wed, 18 Dec 2024, Frank Guthausen wrote:
>>>
>>> The files of my packages are not included in the database of
>>> apt-file, the mandatory ``apt-file update'' does not help to find
>>> files or show files in my packages.
>>>
>>> I guess the problems are related.
>>
>> I'm pretty sure the contents files aren't necessary for a working repo
>> [...]
>>
>> I'm not sure which point is going wrong [...]
>
> The repository itself works fine. But I cannot use apt-file to show or
> find content of packages. And all the Contents-*.gz files are missing.
>
> I don't know any tool to generate them, and I don't know the workflow.

apt-ftparchive generates them. It's not the easiest thing to use though.

And I see you asked on debian-devel and someone there suggested what the
config setting is for reprepo I think.

Tim.

[toc] | [prev] | [next] | [standalone]

#275905

[Multipart message — attachments visible in raw view] — view raw

On Wed, 18 Dec 2024 19:57:42 +0000 (GMT)
Tim Woodall <debianuser@woodall.me.uk> wrote:
> On Wed, 18 Dec 2024, Frank Guthausen wrote:
> >
> > I don't know any tool to generate them, and I don't know the
> > workflow.  
> 
> apt-ftparchive
> generates them. It's not the easiest thing to use though.

My first run before reprepro was this, but without configuration and
deeper understanding. I aim not only for solutions, but for a deeper
understanding. The tool apt-ftparchive didn't fullfill my ecpectations
and needs. Maybe a lack of patience, I don't know.

> And I see you asked on debian-devel and someone there suggested what
> the config setting is for reprepo I think.

Yes, the problem is solved. How can the apt-ftparchive tool support the
average admin expectacion and why is this not default? I'm glad someone
pointed me to the reprepro tool, and with Peter's help I've got the
conf which fullfills my dreams. I took care about the missing line
in the Debian wiki page[1]. I got the conf template from there. I
improved the wiki page with our results in a conservative way, hoping
to find agreement about the topic.

 [1] https://wiki.debian.org/DebianRepository/SetupWithReprepro

-- 
kind regards
Frank

[toc] | [prev] | [next] | [standalone]

#276007

On 2024-12-18, Frank Guthausen <fg.debian@shimps.de> wrote:
>
> I try to setup a repository with a similar structure as Debian's
> original, just smaller. In the Debian repository, e.g. bookworm main[1]
> there are a lot Contents-*.gz files. My repository is created with
> reprepro following the wiki[2]. But I cannot find information about the
> missing Contents-* files. What is the intended workflow to create them?
>
> The files of my packages are not included in the database of apt-file,
> the mandatory ``apt-file update'' does not help to find files or show
> files in my packages.
>
> I guess the problems are related.
>
> What can I do to solve those issues?
>
>  [1] https://ftp.debian.org/debian/dists/bookworm/main/
>  [2] https://wiki.debian.org/DebianRepository/SetupWithReprepro
>

Creating and managing a repository with a structure similar to Debian's
requires generating the Contents-*.gz files, which map the files provided by
packages to their respective packages. These files are crucial for tools like
apt-file to function correctly. If you are using reprepro to manage your
repository, these files are not generated by default. Here's how you can
address this:

The Contents-*.gz files list all the files included in the packages of a
particular section and architecture in the repository. These files allow tools
like apt-file to index and search for files provided by packages.

reprepro is a lightweight tool and does not have built-in support for creating
Contents-*.gz files. This is why you don't see these files in your repository
and why apt-file update doesn't work as expected.

You can generate the Contents-*.gz files manually or use a script to automate
the process. Here’s a general workflow:

Use a script to extract the contents of the data.tar.* (or data.cpio.*)
archives from the .deb files in your repository. For example:

for deb in pool/main/*/*.deb; do dpkg-deb -c "$deb" | awk '{print $6}' | sed
"s|^|$(basename "$deb" .deb) |" done This will produce lines in the format
package file, which map files to their packages.

Organize the output by architecture (e.g., amd64, arm64) and section (e.g.,
main, contrib).

For each architecture and section, write the organized output into a text file
named Contents-<architecture>.gz. For example:

gzip -c > dists/bookworm/main/Contents-amd64.gz Step 4: Automate the Process
You can create a shell or Python script to iterate through all architectures
and sections in your repository, generating the required files.

After generating the Contents-*.gz files, ensure they are included in the
Release file of your repository so that apt-file can detect them:

Update the Release file with the appropriate checksums for the Contents-*.gz
files:

reprepro export Run apt update and then apt-file update on the client system to
test the functionality.

Here is a basic example script:

#!/bin/bash REPO_PATH="/path/to/your/repo" DIST="bookworm" SECTIONS=("main"
"contrib" "non-free") ARCHITECTURES=("amd64" "arm64")

for SECTION in "${SECTIONS[@]}"; do for ARCH in "${ARCHITECTURES[@]}"; do
CONTENTS_FILE="$REPO_PATH/dists/$DIST/$SECTION/Contents-$ARCH"

[toc] | [prev] | [next] | [standalone]

#276016

On Wed, 25 Dec 2024 16:40:36 -0000 (UTC), etphonehomefrance@gmail.com wrote:
> On 2024-12-18, Frank Guthausen <fg.debian@shimps.de> wrote:
> >
> > I try to setup a repository with a similar structure as Debian's
> > original, just smaller. In the Debian repository, e.g. bookworm main[1]
> > there are a lot Contents-*.gz files. My repository is created with
> > reprepro following the wiki[2]. But I cannot find information about the
> > missing Contents-* files. What is the intended workflow to create them?
> >
> > The files of my packages are not included in the database of apt-file,
> > the mandatory ``apt-file update'' does not help to find files or show
> > files in my packages.
> >
> > I guess the problems are related.
> >
> > What can I do to solve those issues?
> >
> >  [1] https://ftp.debian.org/debian/dists/bookworm/main/
> >  [2] https://wiki.debian.org/DebianRepository/SetupWithReprepro
> >
> 
> Creating and managing a repository with a structure similar to Debian's
> requires generating the Contents-*.gz files, which map the files provided by
> packages to their respective packages. These files are crucial for tools
> like apt-file to function correctly. If you are using reprepro to manage
> your repository, these files are not generated by default. Here's how you
> can address this:
> 
> The Contents-*.gz files list all the files included in the packages of a
> particular section and architecture in the repository. These files allow
> tools like apt-file to index and search for files provided by packages.
> 
> reprepro is a lightweight tool and does not have built-in support for
> creating Contents-*.gz files. This is why you don't see these files in your
> repository and why apt-file update doesn't work as expected.
> 
> You can generate the Contents-*.gz files manually or use a script to
> automate the process. Here’s a general workflow:
> 
> Use a script to extract the contents of the data.tar.* (or data.cpio.*)
> archives from the .deb files in your repository. For example:
> 
> for deb in pool/main/*/*.deb; do dpkg-deb -c "$deb" | awk '{print $6}' | sed
> "s|^|$(basename "$deb" .deb) |" done This will produce lines in the format
> package file, which map files to their packages.
> 
> Organize the output by architecture (e.g., amd64, arm64) and section (e.g.,
> main, contrib).
> 
> For each architecture and section, write the organized output into a text
> file named Contents-<architecture>.gz. For example:
> 
> gzip -c > dists/bookworm/main/Contents-amd64.gz Step 4: Automate the Process
> You can create a shell or Python script to iterate through all architectures
> and sections in your repository, generating the required files.
> 
> After generating the Contents-*.gz files, ensure they are included in the
> Release file of your repository so that apt-file can detect them:
> 
> Update the Release file with the appropriate checksums for the Contents-*.gz
> files:
> 
> reprepro export Run apt update and then apt-file update on the client
> system to test the functionality.
> 
> Here is a basic example script:
> 
> #!/bin/bash REPO_PATH="/path/to/your/repo" DIST="bookworm" SECTIONS=("main"
> "contrib" "non-free") ARCHITECTURES=("amd64" "arm64")
> 
> for SECTION in "${SECTIONS[@]}"; do for ARCH in "${ARCHITECTURES[@]}"; do
> CONTENTS_FILE="$REPO_PATH/dists/$DIST/$SECTION/Contents-$ARCH"
> 

man reprepro -> conf/distributions -> Contents

[toc] | [prev] | [next] | [standalone]

#276017

On Wed, 25 Dec 2024 16:40:36 -0000 (UTC), etphonehomefrance@gmail.com wrote:
> On 2024-12-18, Frank Guthausen <fg.debian@shimps.de> wrote:
> >
> > I try to setup a repository with a similar structure as Debian's
> > original, just smaller. In the Debian repository, e.g. bookworm main[1]
> > there are a lot Contents-*.gz files. My repository is created with
> > reprepro following the wiki[2]. But I cannot find information about the
> > missing Contents-* files. What is the intended workflow to create them?
> >
> > The files of my packages are not included in the database of apt-file,
> > the mandatory ``apt-file update'' does not help to find files or show
> > files in my packages.
> >
> > I guess the problems are related.
> >
> > What can I do to solve those issues?
> >
> >  [1] https://ftp.debian.org/debian/dists/bookworm/main/
> >  [2] https://wiki.debian.org/DebianRepository/SetupWithReprepro
> >
> 
> Creating and managing a repository with a structure similar to Debian's
> requires generating the Contents-*.gz files, which map the files provided by
> packages to their respective packages. These files are crucial for tools
> like apt-file to function correctly. If you are using reprepro to manage
> your repository, these files are not generated by default. Here's how you
> can address this:
> 
> The Contents-*.gz files list all the files included in the packages of a
> particular section and architecture in the repository. These files allow
> tools like apt-file to index and search for files provided by packages.
> 
> reprepro is a lightweight tool and does not have built-in support for
> creating Contents-*.gz files. This is why you don't see these files in your
> repository and why apt-file update doesn't work as expected.
> 
> You can generate the Contents-*.gz files manually or use a script to
> automate the process. Here’s a general workflow:
> 
> Use a script to extract the contents of the data.tar.* (or data.cpio.*)
> archives from the .deb files in your repository. For example:
> 
> for deb in pool/main/*/*.deb; do dpkg-deb -c "$deb" | awk '{print $6}' | sed
> "s|^|$(basename "$deb" .deb) |" done This will produce lines in the format
> package file, which map files to their packages.
> 
> Organize the output by architecture (e.g., amd64, arm64) and section (e.g.,
> main, contrib).
> 
> For each architecture and section, write the organized output into a text
> file named Contents-<architecture>.gz. For example:
> 
> gzip -c > dists/bookworm/main/Contents-amd64.gz Step 4: Automate the Process
> You can create a shell or Python script to iterate through all architectures
> and sections in your repository, generating the required files.
> 
> After generating the Contents-*.gz files, ensure they are included in the
> Release file of your repository so that apt-file can detect them:
> 
> Update the Release file with the appropriate checksums for the Contents-*.gz
> files:
> 
> reprepro export Run apt update and then apt-file update on the client
> system to test the functionality.
> 
> Here is a basic example script:
> 
> #!/bin/bash REPO_PATH="/path/to/your/repo" DIST="bookworm" SECTIONS=("main"
> "contrib" "non-free") ARCHITECTURES=("amd64" "arm64")
> 
> for SECTION in "${SECTIONS[@]}"; do for ARCH in "${ARCHITECTURES[@]}"; do
> CONTENTS_FILE="$REPO_PATH/dists/$DIST/$SECTION/Contents-$ARCH"
> 

man reprepro -> conf/distributions -> Contents

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web