Groups | Search | Server Info | Keyboard shortcuts | Login | Register

Groups > comp.os.linux.misc > #68749

Re: wget Uses Multiple Connections??

Show all headers | View raw

Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
> Jack Wallen's enthusiasm, I think, exceeds his grasp on facts, again
> <https://www.zdnet.com/article/heres-a-faster-way-to-download-files-on-linux-without-a-web-browser/>.

One wonders why you keep reading this clickbait stuff then.

> Yes, wget is a convenient way to download large items, and lots of
> items: give it a list of URLs, and it will go away and do them all,
> quietly, in the background. It has lots of options for controlling its
> automatic retry-on-failure feature, too.

Far more than that. The recursive mode is extremely useful for
mirroring sections of websites or downloading files in a
directory index. Unfortunately most of the time it takes many
attempts to stop it from accidentally trying to download the
entire internet, but the web is just not well suited to that
a lot of the time (alternatives like HTTTrack are more capable
but less convenient). When you have a plain server-generated
directory list it works very well.

The "--spider" option is great for checking the size of download
links before deciding if/where to download them.

I also just set up a system using "wget --post-file=" to submit
logs to a Bash CGI script on a web server which sorts and stores
them. Should be more reliable than using scp and running into all
the endless incompatibility pain you get with anything SSH these
days (Dropbear doesn't even support SFTP which OpenSSH calls 'scp'
too now).

> But nowhere in the docs do I find mention that it can take advantage
> of multiple simultaneous connections to speed up a download.

True it doesn't do that. It will reuse the same connection to fetch
multiple files from the same server sequentially though.

> In the standard Debian repo, I see a separate package called "aria2",
> that does indeed advertise the ability to use multiple simultaneous
> connections to speed up downloads <https://aria2.github.io/>.

I've used "lftp" for that in the past, but I don't need it often.

-- 
__          __
#_ < |\| |< _#

Back to comp.os.linux.misc | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

wget Uses Multiple Connections?? Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-12 00:33 +0000
  Re: wget Uses Multiple Connections?? John McCue <jmclnx@SPAMisBADgmail.com> - 2025-06-12 01:47 +0000
    Re: wget Uses Multiple Connections?? Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-12 02:14 +0000
    Re: wget Uses Multiple Connections?? Borax Man <rotflol2@hotmail.com> - 2025-06-12 10:56 +0000
      Re: wget Uses Multiple Connections?? "Carlos E.R." <robin_listas@es.invalid> - 2025-06-12 18:55 +0200
  Re: wget Uses Multiple Connections?? not@telling.you.invalid (Computer Nerd Kev) - 2025-06-13 09:03 +1000
    Re: wget Uses Multiple Connections?? Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-13 23:30 +0000
  Re: wget Uses Multiple Connections?? candycanearter07 <candycanearter07@candycanearter07.nomail.afraid> - 2025-06-13 01:10 +0000
  Re: wget Uses Multiple Connections?? ant@zimage.comANT (Ant) - 2025-06-15 01:18 +0000
    Re: wget Uses Multiple Connections?? candycanearter07 <candycanearter07@candycanearter07.nomail.afraid> - 2025-06-18 05:30 +0000

csiph-web