Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #4134 > unrolled thread

How can I get a plain text directory with wget?

Back to article view | Back to comp.os.linux.misc

Contents

  How can I get a plain text directory with wget? Todd <Todd@invalid.invalid> - 2012-02-05 19:18 -0800
    Re: How can I get a plain text directory with wget? Sam <sam@email-scan.com> - 2012-02-05 21:31 -0600
      Re: How can I get a plain text directory with wget? Todd <Todd@invalid.invalid> - 2012-02-05 19:44 -0800
        Re: How can I get a plain text directory with wget? Sam <sam@email-scan.com> - 2012-02-06 05:51 -0600
        Re: How can I get a plain text directory with wget? Chick Tower <c.tower@deadspam.com> - 2012-02-06 19:30 +0100
          Re: How can I get a plain text directory with wget? root <NoEMail@home.org> - 2012-02-06 18:43 +0000
            Re: How can I get a plain text directory with wget? "Chris F.A. Johnson" <cfajohnson@gmail.com> - 2012-02-07 16:37 -0500
    Re: How can I get a plain text directory with wget? Michael Black <et472@ncf.ca> - 2012-02-05 22:27 -0500
      Re: How can I get a plain text directory with wget? Todd <Todd@invalid.invalid> - 2012-02-05 19:41 -0800
        Re: How can I get a plain text directory with wget? Dan Espen <despen@verizon.net> - 2012-02-06 08:33 -0500
    Re: How can I get a plain text directory with wget? Bit Twister <BitTwister@mouse-potato.com> - 2012-02-06 04:10 +0000
      Re: How can I get a plain text directory with wget? Todd <Todd@invalid.invalid> - 2012-02-05 23:03 -0800
    Re: How can I get a plain text directory with wget? Allodoxaphobia <knock_yourself_out@example.net> - 2012-02-06 20:04 +0000
      Re: How can I get a plain text directory with wget? Michael Black <et472@ncf.ca> - 2012-02-06 17:39 -0500

#4134 — How can I get a plain text directory with wget?

Hi All,

Not having much luck with the man page.

I use the following piece of code in a bash script to
give me a directory listing:

    wget --quiet http://$FtpSite$FtpDir -O - > $Tmp 2>&1

The value pumped into $Tmp in a bunch of HTML code.
It is annoying to have to sift through it.

Is there a way to get the directory listing in plain text
without all the HTML garble-de-gook?  I would like some
thing like I would see with "ls".

Many thanks,
-T

[toc] | [next] | [standalone]

#4135

[Multipart message — attachments visible in raw view] — view raw

Todd writes:

> Hi All,
>
> Not having much luck with the man page.
>
> I use the following piece of code in a bash script to
> give me a directory listing:
>
>    wget --quiet http://$FtpSite$FtpDir -O - > $Tmp 2>&1
>
> The value pumped into $Tmp in a bunch of HTML code.
> It is annoying to have to sift through it.
>
> Is there a way to get the directory listing in plain text
> without all the HTML garble-de-gook?  I would like some
> thing like I would see with "ls".

You have to contact the administrator of the HTTP server you're trying to  
download your directory listing from, and asks for the details of their web  
server's configuration. wget sends a request for a document from the remote  
web server. Aside from specifying the document's URL, wget has no control  
over the contents of the data that it receives. Whatever the remote server  
chooses to respond with, that's what wget will get. If the remote server  
returns an HTML document, that's what wget will give you.

There's a small possibility that by specifying overriding the HTTP 1.1  
Accept: header in the request, using the --header option to wget, the remote  
web server will accept the request to return text/plain content, rather than  
text/html. You can try that, but if that doesn't work, there's nothing that  
wget can do. Your only option would be to take the return HTML document and  
convert to plain text yourself, using elinks, or something similar.

[toc] | [prev] | [next] | [standalone]

#4138

On 02/05/2012 07:31 PM, Sam wrote:
> There's a small possibility that by specifying overriding the HTTP 1.1
> Accept: header in the request, using the --header option to wget, the
> remote web server will accept the request to return text/plain content,
> rather than text/html. You can try that, but if that doesn't work,
> there's nothing that wget can do.

I am calling server mirror servers, so one may work and the other not.

> Your only option would be to take the
> return HTML document and convert to plain text yourself, using elinks,
> or something similar.

elinks sounds like a text based web browser.  Can it be used
from a script to convert html to text?

Is there some other utility to convert html to text that
can be used from a script?

Many thanks,
-T

[toc] | [prev] | [next] | [standalone]

#4149

[Multipart message — attachments visible in raw view] — view raw

Todd writes:

>> Your only option would be to take the
>> return HTML document and convert to plain text yourself, using elinks,
>> or something similar.
>
> elinks sounds like a text based web browser.  Can it be used
> from a script to convert html to text?

Yes. There aren't many options for reformatting it, it basically emits  
whatever it would ordinarily display, as text/plain content, so you'll have  
to work with however elinks ends up formatting the HTML.

[toc] | [prev] | [next] | [standalone]

#4166

On 2012-02-06, Todd <Todd@invalid.invalid> wrote:
> On 02/05/2012 07:31 PM, Sam wrote:
>> Your only option would be to take the
>> return HTML document and convert to plain text yourself, using elinks,
>> or something similar.
>
> elinks sounds like a text based web browser.  Can it be used
> from a script to convert html to text?
>
> Is there some other utility to convert html to text that
> can be used from a script?

I don't know for sure that elinks can do it, but lynx and links can
translate HTML to formatted text.  See the -dump option in the man
pages.  w3m might be able to do it, too, but I don't have it or elinks
installed, so I can't verify that.
-- 
                                 Chick Tower

For e-mail:  colm DOT sent DOT towerboy AT xoxy DOT net

[toc] | [prev] | [next] | [standalone]

#4168

Chick Tower <c.tower@deadspam.com> wrote:
> On 2012-02-06, Todd <Todd@invalid.invalid> wrote:
>> On 02/05/2012 07:31 PM, Sam wrote:
>>> Your only option would be to take the
>>> return HTML document and convert to plain text yourself, using elinks,
>>> or something similar.
>>
>> elinks sounds like a text based web browser.  Can it be used
>> from a script to convert html to text?
>>
>> Is there some other utility to convert html to text that
>> can be used from a script?
> 
> I don't know for sure that elinks can do it, but lynx and links can
> translate HTML to formatted text.  See the -dump option in the man
> pages.  w3m might be able to do it, too, but I don't have it or elinks
> installed, so I can't verify that.

Comparing lynx to w3m:
w3m does the better job with the -dump option. lynx adds all the
html links in the document as an appendix to the text:useful if
you want that.

[toc] | [prev] | [next] | [standalone]

#4195

On 2012-02-06, root wrote:
...
> lynx adds all the html links in the document as an appendix to the
> text:useful if you want that.

   If you don't want it, use the -nolist option.

-- 
   Chris F.A. Johnson, <http://cfajohnson.com>
   Author:
   Pro Bash Programming: Scripting the GNU/Linux Shell (2009, Apress)
   Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)

[toc] | [prev] | [next] | [standalone]

#4136

On Sun, 5 Feb 2012, Todd wrote:

> Hi All,
>
> Not having much luck with the man page.
>
> I use the following piece of code in a bash script to
> give me a directory listing:
>
>   wget --quiet http://$FtpSite$FtpDir -O - > $Tmp 2>&1
>
> The value pumped into $Tmp in a bunch of HTML code.
> It is annoying to have to sift through it.
>
Is it because you are actually getting an html page?  Wouldn't the URL 
start with ftp:  to get an ftp directory?


wget is great for automated work, but it's not clear where in this you 
feel it needs to be automated, not when there are other tools that may be 
simpler if all you need is the directory.


> Is there a way to get the directory listing in plain text
> without all the HTML garble-de-gook?  I would like some
> thing like I would see with "ls".
>
If you're getting html, other than fixing that in the first place, name 
the resulting file something.html and then view it with a browser.

   Michael

> Many thanks,
> -T
>

[toc] | [prev] | [next] | [standalone]

#4137

On 02/05/2012 07:27 PM, Michael Black wrote:
> On Sun, 5 Feb 2012, Todd wrote:
>
>> Hi All,
>>
>> Not having much luck with the man page.
>>
>> I use the following piece of code in a bash script to
>> give me a directory listing:
>>
>> wget --quiet http://$FtpSite$FtpDir -O - > $Tmp 2>&1
>>
>> The value pumped into $Tmp in a bunch of HTML code.
>> It is annoying to have to sift through it.
>>
> Is it because you are actually getting an html page?

Yes.  "bunch of HTML code."

> Wouldn't the URL
> start with ftp: to get an ftp directory?

I did not give your the contents of the variable.  Usually
it is for an ftp site, but very often I feed it an http site.

releases.mozilla.org is particularly a pain-in-the-ass about
not always supporting ftp calls.  I complain to them, they fix
it, and a couple of weeks later they are back to their evil ways.
So I call them with http and it always works.

The code is found in a bash function.  Lots of other stuff
going on around it.  The entire script is several hundreds
of lines long.

> wget is great for automated work, but it's not clear where in this you
> feel it needs to be automated, not when there are other tools that may
> be simpler if all you need is the directory.

Okay, you got me curious.  Which other ones?

>
>> Is there a way to get the directory listing in plain text
>> without all the HTML garble-de-gook? I would like some
>> thing like I would see with "ls".
>>
> If you're getting html, other than fixing that in the first place, name
> the resulting file something.html and then view it with a browser.

Not too practical when this is an automated script.

Thank you for the tips,
-T

[toc] | [prev] | [next] | [standalone]

#4155

Todd <Todd@invalid.invalid> writes:

R> On 02/05/2012 07:27 PM, Michael Black wrote:
>> On Sun, 5 Feb 2012, Todd wrote:
>>
>>> Hi All,
>>>
>>> Not having much luck with the man page.
>>>
>>> I use the following piece of code in a bash script to
>>> give me a directory listing:
>>>
>>> wget --quiet http://$FtpSite$FtpDir -O - > $Tmp 2>&1
>>>
>>> The value pumped into $Tmp in a bunch of HTML code.
>>> It is annoying to have to sift through it.
>>>
>> Is it because you are actually getting an html page?
>
> Yes.  "bunch of HTML code."
>
>> Wouldn't the URL
>> start with ftp: to get an ftp directory?
>
> I did not give your the contents of the variable.  Usually
> it is for an ftp site, but very often I feed it an http site.
>
> releases.mozilla.org is particularly a pain-in-the-ass about
> not always supporting ftp calls.  I complain to them, they fix

There's probably a better API for monitoring Mozilla for changes.
Is this closer to what you want:

lynx --dump ftp://releases.mozilla.org/pub

-- 
Dan Espen

[toc] | [prev] | [next] | [standalone]

#4139

On Sun, 05 Feb 2012 19:18:08 -0800, Todd wrote:
> Hi All,
>
> Not having much luck with the man page.
>
> I use the following piece of code in a bash script to
> give me a directory listing:
>
>     wget --quiet http://$FtpSite$FtpDir -O - > $Tmp 2>&1
>
> The value pumped into $Tmp in a bunch of HTML code.
> It is annoying to have to sift through it.
>
> Is there a way to get the directory listing in plain text
> without all the HTML garble-de-gook?

Without having a url to test with, I'll suggest something like 
html2text -nobs -style pretty -width 132 $FtpSite$FtpDir > $Tmp

[toc] | [prev] | [next] | [standalone]

#4140

On 02/05/2012 08:10 PM, Bit Twister wrote:
> On Sun, 05 Feb 2012 19:18:08 -0800, Todd wrote:
>> Hi All,
>>
>> Not having much luck with the man page.
>>
>> I use the following piece of code in a bash script to
>> give me a directory listing:
>>
>>      wget --quiet http://$FtpSite$FtpDir -O ->  $Tmp 2>&1
>>
>> The value pumped into $Tmp in a bunch of HTML code.
>> It is annoying to have to sift through it.
>>
>> Is there a way to get the directory listing in plain text
>> without all the HTML garble-de-gook?
>
> Without having a url to test with, I'll suggest something like
> html2text -nobs -style pretty -width 132 $FtpSite$FtpDir>  $Tmp


Hi Bit,

Love it! Thank you!   I did not even know html2text existed.
"apropose htlm" did not catch it as it was not installed
(is now).

A few bumps in the road:

    $ html2text -nobs -style pretty -width 132 \
         http://releases.mozilla.org/pub/mozilla.org/firefox/releases/

      HTTP/1.1 505 HTTP Version Not Supported Connection: close
      Date: Mon, 06 Feb 2012 06:26:41 GMT Server: Cherokee/1.0.1
      (UNIX) Content-Length: 314 Content-Type: text/html Cache-
      Control: no-cache Pragma: no-cache

      505 HTTP Version Not Supported
      ----------------------------------------------------------
      Cherokee web server 1.0.1 (UNIX), Port 80


So back to wget and pipe it to html2text:

  wget --quiet \
  http://releases.mozilla.org/pub/mozilla.org/firefox /releases/ \
  -O - |  html2text -nobs -style pretty -width 132 | grep -i DIR

     [DIR]  Parent_Directory                   -
     [DIR]  latest           31-Jan-2012 21:58 link
     [DIR]  latest-3.6       01-Feb-2012 05:38 link
     [DIR]  latest-10.0      31-Jan-2012 21:58 link
     [DIR]  2.0.0.20         18-Dec-2008 09:26 -
     [DIR]  3.0.19-real-real 16-Mar-2010 02:52 -
     [DIR]  3.6.24           03-Nov-2011 17:55 -
     [DIR]  3.6.25           13-Dec-2011 14:04 -
     [DIR]  3.6.26           29-Jan-2012 10:54 -
     [DIR]  8.0.1            21-Nov-2011 06:50 -
     [DIR]  9.0.1            21-Dec-2011 15:57 -
     [DIR]  10.0             31-Jan-2012 17:57 -


And repeating (gets me a different mirror):
    10.0/             2012-Jan-31 16:57:40 -    Directory
    2.0.0.20/         2008-Dec-18 08:26:59 -    Directory
    3.0.19-real-real/ 2010-Mar-16 01:52:50 -    Directory
    3.6.24/           2011-Nov-03 16:55:01 -    Directory
    3.6.25/           2011-Dec-13 13:04:11 -    Directory
    3.6.26/           2012-Jan-29 09:54:13 -    Directory
    8.0.1/            2011-Nov-21 05:50:19 -    Directory
    9.0.1/            2011-Dec-21 14:57:25 -    Directory
    latest/           2012-Jan-31 16:57:40 -    Directory
    latest-10.0/      2012-Jan-31 16:57:40 -    Directory
    latest-3.6/       2012-Jan-29 09:54:13 -    Directory


Oh Crap!  A third run freezes (yet another mirror).

And the directory I want is either $1 or $2 depending
on the mirror.  This would explain some of the problems
I have been having.   So I am going to really have to
think about how to carve out the directory name.

AAAHHHHH!

Okay, promise not to laugh:

$ wget --quiet \
    http://releases.mozilla.org/pub/mozilla.org/firefox/releases/ \
    -O - |  \
    html2text -nobs -style pretty -width 132 | \
    grep -i DIR | \
    sed -e "s/\[DIR\]//" | \
    awk '{print $1}' | \
    sed -e "s/\///"

Did I hear you just laugh?!?!

Thank you for the help!
-T

[toc] | [prev] | [next] | [standalone]

#4169

On Sun, 05 Feb 2012 19:18:08 -0800, Todd wrote:
> Hi All,
>
> Not having much luck with the man page.
>
> I use the following piece of code in a bash script to
> give me a directory listing:
>
>     wget --quiet http://$FtpSite$FtpDir -O - > $Tmp 2>&1
>
> The value pumped into $Tmp in a bunch of HTML code.
> It is annoying to have to sift through it.
>
> Is there a way to get the directory listing in plain text
> without all the HTML garble-de-gook?  I would like some
> thing like I would see with "ls".

 lynx -dump -nolist "http://$FtpSite$FtpDir" > dir.list.ing

HTH
Jonesy

[toc] | [prev] | [next] | [standalone]

#4173

On Mon, 6 Feb 2012, Allodoxaphobia wrote:

> On Sun, 05 Feb 2012 19:18:08 -0800, Todd wrote:
>> Hi All,
>>
>> Not having much luck with the man page.
>>
>> I use the following piece of code in a bash script to
>> give me a directory listing:
>>
>>     wget --quiet http://$FtpSite$FtpDir -O - > $Tmp 2>&1
>>
>> The value pumped into $Tmp in a bunch of HTML code.
>> It is annoying to have to sift through it.
>>
>> Is there a way to get the directory listing in plain text
>> without all the HTML garble-de-gook?  I would like some
>> thing like I would see with "ls".
>
> lynx -dump -nolist "http://$FtpSite$FtpDir" > dir.list.ing
>
I think that's assuming that lynx has "user mode" is set for novice (maybe 
intermediate does it too).  Advanced doesn't put the list of links at the 
end of the page.

    Michael

[toc] | [prev] | [standalone]

Back to top | Article view | comp.os.linux.misc

csiph-web