Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #195750 > unrolled thread

"pre-treating" documents from certain remote URLs before a web browser renders them

Back to article view | Back to linux.debian.user

Contents

  "pre-treating" documents from certain remote URLs before a web  browser renders them davidson <davidson@freevolt.org> - 2018-05-15 02:20 +0200
    Re: "pre-treating" documents from certain remote URLs before a web  browser renders them <tomas@tuxteam.de> - 2018-05-15 08:40 +0200
    Re: "pre-treating" documents from certain remote URLs before a web  browser renders them Reco <recoverym4n@gmail.com> - 2018-05-15 14:00 +0200
      Re: "pre-treating" documents from certain remote URLs before a web  browser renders them Celejar <celejar@gmail.com> - 2018-05-15 19:30 +0200
    Re: "pre-treating" documents from certain remote URLs before a web  browser renders them "Alexander V. Makartsev" <avbetev@gmail.com> - 2018-05-15 14:30 +0200
    Re: "pre-treating" documents from certain remote URLs before a web  browser renders them Dan Ritter <dsr@randomstring.org> - 2018-05-15 23:20 +0200
    Re: "pre-treating" documents from certain remote URLs before a web  browser renders them davidson <davidson@freevolt.org> - 2018-05-17 02:20 +0200
      Re: "pre-treating" documents from certain remote URLs before a web browser renders them Kushal Kumaran <kushal@locationd.net> - 2018-05-17 04:40 +0200
        Re: "pre-treating" documents from certain remote URLs before a web  browser renders them <tomas@tuxteam.de> - 2018-05-17 09:00 +0200
          Re: "pre-treating" documents from certain remote URLs before a web  browser renders them Celejar <celejar@gmail.com> - 2018-05-17 14:00 +0200
            Re: "pre-treating" documents from certain remote URLs before a web  browser renders them <tomas@tuxteam.de> - 2018-05-17 14:00 +0200
          Re: "pre-treating" documents from certain remote URLs before a web  browser renders them Reco <recoverym4n@gmail.com> - 2018-05-17 14:50 +0200
            Re: "pre-treating" documents from certain remote URLs before a web  browser renders them <tomas@tuxteam.de> - 2018-05-17 16:50 +0200

#195750 — "pre-treating" documents from certain remote URLs before a web browser renders them

I have a problem: The more frequently I browse a web site, the more I
notice all the things I hate about its web pages.

And I seem to have a partial solution to this problem: I can make XSLT
stylesheets[1] that will transform a web page A, as received from a
remote site, into a XHTML document B that better suits my purposes.[2]
I find it entertaining to make these, so I would like to figure out
how to incorporate them into a solution to the problem above.

But at the moment I do not know a good solution to the rest of the
problem, namely how to incorporate the application of stylesheets (and
preliminary preprocessing) into the web browsing activity.

I would like to launch a web browser[3], browse pages at domain X, and
know that when I go to http://X/page, or https://X/page, etc, the
browser will render not the page served from the remote site, but will
render instead that page as transformed by the appropriate stylesheet
tailored for pages from X.

Do I want a local proxy server that I can instruct to apply
appropriate transformations to documents received from certain
domains? This seems sensible, but I haven't examined too deeply what
is available along these lines, and I would rather not spend time
digging into a dead end.

I would appreciate any suggestions, experiences (bad or good) along
these lines, etc.

NOTES

1. "Stylesheet" *really* feels like the wrong term here, but I'm not
aware of a better conventional one. I'm not, as a rule, imposing style
so much as pruning document structure. I mostly just censor elements
that I consider obstacles, and rearrange the remainder so that I can
more reliably/simply locate whatever I'm looking for.

2. In order to transform page A into valid XML before applying such
transformations, I first run A through a woodchipper called tidy:
http://www.html-tidy.org/

3. And ideally an arbitrary web browser. If for some reason I have to
pick just one, lynx is a strong preference, all else equal. But, of
course, I'm interested in hearing about alternative web clients if
they provide some advantage here.

[toc] | [next] | [standalone]

#195758

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, May 15, 2018 at 12:58:04AM +0000, davidson wrote:
> I have a problem: The more frequently I browse a web site, the more I
> notice all the things I hate about its web pages.

[...]

I think you're looking for a HTTP proxy.

Cheers
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlr6gBgACgkQBcgs9XrR2kasQwCdG6qtJqLVrZoDtK/komaECbvd
BboAnilMlBRKrxx0aGiZxYYquvB/YgV+
=i36s
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#195766

	Hi.

On Tue, May 15, 2018 at 12:58:04AM +0000, davidson wrote:
> I would like to launch a web browser[3], browse pages at domain X, and
> know that when I go to http://X/page, or https://X/page, etc, the
> browser will render not the page served from the remote site, but will
> render instead that page as transformed by the appropriate stylesheet
> tailored for pages from X.

I can think of two things that can do the job:

1) HTTP proxy.

Squid may be able to pull the trick you want, but I'm unsure of that.
Nginx (configured as a conventional proxy) can do it via
mod-http-xslt-filter, but writing nginx.conf for this will be a PITA.

2) Client-side javascript invoking needed XSLT transformation.

The extension you need is called Greasemonkey for Firefox and
Violentmonkey for Chromium.
An approach requires minimal knowledge of Javascript.

Reco

[toc] | [prev] | [next] | [standalone]

#195775

On Tue, 15 May 2018 14:52:16 +0300
Reco <recoverym4n@gmail.com> wrote:

> 	Hi.
> 
> On Tue, May 15, 2018 at 12:58:04AM +0000, davidson wrote:
> > I would like to launch a web browser[3], browse pages at domain X, and
> > know that when I go to http://X/page, or https://X/page, etc, the
> > browser will render not the page served from the remote site, but will
> > render instead that page as transformed by the appropriate stylesheet
> > tailored for pages from X.
> 
> I can think of two things that can do the job:
> 
> 1) HTTP proxy.
> 
> Squid may be able to pull the trick you want, but I'm unsure of that.
> Nginx (configured as a conventional proxy) can do it via
> mod-http-xslt-filter, but writing nginx.conf for this will be a PITA.

You want privoxy. Insofar as you can script your transformation,
configure privoxy, using its "external-filter" action, to apply your
script to the appropriate pages:

https://www.privoxy.org/user-manual/actions-file.html

[section "8.5.14. external-filter"]

Celejar

[toc] | [prev] | [next] | [standalone]

#195767

[Multipart message — attachments visible in raw view] — view raw

On 15.05.2018 05:58, davidson wrote:
> Do I want a local proxy server that I can instruct to apply
> appropriate transformations to documents received from certain
> domains? This seems sensible, but I haven't examined too deeply what
> is available along these lines, and I would rather not spend time
> digging into a dead end.
>
> I would appreciate any suggestions, experiences (bad or good) along
> these lines, etc. 
I also suggest to use filtering local proxy for that.
Try "Privoxy". It was made to be used for web filtering and http
protocol debugging and served me well for many years for similar
purposes (Ads, tracking and annoyances filtering).

-- 
With kindest regards, Alexander.

⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀ 

[toc] | [prev] | [next] | [standalone]

#195777

On Tue, May 15, 2018 at 12:58:04AM +0000, davidson wrote:
> And I seem to have a partial solution to this problem: I can make XSLT
> stylesheets[1] that will transform a web page A, as received from a
> remote site, into a XHTML document B that better suits my purposes.[2]
> I find it entertaining to make these, so I would like to figure out
> how to incorporate them into a solution to the problem above.
> 
> But at the moment I do not know a good solution to the rest of the
> problem, namely how to incorporate the application of stylesheets (and
> preliminary preprocessing) into the web browsing activity.
> 
> I would like to launch a web browser[3], browse pages at domain X, and
> know that when I go to http://X/page, or https://X/page, etc, the
> browser will render not the page served from the remote site, but will
> render instead that page as transformed by the appropriate stylesheet
> tailored for pages from X.

An amusingly large percentage of these things can be done in
Firefox via JavaScript and CSS.

See GreaseMonkey:  https://wiki.greasespot.net/Main_Page

-dsr-

[toc] | [prev] | [next] | [standalone]

#195824

On Tue, 15 May 2018, davidson wrote:

> I have a problem: The more frequently I browse a web site, the more I
> notice all the things I hate about its web pages.
>
> And I seem to have a partial solution to this problem: I can make XSLT
> stylesheets[1] that will transform a web page A, as received from a
> remote site, into a XHTML document B that better suits my purposes.[2]
> I find it entertaining to make these, so I would like to figure out
> how to incorporate them into a solution to the problem above.
>
> But at the moment I do not know a good solution to the rest of the
> problem, namely how to incorporate the application of stylesheets (and
> preliminary preprocessing) into the web browsing activity.

[snip]

> I would appreciate any suggestions, experiences (bad or good) along
> these lines, etc.

I would like to thank everybody who took some time to reply with
suggestions and experience. So, thanks!

I will experiment with privoxy first, but I've taken note of the clues
and hints about alternatives too.

Grateful for the help.

[toc] | [prev] | [next] | [standalone]

#195825 — Re: "pre-treating" documents from certain remote URLs before a web browser renders them

davidson <davidson@freevolt.org> writes:

> On Tue, 15 May 2018, davidson wrote:
>
>> I have a problem: The more frequently I browse a web site, the more I
>> notice all the things I hate about its web pages.
>>
>> And I seem to have a partial solution to this problem: I can make XSLT
>> stylesheets[1] that will transform a web page A, as received from a
>> remote site, into a XHTML document B that better suits my purposes.[2]
>> I find it entertaining to make these, so I would like to figure out
>> how to incorporate them into a solution to the problem above.
>>
>> But at the moment I do not know a good solution to the rest of the
>> problem, namely how to incorporate the application of stylesheets (and
>> preliminary preprocessing) into the web browsing activity.
>
> [snip]
>
>> I would appreciate any suggestions, experiences (bad or good) along
>> these lines, etc.
>
> I would like to thank everybody who took some time to reply with
> suggestions and experience. So, thanks!
>
> I will experiment with privoxy first, but I've taken note of the clues
> and hints about alternatives too.
>
> Grateful for the help.

You should note that HTTP-proxy based systems will not be able to do any
inspection or modification of traffic for sites using HTTPS.
Browser-based systems like greasemonkey work better for that.  Since you
mentioned xslt, there is someone at [1] mangling web pages using xslt
with a greasemonkey script that you could use as inspiration.

[1] https://stackoverflow.com/questions/17998446/how-to-transform-an-xml-file-with-xslt-using-a-greasemonkey-script

-- 
regards,
kushal

[toc] | [prev] | [next] | [standalone]

#195826

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, May 16, 2018 at 07:35:51PM -0700, Kushal Kumaran wrote:

[...]

> You should note that HTTP-proxy based systems will not be able to do any
> inspection or modification of traffic for sites using HTTPS.

This is true... and then it's not :-)

If your proxy terminates the HTTPS connection, effectively doing a
"man-in-the-middle" (but controlled by you), it can: most probably
you'd have to fool your browser by offering it a HTTPS connection
from the proxy, and by installing a trusted root certificate you
create yourself. Basically what the proxy in your $CORPORATION does
all of the time.

I don't know whether privoxy or squid can do that (I'd love to know,
mind you, but days are so short).

Cheers
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlr9JeAACgkQBcgs9XrR2kZlfgCfUj93U5VfwlbzO6HPxLCjQIIP
a20AnjQmH9FHDdVbIujYUJTKE+OTqkYJ
=Lits
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#195837

On Thu, 17 May 2018 08:49:04 +0200
<tomas@tuxteam.de> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Wed, May 16, 2018 at 07:35:51PM -0700, Kushal Kumaran wrote:
> 
> [...]
> 
> > You should note that HTTP-proxy based systems will not be able to do any
> > inspection or modification of traffic for sites using HTTPS.
> 
> This is true... and then it's not :-)
> 
> If your proxy terminates the HTTPS connection, effectively doing a
> "man-in-the-middle" (but controlled by you), it can: most probably
> you'd have to fool your browser by offering it a HTTPS connection
> from the proxy, and by installing a trusted root certificate you
> create yourself. Basically what the proxy in your $CORPORATION does
> all of the time.
> 
> I don't know whether privoxy or squid can do that (I'd love to know,
> mind you, but days are so short).

Privoxy apparently has no native support for this, but people have
apparenly constructed working solutions using things like stunnel and
ProxHTTPSProxy:

https://www.stunnel.org/pipermail/stunnel-users/2006-April/001083.html
https://sourceforge.net/p/ijbswa/support-requests/1512/
https://sourceforge.net/p/ijbswa/support-requests/1667/
https://sourceforge.net/p/ijbswa/support-requests/1654/
https://news.ycombinator.com/item?id=8822974


Celejar

[toc] | [prev] | [next] | [standalone]

#195838

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, May 17, 2018 at 07:52:55AM -0400, Celejar wrote:
> On Thu, 17 May 2018 08:49:04 +0200
> <tomas@tuxteam.de> wrote:

[... proxy as SSL endpoint...]

> > I don't know whether privoxy or squid can do that (I'd love to know,
> > mind you, but days are so short).
> 
> Privoxy apparently has no native support for this, but people have
> apparenly constructed working solutions using things like stunnel and
> ProxHTTPSProxy:
> 
> https://www.stunnel.org/pipermail/stunnel-users/2006-April/001083.html
> https://sourceforge.net/p/ijbswa/support-requests/1512/
> https://sourceforge.net/p/ijbswa/support-requests/1667/
> https://sourceforge.net/p/ijbswa/support-requests/1654/
> https://news.ycombinator.com/item?id=8822974

Thanks :)

Cheers
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlr9blQACgkQBcgs9XrR2kb6twCfeqhf31EAMdjdf/7hEtlWAHH7
uDoAn2iZeKbQ1jOekXvVok7VmuTem+jk
=bxPE
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#195840

	Hi.

On Thu, May 17, 2018 at 08:49:04AM +0200, tomas@tuxteam.de wrote:
> On Wed, May 16, 2018 at 07:35:51PM -0700, Kushal Kumaran wrote:
> 
> [...]
> 
> > You should note that HTTP-proxy based systems will not be able to do any
> > inspection or modification of traffic for sites using HTTPS.
> 
> This is true... and then it's not :-)
> 
> If your proxy terminates the HTTPS connection, effectively doing a
> "man-in-the-middle" (but controlled by you), it can: most probably
> you'd have to fool your browser by offering it a HTTPS connection
> from the proxy, and by installing a trusted root certificate you
> create yourself. Basically what the proxy in your $CORPORATION does
> all of the time.
> 
> I don't know whether privoxy or squid can do that (I'd love to know,
> mind you, but days are so short).

Squid can do it. It was called SSL Bump in old (pre 3.4) Squid, now they
renamed it to SSL Peek and Splice - [1].

Reco

https://wiki.squid-cache.org/Features/SslPeekAndSplice

[toc] | [prev] | [next] | [standalone]

#195841

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, May 17, 2018 at 03:40:24PM +0300, Reco wrote:

[...]

> Squid can do it. It was called SSL Bump in old (pre 3.4) Squid, now they
> renamed it to SSL Peek and Splice - [1].

Thanks!

- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlr9lVUACgkQBcgs9XrR2kZKQwCfeq9ciQccW5icyx6mjlCJtqyV
PfcAn212kZ1EV3HtgetETHgd/NFsGpE6
=IRRz
-----END PGP SIGNATURE-----

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web