Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #54728 > unrolled thread

Referrer key missing form os.environ dictionary?

Back to article view | Back to comp.lang.python

Contents

  Referrer key missing form os.environ dictionary? Νίκος <nikos.gr33k@gmail.com> - 2013-09-25 15:45 +0300
    Re: Referrer key missing form os.environ dictionary? Νίκος <nikos.gr33k@gmail.com> - 2013-09-25 16:14 +0300
    Re: Referrer key missing form os.environ dictionary? Chris “Kwpolska” Warrick <kwpolska@gmail.com> - 2013-09-25 16:01 +0200
      Re: Referrer key missing form os.environ dictionary? Νίκος <nikos.gr33k@gmail.com> - 2013-09-25 17:04 +0300
        Re: Referrer key missing form os.environ dictionary? John Gordon <gordon@panix.com> - 2013-09-25 14:26 +0000
          Re: Referrer key missing form os.environ dictionary? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-09-25 15:04 +0000
            Re: Referrer key missing form os.environ dictionary? Νίκος <nikos.gr33k@gmail.com> - 2013-09-25 18:16 +0300
              Re: Referrer key missing form os.environ dictionary? Tim Chase <python.list@tim.thechases.com> - 2013-09-25 10:42 -0500
              Re: Referrer key missing form os.environ dictionary? Denis McMahon <denismfmcmahon@gmail.com> - 2013-09-25 21:17 +0000
          Re: Referrer key missing form os.environ dictionary? Ned Batchelder <ned@nedbatchelder.com> - 2013-09-25 11:09 -0400
        Re: Referrer key missing form os.environ dictionary? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-09-25 14:52 +0000
          Re: Referrer key missing form os.environ dictionary? Νίκος <nikos.gr33k@gmail.com> - 2013-09-25 18:02 +0300
            Re: Referrer key missing form os.environ dictionary? Xaxa Urtiz <urtizvereaxaxa@gmail.com> - 2013-09-25 08:09 -0700
            Re: Referrer key missing form os.environ dictionary? Tim Chase <python.list@tim.thechases.com> - 2013-09-25 10:14 -0500
              Re: Referrer key missing form os.environ dictionary? Νίκος <nikos.gr33k@gmail.com> - 2013-09-25 18:35 +0300
          Re: Referrer key missing form os.environ dictionary? Grant Edwards <invalid@invalid.invalid> - 2013-09-25 15:18 +0000
            Re: Referrer key missing form os.environ dictionary? Νίκος <nikos.gr33k@gmail.com> - 2013-09-25 18:41 +0300
              Re: Referrer key missing form os.environ dictionary? John Gordon <gordon@panix.com> - 2013-09-25 16:27 +0000
              Re: Referrer key missing form os.environ dictionary? Grant Edwards <invalid@invalid.invalid> - 2013-09-25 18:37 +0000
            Re: Referrer key missing form os.environ dictionary? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-09-26 00:11 +0000
              Re: Referrer key missing form os.environ dictionary? Terry Reedy <tjreedy@udel.edu> - 2013-09-25 21:32 -0400
              Re: Referrer key missing form os.environ dictionary? Chris Angelico <rosuav@gmail.com> - 2013-09-26 13:25 +1000
                Re: Referrer key missing form os.environ dictionary? Grant Edwards <invalid@invalid.invalid> - 2013-09-26 14:16 +0000
              Re: Referrer key missing form os.environ dictionary? Robert Kern <robert.kern@gmail.com> - 2013-09-26 11:15 +0100
              Re: Referrer key missing form os.environ dictionary? Grant Edwards <invalid@invalid.invalid> - 2013-09-26 14:11 +0000
                Re: Referrer key missing form os.environ dictionary? Chris Angelico <rosuav@gmail.com> - 2013-09-27 01:17 +1000
                  Re: Referrer key missing form os.environ dictionary? Grant Edwards <invalid@invalid.invalid> - 2013-09-26 23:27 +0000
                    Re: Referrer key missing form os.environ dictionary? Chris Angelico <rosuav@gmail.com> - 2013-09-27 16:01 +1000
      Re: Referrer key missing form os.environ dictionary? Νίκος <nikos.gr33k@gmail.com> - 2013-09-25 17:07 +0300
        Re: Referrer key missing form os.environ dictionary? Robert Kern <robert.kern@gmail.com> - 2013-09-25 15:26 +0100
        Re: Referrer key missing form os.environ dictionary? John Gordon <gordon@panix.com> - 2013-09-25 14:30 +0000
        Re: Referrer key missing form os.environ dictionary? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-09-25 14:46 +0000
        Re: Referrer key missing form os.environ dictionary? Piet van Oostrum <piet@vanoostrum.org> - 2013-09-25 15:59 -0400
    Re: Referrer key missing form os.environ dictionary? John Gordon <gordon@panix.com> - 2013-09-25 14:18 +0000
      Re: Referrer key missing form os.environ dictionary? Tim Chase <python.list@tim.thechases.com> - 2013-09-25 09:58 -0500

Page 2 of 2 — ← Prev page 1 [2]

#54773

On 9/25/2013 8:11 PM, Steven D'Aprano wrote:
> On Wed, 25 Sep 2013 15:18:41 +0000, Grant Edwards wrote:
>
>>> The Referer is not an environment variable.
>>
>> It is when you're writing a CGI app.
>>
>>> How would your shell know what URL you were just browsing?
>>
>> Because the HTTP server sets those environment variables before invoking
>> the CGI app.
>
>
> I stand corrected.
>
>
> That's a pretty shitty design though, isn't it? Communicating via
> environment variables. What is this, 1998? :-)

1993 https://en.wikipedia.org/wiki/Common_Gateway_Interface

> Mind you, I'm not sure what other alternatives exist.

Send a series of lines with the same info over an input channel, as was 
done 3 years later for FastCGI. Since CGI uses stdout for the finished 
product, it could have used stdin for the input. Using a serial channel 
does put more burden on the page server to parse the input. But is 
allows it to be on a different machine.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#54776

On Thu, Sep 26, 2013 at 11:32 AM, Terry Reedy <tjreedy@udel.edu> wrote:
> Since CGI uses stdout for the finished product, it could have used stdin for
> the input.

Haven't used CGI in years, but I thought POST data came on stdin?

ChrisA

[toc] | [prev] | [next] | [standalone]

#54823

On 2013-09-26, Chris Angelico <rosuav@gmail.com> wrote:
> On Thu, Sep 26, 2013 at 11:32 AM, Terry Reedy <tjreedy@udel.edu> wrote:
>> Since CGI uses stdout for the finished product, it could have used stdin for
>> the input.
>
> Haven't used CGI in years, but I thought POST data came on stdin?

Yes.  The user data is read via stdin.  The stuff passed via the
environemnt "dictionary" is meta-data (stuff from the HTTP request
headers, and stuff synthesized by the HTTP server).

-- 
Grant Edwards               grant.b.edwards        Yow! !  Now I understand
                                  at               advanced MICROBIOLOGY and
                              gmail.com            th' new TAX REFORM laws!!

[toc] | [prev] | [next] | [standalone]

#54805

On 2013-09-26 04:25, Chris Angelico wrote:
> On Thu, Sep 26, 2013 at 11:32 AM, Terry Reedy <tjreedy@udel.edu> wrote:
>> Since CGI uses stdout for the finished product, it could have used stdin for
>> the input.
>
> Haven't used CGI in years, but I thought POST data came on stdin?

You could just put the whole HTTP request, headers and body, through stdin and 
just make the program parse them apart.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#54822

On 2013-09-26, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> On Wed, 25 Sep 2013 15:18:41 +0000, Grant Edwards wrote:
>
>>> The Referer is not an environment variable.
>> 
>> It is when you're writing a CGI app.
>> 
>>> How would your shell know what URL you were just browsing?
>> 
>> Because the HTTP server sets those environment variables before invoking
>> the CGI app.
>
> I stand corrected.
>
> That's a pretty shitty design though, isn't it?

On a Unix system when you invoke a program, you "pass" it four things:

 1) A dictionary where keys/values are both strings [enviornment variables]
 2) A list of strings [command line args]
 3) A set of open file descriptors. 
 4) The current working directory. 

You can provide input values to the program through any of these.

For interactive programs, 2 and 3 are the most convenient. For
programs intended to be invoked non-interactively via another program
the first option can be very elegent and versatile -- but it does make
use of the program interactively rather awkward.  Semantically,
passing values to a program via environment variables is very similar
to keyword arguments to a Python function, while command line
arguments are like positional arguments to a Python function.
 
> Communicating via environment variables. What is this, 1998? :-)
>
> Mind you, I'm not sure what other alternatives exist.

Command line arguments, open file descriptors, or files in the CWD. 
All are more difficult to use programmatically than environment
variables.

-- 
Grant Edwards               grant.b.edwards        Yow! I guess you guys got
                                  at               BIG MUSCLES from doing too
                              gmail.com            much STUDYING!

[toc] | [prev] | [next] | [standalone]

#54833

On Fri, Sep 27, 2013 at 12:11 AM, Grant Edwards <invalid@invalid.invalid> wrote:
> On a Unix system when you invoke a program, you "pass" it four things:
>
>  1) A dictionary where keys/values are both strings [enviornment variables]
>  2) A list of strings [command line args]
>  3) A set of open file descriptors.
>  4) The current working directory.
>
> You can provide input values to the program through any of these.
>
> For interactive programs, 2 and 3 are the most convenient.

Hrm, not sure about #3 for interactive programs, unless you
specifically mean the three standard streams. With most Unix shells,
you should be able to set environment variables:

PGUSER=fred PGPASSWORD=secret psql

Not as convenient as #2, but far easier than passing an open file
descriptor. Of course, passing file descriptors around is pretty easy
programmatically, but when you say "interactive" I assume you're
talking also about an interactive shell.

GUI interactions of course follow their own rules completely. In most
systems, it's really easy to invoke an application with one argument,
a file name; it's generally much harder to customize the arguments at
the keyboard. OS/2 had a facility for doing that. You just put square
brackets into the configured args and it'd turn it into a prompt:

--foo=bar --mode=[Choose mode, 1-3:] %*

You'd get a nice little popup with the prompt you specified, and
whatever you type gets put into the args. Haven't seen that in any
other system - at least, not as conveniently.

ChrisA

[toc] | [prev] | [next] | [standalone]

#54856

On 2013-09-26, Chris Angelico <rosuav@gmail.com> wrote:
> On Fri, Sep 27, 2013 at 12:11 AM, Grant Edwards <invalid@invalid.invalid> wrote:
>> On a Unix system when you invoke a program, you "pass" it four things:
>>
>>  1) A dictionary where keys/values are both strings [enviornment variables]
>>  2) A list of strings [command line args]
>>  3) A set of open file descriptors.
>>  4) The current working directory.
>>
>> You can provide input values to the program through any of these.
>>
>> For interactive programs, 2 and 3 are the most convenient.
>
> Hrm, not sure about #3 for interactive programs, unless you
> specifically mean the three standard streams.

Yes, that's what I mean: shell redirection/pipelines.  Hooking stuff
to stdin is pretty much the only example of this you'll ever see in
the wild for input data:

 foo < inputdata
 
 bar | foo
 
 foo <<EOF
 this is input 
 data that is passed
 to program foo using
 file descriptor 0
 EOF 

> With most Unix shells, you should be able to set environment
> variables:
>
> PGUSER=fred PGPASSWORD=secret psql

Yep, that's rather rarely used.  The syntax/sematics for values passed
that way is very limited compared to the syntax/semantics that can be
used for command line arguments, so the latter is much more versatile.

> Not as convenient as #2, but far easier than passing an open file
> descriptor. Of course, passing file descriptors around is pretty easy
> programmatically, but when you say "interactive" I assume you're
> talking also about an interactive shell.

I probably should have said "stdin", but in theory you can pass data
in via multiple file descriptors.  Nobody does that except people
cooking up obscure examples for advanced shell scripting guides...
 
> GUI interactions of course follow their own rules completely. In most
> systems, it's really easy to invoke an application with one argument,
> a file name; it's generally much harder to customize the arguments at
> the keyboard. OS/2 had a facility for doing that. You just put square
> brackets into the configured args and it'd turn it into a prompt:
>
> --foo=bar --mode=[Choose mode, 1-3:] %*
>
> You'd get a nice little popup with the prompt you specified, and
> whatever you type gets put into the args. Haven't seen that in any
> other system - at least, not as conveniently.

-- 
Grant Edwards               grant.b.edwards        Yow! I request a weekend in
                                  at               Havana with Phil Silvers!
                              gmail.com            

[toc] | [prev] | [next] | [standalone]

#54864

On Fri, Sep 27, 2013 at 9:27 AM, Grant Edwards <invalid@invalid.invalid> wrote:
> I probably should have said "stdin", but in theory you can pass data
> in via multiple file descriptors.  Nobody does that except people
> cooking up obscure examples for advanced shell scripting guides...

Yep. With that variant, I agree that it's more common than env vars
(apart from standard ones - *heaps* of programs are affected by stuff
like LANG or HOME or PATH, but they're not passed at the command
line); what your description put me in mind of, though, was the
special handling sometimes done between two programs, like when the
Unix program loader finds a #! and exec's an interpreter to process
the script - for security, it has to pass the fd, not the file name,
to the interpreter. But that's really esoteric!

Of course, whenever you fork without execing (execking?), you can pass
(or share) file descriptors easily. Very common, but not exactly
command-line stuff.

ChrisA

[toc] | [prev] | [next] | [standalone]

#54733

Στις 25/9/2013 5:01 μμ, ο/η Chris “Kwpolska” Warrick έγραψε:
> On Wed, Sep 25, 2013 at 2:45 PM, Νίκος <nikos.gr33k@gmail.com> wrote:
>> Hello, i decided am ong other os.environ variables to also grab the
>> 'HTTP_REFERER' fiel but when i try to run my script i was seeing a KeyError
>> complaining that 'HTTP_REFERER' didnt exist.
>>
>> So, to see what existed in the os.environ dictionary i issues a print(
>> os.environ ) to see all available keys and their values:
>
> The Referer header is not mandatory by any means.  Your client
> probably does not send it.
>
But one other problem appeared too:

nikos@superhost.gr [~/www/cgi-bin]# python metrites.py
   File "metrites.py", line 27
     host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 
'UnKnown Host'
        ^
SyntaxError: invalid syntax


i dont see anything wrong with that line, and the carret is actually 
pointing to the "host".

[toc] | [prev] | [next] | [standalone]

#54736

On 2013-09-25 15:07, Νίκος wrote:
> Στις 25/9/2013 5:01 μμ, ο/η Chris “Kwpolska” Warrick έγραψε:
>> On Wed, Sep 25, 2013 at 2:45 PM, Νίκος <nikos.gr33k@gmail.com> wrote:
>>> Hello, i decided am ong other os.environ variables to also grab the
>>> 'HTTP_REFERER' fiel but when i try to run my script i was seeing a KeyError
>>> complaining that 'HTTP_REFERER' didnt exist.
>>>
>>> So, to see what existed in the os.environ dictionary i issues a print(
>>> os.environ ) to see all available keys and their values:
>>
>> The Referer header is not mandatory by any means.  Your client
>> probably does not send it.
>>
> But one other problem appeared too:
>
> nikos@superhost.gr [~/www/cgi-bin]# python metrites.py
>    File "metrites.py", line 27
>      host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnKnown Host'
>         ^
> SyntaxError: invalid syntax
>
>
> i dont see anything wrong with that line, and the carret is actually pointing to
> the "host".

As has been explained to you before, SyntaxErrors just point to the place where 
the parser saw something unexpected, not the exact point where you made the 
error. It is not omniscient. If you don't see the problem on the reported line, 
you have probably left off a closing bracket or something similar in a previous 
line. Look back a few lines.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [next] | [standalone]

#54737

In <l1uqna$qi5$2@dont-email.me> =?UTF-8?B?zp3Or866zr/Pgg==?= <nikos.gr33k@gmail.com> writes:

> But one other problem appeared too:

> nikos@superhost.gr [~/www/cgi-bin]# python metrites.py
>    File "metrites.py", line 27
>      host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 
> 'UnKnown Host'
>         ^
> SyntaxError: invalid syntax

> i dont see anything wrong with that line, and the carret is actually 
> pointing to the "host".

Your post has the code broken up into two separate lines.  Is that how
the code actually appears?  If so, then that's the problem.  You can't
arbitrarily break a statement into separate lines.

If not, then you'll have to post more of the code; the problem isn't on
this line.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon@panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"

[toc] | [prev] | [next] | [standalone]

#54738

On Wed, 25 Sep 2013 17:07:38 +0300, Νίκος wrote:

> nikos@superhost.gr [~/www/cgi-bin]# python metrites.py
>    File "metrites.py", line 27
>      host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
> 'UnKnown Host'
>         ^
> SyntaxError: invalid syntax
> 
> 
> i dont see anything wrong with that line, and the carret is actually
> pointing to the "host".

If the caret is pointing to "host", then the syntax error isn't 
discovered until that point. That means the actual syntax error occurs 
before that point, probably on the previous line. 

How long have you been doing web development with Python? Six months? A 
year? You're not a beginner any more. If you can't solve syntax errors by 
yourself by now, it's probably time to give up and find a job more suited 
to your skills.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#54755

Νίκος <nikos.gr33k@gmail.com> writes:

> Στις 25/9/2013 5:01 μμ, ο/η Chris “Kwpolska” Warrick έγραψε:
>> On Wed, Sep 25, 2013 at 2:45 PM, Νίκος <nikos.gr33k@gmail.com> wrote:
>>> Hello, i decided am ong other os.environ variables to also grab the
>>> 'HTTP_REFERER' fiel but when i try to run my script i was seeing a KeyError
>>> complaining that 'HTTP_REFERER' didnt exist.
>>>
>>> So, to see what existed in the os.environ dictionary i issues a print(
>>> os.environ ) to see all available keys and their values:
>>
>> The Referer header is not mandatory by any means.  Your client
>> probably does not send it.
>>
> But one other problem appeared too:
>
> nikos@superhost.gr [~/www/cgi-bin]# python metrites.py
>   File "metrites.py", line 27
>     host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
> UnKnown Host'
>        ^
> SyntaxError: invalid syntax
>
>
> i dont see anything wrong with that line, and the carret is actually
> pointing to the "host".

There is an apostrophe (') missing before UnKnown.

  host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnKnown Host'
-- 
Piet van Oostrum <piet@vanoostrum.org>
WWW: http://pietvanoostrum.com/
PGP key: [8DAE142BE17999C4]

[toc] | [prev] | [next] | [standalone]

#54734

In <l1ulsv$4mv$1@dont-email.me> =?UTF-8?B?zp3Or866zr/Pgg==?= <nikos.gr33k@gmail.com> writes:

> referrer = os.environ['HTTP_REFERER']

> Do i miss something? its a suprise to me that the environ dictioanry has 
> almost anythign but a referrer key.

HTTP_REFERER is used to indicate the URL containing the link that led
the user to the current URL.  So for example if the user is viewing
foo.html and then clicks a link on that page which leads to bar.html, the
HTTP_REFERER for that request would be foo.html.

However, if the user did not arrive from another page, then HTTP_REFERER
will be missing.  This happens when the user types the web address directly
into their browser, or clicks on a bookmark, or many other ways.

Also, obviously, it's up to the browser to truthfully report HTTP_REFERER;
the server itself has no idea what page you were on previously.  What
browser are you using?

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon@panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"

[toc] | [prev] | [next] | [standalone]

#54740

On 2013-09-25 14:18, John Gordon wrote:
> However, if the user did not arrive from another page, then
> HTTP_REFERER will be missing.  This happens when the user types the
> web address directly into their browser, or clicks on a bookmark,
> or many other ways.
> 
> Also, obviously, it's up to the browser to truthfully report
> HTTP_REFERER;

There are browser plugins that allow blocking or manually-overriding
the outbound refer[r]er header which help mitigate data leakage such
as search-engine query strings or work around website limitations.  So
server-side code should always assume that the HTTP_REFERER header can
be absent or easily be spoofed, treating it as a hint, not absolute
truth.

-tkc

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.python

csiph-web