Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=psPO14Oc3QkKv7A+337l6o6ajBeKQ2ZX7I1m8DBlOMLza/T51VX+hdeUM+/r3Jt37G 120GmQ+OOX51rHF6/ZL6ZomqGiDyDTY8y1vzbv245h91uXitnNPVzOYGtitXJNiz5iQb WF9qkri+BC79oZGdT8YICVq4AQmkzChSimldE=
MIME-Version: 1.0
In-Reply-To: <D7wpp.14226$sy5.2295@newsfe22.iad>
References: <mailman.65.1302065189.9059.python-list@python.org> <t7Vmp.7632$zn.3197@newsfe19.iad> <mailman.72.1302078388.9059.python-list@python.org> <4d9c5ca5$0$29991$c3e8da3$5496439d@news.astraweb.com> <87tyebf3r3.fsf_-_@benfinney.id.au> <4d9d6a4a$0$29992$c3e8da3$5496439d@news.astraweb.com> <6Nknp.10080$lx3.7480@newsfe02.iad> <BANLkTi=-9_F7ZQ8S23WCxGSWP-5mSt-LMw@mail.gmail.com> <mailman.118.1302200288.9059.python-list@python.org> <Ynonp.2277$YL5.1440@newsfe05.iad> <4d9e6551$0$29991$c3e8da3$5496439d@news.astraweb.com> <Z2ynp.10498$0s5.3031@newsfe17.iad> <4da0f1e1$0$29986$c3e8da3$5496439d@news.astraweb.com> <cArop.17646$sS4.7406@newsfe11.iad> <mailman.219.1302542844.9059.python-list@python.org> <sRUop.2627$7N3.197@newsfe10.iad> <mailman.279.1302637061.9059.python-list@python.org> <D7wpp.14226$sy5.2295@newsfe22.iad>
Date: Thu, 14 Apr 2011 19:15:05 +1000
Subject: Re: [OT] Free software versus software idea patents
From: Chris Angelico <rosuav@gmail.com>
To: python-list@python.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.353.1302772509.9059.python-list@python.org>
Lines: 85
NNTP-Posting-Host: 82.94.164.166
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:3190

On Thu, Apr 14, 2011 at 4:04 PM, harrismh777 <harrismh777@charter.net> wrot=
e:
>
> =A0 =A0How many web crawlers have you built? Are there any web programmer=
s out
> there who need a web bot to hit multiple sites zillions of times a month
> from different places on earth to 'up' the number of hits for economic
> reasons? I've seen my share of this.

A well-behaved spider will (a) have a UA that identifies itself (as a
bot, and preferably as itself - eg "GoogleBot", etc - some even go so
far as to include a URL for more info), and (b) start by fetching
/robots.txt before they go any further. Servers can recognize
properly-built crawlers. And improperly-built crawlers, deliberately
trying to hammer a server to lie about browser stats? Seriously, do
you think people actually care THAT much?

> =A0 =A0How mamy times have you altered the identity of your web browser s=
o that
> the web site would 'work'? You know, stupid messages from the server that
> say, "We only support IE 6+, upgrade your browser...", =A0so you tell it
> you're using IE 6 and, well no problem.

Yep. Which means that the figures will always be skewed toward IE a
bit. But it's a lot less than you might think; most people don't leave
UA switchers active all the time, and the number of web sites that
require them is dropping. It's true that UA switching will tip toward
IE (I've never seen a site where you have to pretend to be Google
Chrome), but the epidemology is, I believe, not all that high.

> =A0 =A0Web site data is bogus. It assumes even distributions... it assume=
s even
> usage of the site from all surfers, it assumes no web crawlers and no bot=
s,
> it assumes no browser identity tampering, and it assumes that there aren'=
t
> those who for economic reasons are not inflating the numbers deliberately
> (no, really??) from world-owned bot farms.

Even distributions of what?

1) Assuming nothing, it merely gives data. About one site. That's why
overall "browser marketshare" stats have to be done by averaging
multiple sites.

2) Web crawlers - see above. If you've ever looked at AWStats or
Webalizer or *insert stats engine here*, you'll have seen that it will
identify them. AWStats goes a bit further and will identify "viewed
traffic" and "not viewed traffic" even if it's unable to identify the
specific bot.

3) Yes, it assumes no UA switchers, obviously. It's just based on
headers. But I reckon you could easily identify someone who's using a
switcher, based on other headers - for instance, I doubt very much
that IE6 will send "Accept-Encoding: gzip,deflate,sdch" (which my
Chrome does).

4) Assumes people aren't deliberately fiddling the figures. Yeah, that
would be correct. We're in the realm of conspiracy theories here...
does anyone seriously think that browser stats are THAT important that
they'd go to multiple web servers with deceitful hits? Not forgetting
that they'd have to mix up the IPs, make plausible "browsing sessions"
(with referers and image retrieval and so on), vary the date/times,
etc, etc, etc, etc... and generate enough hits to make a reasonable
dent in the figures.

> =A0 =A0There is no reliable way to measure free software usage. But, ther=
e sure
> is a lot of posturing going on in the market place ... =A0wonder why?

Sure, and there's no reliable way to measure non-free software usage
either. What's the difference? You could count sales of Microsoft
Office, and you could count downloads of Open Office. Neither is any
more accurate than the other; although I think the 24-hour figures for
Firefox 4 / IE 9 downloads are fairly indicative, since people can't
get them off their respective OS install CDs. And this isn't
restricted to electronica either. Which is more popular, Coca-Cola or
Pepsi? Do more people vote Liberal or Labour, Republican or Democrat,
Whig or Tory?

Statisticking is a huge science. Most of it involves figuring out
what's important - anyone can get data, but getting useful information
out of the data takes some work.

Chris Angelico