mrjob v0.2.5 released

Path	csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!news.albasani.net!newsfeed.freenet.ag!news2.euro.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<jimmy@retzlaff.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.001
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'handled': 0.03; 'subject:released': 0.03; 'args': 0.07; 'feeds': 0.07; 'python': 0.07; 'url:pypi': 0.09; 'setup.py': 0.09; 'ssh': 0.09; 'tracker': 0.09; 'url:github': 0.09; 'url:packages': 0.09; 'scripts': 0.10; 'output': 0.12; '(issue': 0.16; 'elastic': 0.16; 'url:yelp': 0.16; 'input': 0.18; 'interpret': 0.19; 'tree': 0.20; 'to:2**1': 0.20; 'e.g.': 0.22; 'hourly': 0.23; 'set.': 0.23; 'sfxlen:2': 0.24; "what's": 0.24; 'specify': 0.25; '(e.g.': 0.26; 'received:209.85.161.46': 0.26; 'received:mail- fx0-f46.google.com': 0.26; 'environment': 0.26; 'message- id:@mail.gmail.com': 0.28; 'supports': 0.29; 'packages': 0.29; 'received:209.85.161': 0.29; 'error': 0.29; 'mode': 0.29; 'formats': 0.29; 'minimal': 0.29; 'variables': 0.29; 'skip:( 20': 0.31; 'to:addr:python-list': 0.32; 'source': 0.32; 'hosted': 0.33; '(for': 0.33; 'options': 0.34; 'file': 0.35; 'url:group': 0.35; 'put': 0.35; 'only)': 0.35; 'some': 0.37; 'received:209.85': 0.37; 'url:python': 0.37; 'skip:- 10': 0.37; 'run': 0.37; 'streaming': 0.38; 'received:google.com': 0.38; 'install': 0.38; 'url:google': 0.38; 'url:org': 0.38; 'set': 0.39; 'to:addr:python.org': 0.39; 'received:209': 0.39; 'works': 0.40; 'logs': 0.40; 'header:Received:5': 0.40; 'allows': 0.40; 'skip:h 20': 0.60; 'url:groups': 0.63; 'production': 0.65; 'skip:$ 10': 0.67; 'buy': 0.67; 'features:': 0.73; 'locally': 0.91; 'tunnel': 0.91
MIME-Version	1.0
From	Jimmy Retzlaff <jimmy@retzlaff.com>
Date	Sat, 30 Apr 2011 10:20:44 -0700
Subject	mrjob v0.2.5 released
To	python-announce@python.org, python-list@python.org
Content-Type	text/plain; charset=ISO-8859-1
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.12
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.1024.1304184062.9059.python-list@python.org> (permalink)
Lines	44
NNTP-Posting-Host	82.94.164.166
X-Trace	1304184063 news.xs4all.nl 32470 [::ffff:82.94.164.166]:41055
X-Complaints-To	abuse@xs4all.nl
Xref	x330-a1.tempe.blueboxinc.net comp.lang.python:4362

Show key headers only | View raw

What is mrjob?
-----------------------

mrjob is a Python package that helps you write and run Hadoop Streaming jobs.

mrjob fully supports Amazon's Elastic MapReduce (EMR) service, which
allows you to buy time on a Hadoop cluster on an hourly basis. It also
works with your own Hadoop cluster.

Some important features:

  * Run jobs on EMR, your own Hadoop cluster, or locally (for testing).
  * Write multi-step jobs (one map-reduce step feeds into the next)
  * Duplicate your production environment inside Hadoop
      * Upload your source tree and put it in your job's $PYTHONPATH
      * Run make and other setup scripts
      * Set environment variables (e.g. $TZ)
      * Easily install python packages from tarballs (EMR only)
      * Setup handled transparently by mrjob.conf config file
  * Automatically interpret error logs from EMR
  * SSH tunnel to hadoop job tracker on EMR
  * Minimal setup
      * To run on EMR, set $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY
      * To run on your Hadoop cluster, install simplejson and make
sure $HADOOP_HOME is set.

More info:

  * Install mrjob: python setup.py install
  * Documentation: http://packages.python.org/mrjob/
  * PyPI: http://pypi.python.org/pypi/mrjob
  * Discussion: http://groups.google.com/group/mrjob
  * Development is hosted at github: http://github.com/Yelp/mrjob


What's new?
-------------------

v0.2.5, 2011-04-29 -- Hadoop input and output formats
  * Added hadoop_input/output_format options
  * You can now specify a custom Hadoop streaming jar (hadoop_streaming_jar)
  * extra args to hadoop now come before -mapper/-reducer on EMR, so
    that e.g. -libjar will work (worked in hadoop mode since v0.2.2)
  * hadoop mode now supports s3n:// URIs (Issue #53)

Back to comp.lang.python | Previous | Next | Find similar

Thread

mrjob v0.2.5 released Jimmy Retzlaff <jimmy@retzlaff.com> - 2011-04-30 10:20 -0700

csiph-web