Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Tue, 03 Feb 2015 18:02:52 -0500
From: Dave Angel <davea@davea.name>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Problem while reading files from hdfs using python
References: <19c7923f-2f26-4752-ac2b-1b6dfc093631@googlegroups.com>
In-Reply-To: <19c7923f-2f26-4752-ac2b-1b6dfc093631@googlegroups.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.18448.1423004594.18130.python-list@python.org>
Lines: 77
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:85183

On 01/25/2015 03:23 PM, Shalini Ravishankar wrote:
> Hello Everyone,
>
> I am trying to read(open) and write files in hdfs inside a python script. But having error.

Please copy/paste the full error traceback.

> Can someone tell me what is wrong here.
>
> Code (full): sample.py
>
>      #!/usr/bin/python
>
>
>      from subprocess import Popen, PIPE
>
>      print "Before Loop"
>
>      cat = Popen(["hadoop", "fs", "-cat", "./sample.txt"],
>                  stdout=PIPE)

I don't know anything about hadoop, and when you run it separately, you 
used different parameters.  So you can do a lot towards testing it yourself.

Start by running hadoop fs -cat ...  from shell to see whether it 
displays anything.  You should be able to use exactly the same arguments 
as you use in the Popen call.

Then if that seems to work as you expect, comment out your 'put' code 
below, and add some prints to the loop.  Does that look reasonable?

At that point, if both look reasonable, then try the inverse.  Write 
some known data to the 'put' command, and see if it makes it into the 
appropriate file.  Once again, you should  be able to also test the 
program parameters and behavior from the shell, typing manually into stdin.


>      put = Popen(["hadoop", "fs", "-put", "-", "./modifiedfile.txt"],
>                  stdin=PIPE)
>      for line in cat.stdout:
>          line += "Blah"
>          print line
>          put.stdin.write(line)
>
>      cat.stdout.close()
>      cat.wait()
>      put.stdin.close()
>      put.wait()
>
> When I execute :
>
>      hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.5.1.jar -file ./sample.py -mapper './sample.py' -input sample.txt -output fileRead
>
> It executes properly I couldn't find the file which supposed to create in hdfs modifiedfile
>
> And When I execute :
>
>       hadoop fs -getmerge ./fileRead/ file.txt
>
> Inside the file.txt, I got :
>
>      Before Loop	
>      Before Loop
>
> Can someone please tell me what I am doing wrong here ?? I dont think it reads from the sample.txt
>
> I would really appreciate the help.
>
>
> --
> Thanks & Regards,
> Shalini Ravishankar.
>


-- 
DaveA