Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #85183

Re: Problem while reading files from hdfs using python

Show key headers only | View raw

On 01/25/2015 03:23 PM, Shalini Ravishankar wrote:
> Hello Everyone,
>
> I am trying to read(open) and write files in hdfs inside a python script. But having error.

Please copy/paste the full error traceback.

> Can someone tell me what is wrong here.
>
> Code (full): sample.py
>
>      #!/usr/bin/python
>
>
>      from subprocess import Popen, PIPE
>
>      print "Before Loop"
>
>      cat = Popen(["hadoop", "fs", "-cat", "./sample.txt"],
>                  stdout=PIPE)

I don't know anything about hadoop, and when you run it separately, you 
used different parameters.  So you can do a lot towards testing it yourself.

Start by running hadoop fs -cat ...  from shell to see whether it 
displays anything.  You should be able to use exactly the same arguments 
as you use in the Popen call.

Then if that seems to work as you expect, comment out your 'put' code 
below, and add some prints to the loop.  Does that look reasonable?

At that point, if both look reasonable, then try the inverse.  Write 
some known data to the 'put' command, and see if it makes it into the 
appropriate file.  Once again, you should  be able to also test the 
program parameters and behavior from the shell, typing manually into stdin.


>      put = Popen(["hadoop", "fs", "-put", "-", "./modifiedfile.txt"],
>                  stdin=PIPE)
>      for line in cat.stdout:
>          line += "Blah"
>          print line
>          put.stdin.write(line)
>
>      cat.stdout.close()
>      cat.wait()
>      put.stdin.close()
>      put.wait()
>
> When I execute :
>
>      hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.5.1.jar -file ./sample.py -mapper './sample.py' -input sample.txt -output fileRead
>
> It executes properly I couldn't find the file which supposed to create in hdfs modifiedfile
>
> And When I execute :
>
>       hadoop fs -getmerge ./fileRead/ file.txt
>
> Inside the file.txt, I got :
>
>      Before Loop	
>      Before Loop
>
> Can someone please tell me what I am doing wrong here ?? I dont think it reads from the sample.txt
>
> I would really appreciate the help.
>
>
> --
> Thanks & Regards,
> Shalini Ravishankar.
>


-- 
DaveA

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Problem while reading files from hdfs using python Shalini Ravishankar <shalini.ravishankar@gmail.com> - 2015-01-25 12:23 -0800
  Re: Problem while reading files from hdfs using python Dave Angel <davea@davea.name> - 2015-02-03 18:02 -0500

csiph-web