Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Joaquin Alzola <Joaquin.Alzola@lebara.com>
Newsgroups: comp.lang.python
Subject: RE: Review Request of Python Code
Date: Thu, 10 Mar 2016 19:12:37 +0000
Lines: 150
Message-ID: <mailman.148.1457638025.15725.python-list@python.org>
References: <f0973a0d-62ba-402b-ab23-cb68bdd15323@googlegroups.com> <af65a7a6-3179-4bca-9022-ae0d2ec61a11@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Thread-Topic: Review Request of Python Code
Thread-Index: AQHRevjyQX/WRcnrAE6v6aQJ20neAJ9TCosQ
In-Reply-To: <af65a7a6-3179-4bca-9022-ae0d2ec61a11@googlegroups.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
spamdiagnosticoutput: 1:23
spamdiagnosticmetadata: NSPM
X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Mar 2016 19:12:37.1597 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: d7093539-83cd-4991-b1b3-aacef74cf097
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR07MB1495
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref: csiph.com comp.lang.python:104544

SQL doesn't allow decimal numbers for LIMIT.
Use decimal numbers it still work but is the proper way.

Then clean up a bit your code and remove the commented lines #

-----Original Message-----
From: Python-list [mailto:python-list-bounces+joaquin.alzola=3Dlebara.com@p=
ython.org] On Behalf Of subhabangalore@gmail.com
Sent: 10 March 2016 18:12
To: python-list@python.org
Subject: Re: Review Request of Python Code

On Wednesday, March 9, 2016 at 9:49:17 AM UTC+5:30, subhaba...@gmail.com wr=
ote:
> Dear Group,
>
> I am trying to write a code for pulling data from MySQL at the backend an=
d annotating words and trying to put the results as separated sentences wit=
h each line. The code is generally running fine but I am feeling it may be =
better in the end of giving out sentences, and for small data sets it is ok=
ay but with 50,000 news articles it is performing dead slow. I am using Pyt=
hon2.7.11 on Windows 7 with 8GB RAM.
>
> I am trying to copy the code here, for your kind review.
>
> import MySQLdb
> import nltk
> def sql_connect_NewTest1():
>     db =3D MySQLdb.connect(host=3D"localhost",
>                      user=3D"*****",
>                      passwd=3D"*****",
>                      db=3D"abcd_efgh")
>     cur =3D db.cursor()
>     #cur.execute("SELECT * FROM newsinput limit 0,50000;") #REPORTING RUN=
TIME ERROR
>     cur.execute("SELECT * FROM newsinput limit 0,50;")
>     dict_open=3Dopen("/python27/NewTotalTag.txt","r") #OPENING THE DICTIO=
NARY FILE
>     dict_read=3Ddict_open.read()
>     dict_word=3Ddict_read.split()
>     a4=3Ddict_word #Assignment for code.
>     list1=3D[]
>     flist1=3D[]
>     nlist=3D[]
>     for row in cur.fetchall():
>         #print row[2]
>         var1=3Drow[3]
>         #print var1 #Printing lines
>         #var2=3Dlen(var1) # Length of file
>         var3=3Dvar1.split(".") #SPLITTING INTO LINES
>         #print var3 #Printing The Lines
>         #list1.append(var1)
>         var4=3Dlen(var3) #Number of all lines
>         #print "No",var4
>         for line in var3:
>             #print line
>             #flist1.append(line)
>             linew=3Dline.split()
>             for word in linew:
>                 if word in a4:
>                     windex=3Da4.index(word)
>                     windex1=3Dwindex+1
>                     word1=3Da4[windex1]
>                     word2=3Dword+"/"+word1
>                     nlist.append(word2)
>                     #print list1
>                     #print nlist
>                 elif word not in a4:
>                     word3=3Dword+"/"+"NA"
>                     nlist.append(word3)
>                     #print list1
>                     #print nlist
>                 else:
>                     print "None"
>
>     #print "###",flist1
>     #print len(flist1)
>     #db.close()
>     #print nlist
>     lol =3D lambda lst, sz: [lst[i:i+sz] for i in range(0, len(lst), sz)]=
 #TRYING TO SPLIT THE RESULTS AS SENTENCES
>     nlist1=3Dlol(nlist,7)
>     #print nlist1
>     for i in nlist1:
>         string1=3D" ".join(i)
>         print i
>         #print string1
>
>
> Thanks in Advance.

***************************************************************************=
*
Dear Group,

Thank you all, for your kind time and all suggestions in helping me.

Thank you Steve for writing the whole code. It is working full and fine. Bu=
t speed is still an issue. We need to speed up.

Inada I tried to change to
cur =3D db.cursor(MySQLdb.cursors.SSCursor) but my System Admin said that m=
ay not be an issue.

Freidrich, my problem is I have a big text repository of .txt files in MySQ=
L in the backend. I have another list of words with their possible tags. Th=
e tags are not conventional Parts of Speech(PoS) tags,  and bit defined by =
others.
The code is expected to read each file and its each line.
On reading each line it will scan the list for appropriate tag, if it is fo=
und it would assign, else would assign NA.
The assignment should be in the format of /tag, so that if there is a strin=
g of n words, it should look like, w1/tag w2/tag w3/tag w4/tag ....wn/tag,

where tag may be tag in the list or NA as per the situation.

This format is taken because the files are expected to be tagged in Brown C=
orpus format. There is a Python Library named NLTK.
If I want to save my data for use with their models, I need some specificat=
ions. I want to use it as Tagged Corpus format.

Now the tagged data coming out in this format, should be one tagged sentenc=
es in each new line or a lattice.

They expect the data to be saved in .pos format but presently I am not doin=
g in this code, I may do that later.

Please let me know if I need to give any more information.

Matt, thank you for if...else suggestion, the data of NewTotalTag.txt is li=
ke a simple list of words with unconventional tags, like,

w1 tag1
w2 tag2
w3 tag3
...
...
w3  tag3

like that.

Regards,
Subhabrata


--
https://mail.python.org/mailman/listinfo/python-list
This email is confidential and may be subject to privilege. If you are not =
the intended recipient, please do not copy or disclose its content but cont=
act the sender immediately upon receipt.