Path: csiph.com!usenet.pasdenom.info!news.albasani.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Wed, 02 Jul 2014 17:51:05 +0200
From: "F.R." <anthra.norell@bluewin.ch>
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Thunderbird/24.5.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: fixing an horrific formatted csv file.
References: <47e2e29d-b5c3-4aa6-abf9-3b1e46eb0dec@googlegroups.com> <mailman.11385.1404247829.18130.python-list@python.org> <0d3871c6-81d4-4168-9408-ad85299b0955@googlegroups.com> <mailman.11392.1404264061.18130.python-list@python.org> <a84826ea-4018-40bc-88c1-812be5417e6b@googlegroups.com>
In-Reply-To: <a84826ea-4018-40bc-88c1-812be5417e6b@googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.11411.1404316334.18130.python-list@python.org>
Lines: 90
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:73840

On 07/02/2014 11:13 AM, flebber wrote:
>>>>> TM = TX.Table_Maker (headings =
>> ('Meeting','Date','Race','Number','Name','Trainer','Location'))
>>>>> TM (race_table (your_csv_text)).write ()
> Where do I find TX? Found this mention in the list, was it available in pip by any name?
> https://mail.python.org/pipermail/python-list/2014-February/667464.html
>
> Sayth

I'd have to make it available. I proposed it some time ago and received 
a couple of suggestions in return. It is a modular transformation 
framework written entirely in python (2.7). It consists essentially of a 
base class "Transformer" that handles input and output in such a way 
that Transformer objects can be chained. It saved me from drowning an a 
horrible and growing tangle of hacks. Finding something usable I had 
previously done took time. Understanding how it worked took more time 
and adapting it took still more time, so that writing yet another hack 
from scratch was faster.
     A number of hacks I could quickly wrap into a Transformer object 
and so could start building a library of standard Transformers. The 
Table_Maker is one of them. The table making code is quite bad. It 
suffers from feature overload. I would clean it up for distribution.
     I'd be happy to distribute the base class and a few standard 
Translators, such as I use every day. (File Reader, File Writer, DB Run 
Command, DB Write, Table Maker, PDF To Text, Text To Lines, Lines To 
Text, Sort, Sort And Unique, etc.) Writing one's own Transformers is a 
breeze. Testing too, because a Transformer keeps its input and output 
and, in line with the system's design philosophy, does only its own 
single thing.
     A Chain is a list of Transformers that run in sequence. It is 
itself derived from Transformer and is a functional equivalent. So 
Chains nest. Fixing a Chain that nothing comes out of is a 
straightforward matter too. It will still have run up to the failing 
element. Chain.show () reveals the culprit as the first one to have no 
output.
     I am not up to date on distributing and would depend on qualified 
help on that.

Frederic


--------------------------------------------------------------------------------


A brief overview


The TX solution to your race table would be (TX is the name of the module):

     class Race_Table (TX.Transformer):
         '''
         In: CSV text
         Out: Tabular data (2-dimensional list)
         '''
         name = 'Race_Table'
         @TX.setup   # Checks timestamps to prevent needless reruns in 
the absence of new input
         def transform (self):
             for line in self.Input.data:
                 # See my post
             self.Output.take (output_table)

     Example file to file:
     >>> Race_Schedule_F2F = TX.Chain (TX.File_Reader (), Race_Table (), 
TX.List_To_CSV (delimiter = ';'), TX.File_Writer (terminal = out_file_name)
     >>> Race_Schedule_F2F (input_file_name)   # Does it all!

     Example web to database:
     >>> Race_Schedule_WWW2DB = TX.Chain (TX.WWW_Reader (), 
Race_Schedule_HTML_Reader (), Race_Table (), TX.DB_Writer (table_name = 
'horses'))
     >>> Race_Schedule_WWW2DB (url)   # Does is all! You'd have to write 
the Race_Schedule_HTML_Reader

     Verify your table:
     >>> Table_Viewer = TX.Chain (TX.Table_Maker (), TX.Table_Writer ())
     >>> Race_Schedule_WWW2DB.show_tree () # See which one should display
     Chain
     Chain[0] - WWW Reader
     Chain[1] - Race_Schedule_HTML_Reader
     Chain[2] - Race_Table
     Chain[3] - DB Writer
     >>> print Table_Viewer (Race_Schedule_WWW2DB[2]()) # All 
Transformers keep their data
     (Display of table)

     Verify database:
     >>> print Table_Viewer (TX.DB_Reader (table_name = 'horses')())
     (Display of database table)