Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #77270 > unrolled thread

Keeping python code and database in sync

Back to article view | Back to comp.lang.python

Contents

  Keeping python code and database in sync "Frank Millman" <frank@chagford.com> - 2014-08-29 14:42 +0200
    Re: Keeping python code and database in sync Rustom Mody <rustompmody@gmail.com> - 2014-08-29 05:55 -0700
      Re: Keeping python code and database in sync "Frank Millman" <frank@chagford.com> - 2014-08-29 15:19 +0200
    Re: Keeping python code and database in sync Roy Smith <roy@panix.com> - 2014-08-29 10:54 -0400
      Re: Keeping python code and database in sync Skip Montanaro <skip@pobox.com> - 2014-08-29 12:04 -0500
      suckitude classifications [was Re: Keeping python code and database in sync] Ethan Furman <ethan@stoneleaf.us> - 2014-08-29 12:02 -0700
      Re: Keeping python code and database in sync Ben Finney <ben+python@benfinney.id.au> - 2014-08-30 06:34 +1000
      Re: Keeping python code and database in sync Chris Angelico <rosuav@gmail.com> - 2014-08-30 08:31 +1000
      Re: suckitude classifications [was Re: Keeping python code and database in sync] Chris Angelico <rosuav@gmail.com> - 2014-08-30 08:38 +1000
        Re: suckitude classifications [was Re: Keeping python code and database in sync] Roy Smith <roy@panix.com> - 2014-08-29 19:19 -0400
          Re: suckitude classifications [was Re: Keeping python code and database in sync] Chris Angelico <rosuav@gmail.com> - 2014-08-30 09:38 +1000
      Re: Keeping python code and database in sync Skip Montanaro <skip.montanaro@gmail.com> - 2014-08-29 18:47 -0500
      Re: Keeping python code and database in sync Ethan Furman <ethan@stoneleaf.us> - 2014-08-29 16:54 -0700
      Re: Keeping python code and database in sync Skip Montanaro <skip@pobox.com> - 2014-08-29 21:14 -0500
      Re: Keeping python code and database in sync Chris Angelico <rosuav@gmail.com> - 2014-08-30 12:28 +1000
      Re: suckitude classifications {taken to absurdity} Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-08-30 11:13 -0400

#77270 — Keeping python code and database in sync

Hi all

Now that I have bitten the bullet and published my repository, I am forced 
to change my working practices (which is a good thing!).

The project is inherently database-driven. The python code expects to find 
certain tables and columns in the database. As I develop new features, I 
sometimes need to modify the database structure. In the bad old days (like 
yesterday) I would just make the modifications and carry on. Now I have to 
be aware that others may have downloaded the project, so I have to consider 
how to ensure that their database is kept up to date.

It is a simple matter to write a program that updates the database 
automatically. The question is, what should trigger such an update? My first 
thought is to use a version number - store a version number in the working 
directory, and have a matching number in the code. If someone downloads the 
latest version, the numbers will no longer match, and I can run the upgrade 
program.

The problem with that is that version numbers are usually reserved for 
tagged releases, but this could happen as the result of any commit in the 
current development cycle.

Any suggestions?

Frank Millman

[toc] | [next] | [standalone]

#77271

On Friday, August 29, 2014 6:12:06 PM UTC+5:30, Frank Millman wrote:
> Hi all

> Now that I have bitten the bullet and published my repository, I am forced 
> to change my working practices (which is a good thing!).

> The project is inherently database-driven. The python code expects to find 
> certain tables and columns in the database. As I develop new features, I 
> sometimes need to modify the database structure. In the bad old days (like 
> yesterday) I would just make the modifications and carry on. Now I have to 
> be aware that others may have downloaded the project, so I have to consider 
> how to ensure that their database is kept up to date.


There are tools like this
http://alembic.readthedocs.org/en/latest/

It may help to read that to avoid reinvention



> It is a simple matter to write a program that updates the database 
> automatically. The question is, what should trigger such an update? My first 
> thought is to use a version number - store a version number in the working 
> directory, and have a matching number in the code. If someone downloads the 
> latest version, the numbers will no longer match, and I can run the upgrade 
> program.

> The problem with that is that version numbers are usually reserved for 
> tagged releases, but this could happen as the result of any commit in the 
> current development cycle.


I dont think alembic can solve that.
Still it may help to study it

[toc] | [prev] | [next] | [standalone]

#77273

"Rustom Mody" <rustompmody@gmail.com> wrote in message 
news:1cdf6e52-e09b-40f1-8db1-db6cbbee9512@googlegroups.com...
> On Friday, August 29, 2014 6:12:06 PM UTC+5:30, Frank Millman wrote:
>> Hi all
>
>> Now that I have bitten the bullet and published my repository, I am 
>> forced
>> to change my working practices (which is a good thing!).
>
>> The project is inherently database-driven. The python code expects to 
>> find
>> certain tables and columns in the database. As I develop new features, I
>> sometimes need to modify the database structure. In the bad old days 
>> (like
>> yesterday) I would just make the modifications and carry on. Now I have 
>> to
>> be aware that others may have downloaded the project, so I have to 
>> consider
>> how to ensure that their database is kept up to date.
>
>
> There are tools like this
> http://alembic.readthedocs.org/en/latest/
>
> It may help to read that to avoid reinvention
>

Thanks for the link, Rustom.

I glanced at it, and it looks very powerful, but I will have to find the 
time to study it at leisure.

In the meantime, Chris' suggestion is simple to implement and adequate for 
my present needs, so I will run with that for now.

Frank

[toc] | [prev] | [next] | [standalone]

#77278

In article <mailman.13604.1409316126.18130.python-list@python.org>,
 "Frank Millman" <frank@chagford.com> wrote:

> The project is inherently database-driven. The python code expects to find 
> certain tables and columns in the database. As I develop new features, I 
> sometimes need to modify the database structure. In the bad old days (like 
> yesterday) I would just make the modifications and carry on. Now I have to 
> be aware that others may have downloaded the project, so I have to consider 
> how to ensure that their database is kept up to date.

Yeah, schema migration is an ugly problem.  There's a number of tools to 
help here, most of which reduce the suckitude, but don't eliminate it 
completely.  Some things you might want to look at:

* SQLAlchemy Migrate
* South (django-specific)
* yoyo-migrations
* alembic

Google for "python schema migration tools" and you'll probably find 
others.

[toc] | [prev] | [next] | [standalone]

#77282

On Fri, Aug 29, 2014 at 9:54 AM, Roy Smith <roy@panix.com> wrote:
> Yeah, schema migration is an ugly problem.

It's not really any worse than any other sort of complex data
structure change, is it? If your persistent data lived in a pickle
file, it would likely be as bad or worse.

> ... suckitude ...

Nice word. Let's use it more so my polly app will see it as a common
word and maybe offer it to me in a potential XKCD 936 password. :-)

suckitude suckitude suckitude suckitude suckitude suckitude suckitude

:-)

Skip

[toc] | [prev] | [next] | [standalone]

#77283 — suckitude classifications [was Re: Keeping python code and database in sync]

On 08/29/2014 10:04 AM, Skip Montanaro wrote:
> On Fri, Aug 29, 2014 at 9:54 AM, Roy Smith <roy@panix.com> wrote:
>> Yeah, schema migration is an ugly problem.
>
> It's not really any worse than any other sort of complex data
> structure change, is it? If your persistent data lived in a pickle
> file, it would likely be as bad or worse.
>
>> ... suckitude ...
>
> Nice word. Let's use it more so my polly app will see it as a common
> word and maybe offer it to me in a potential XKCD 936 password. :-)
>
> suckitude suckitude suckitude suckitude suckitude suckitude suckitude

Speaking of suckitude, we could classify technologies that way:

xml: major suckitude

rpc: no suckitude

python: negative suckitude

oh, and suckitude is neither cruft nor corpus !

Polly want a cracker?  ;)

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

#77285

Roy Smith <roy@panix.com> writes:

> Yeah, schema migration is an ugly problem. There's a number of tools
> to help here, most of which reduce the suckitude, but don't eliminate
> it completely. Some things you might want to look at:
>
> * SQLAlchemy Migrate
> * alembic

I can strongly recommend SQLAlchemy. It has several levels of working
with the RDBMS, and they all work well together; you can code primarily
to one API and occasionally use a different part, and it all works
together.

I've never used Alembic, but it is a migration tool built on SQLAlchemy.

> * South (django-specific)

It's worth noting the South is no longer developed as a separate
library:

    Please note that South is now end of lifed in favour of the new
    migrations framework in Django 1.7, which is based on South but with
    significant design improvements. South will not work with Django
    1.7; it supports only versions 1.4, 1.5 and 1.6.

    <URL:http://south.aeracode.org/>

-- 
 \         “I think Western civilization is more enlightened precisely |
  `\     because we have learned how to ignore our religious leaders.” |
_o__)                                                —Bill Maher, 2003 |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#77292

On Sat, Aug 30, 2014 at 3:04 AM, Skip Montanaro <skip@pobox.com> wrote:
> On Fri, Aug 29, 2014 at 9:54 AM, Roy Smith <roy@panix.com> wrote:
>> Yeah, schema migration is an ugly problem.
>
> It's not really any worse than any other sort of complex data
> structure change, is it? If your persistent data lived in a pickle
> file, it would likely be as bad or worse.

Well, correct. The problem isn't because it's in a database; the
problem is a consequence of persistent structured data that can get
out of sync.

It's easy to solve in a simple way that breaks on any sort of
confusion. It takes a bit more complexity (like the scheme I
suggested) to handle a few more cases. It takes a lot more complexity
(like the migration tools Roy listed) to cope with lots of awkward
cases (I suspect at least some of them will handle back-levelling,
which my scheme doesn't). And I doubt any of them is absolutely
perfect.

>> ... suckitude ...
>
> Nice word. Let's use it more so my polly app will see it as a common
> word and maybe offer it to me in a potential XKCD 936 password. :-)
>
> suckitude suckitude suckitude suckitude suckitude suckitude suckitude
>

Yeah, it's a great word. As a general rule, suckitude increases with
the square of design complexity and superlinearly with number of bugs.
I'm not sure how suckitude is affected by bugs, exactly; possibly O(N
log N), because each bug has a small probability of affecting another
bug.

ChrisA

[toc] | [prev] | [next] | [standalone]

#77293 — Re: suckitude classifications [was Re: Keeping python code and database in sync]

On Sat, Aug 30, 2014 at 5:02 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
> Speaking of suckitude, we could classify technologies that way:
>
> xml: major suckitude
>
> rpc: no suckitude
>
> python: negative suckitude

I disagree with your last two qualifications. RPC still sucks, just
not as much as some things do. And your last statement implies that
Python actually fixes other problems, which violates one of the
fundamental laws of physics: the Law of Conservation of Suckitude and
Frustration. It's possible to reduce suckitude in one system by
shifting it to another system, and it's possible to overall reduce
suckitude in the universe by engaging in a very frustrating job, but
merely combining two entities cannot actively reduce suckitude. Of
course, sometimes you can wrap the suckitude up in another layer, thus
reducing *apparent* suckitude, and this can make a system safer to
use; but maintenance on the wrapper layer will reveal that nothing has
been destroyed.

Hello, Polly!

ChrisA

[toc] | [prev] | [next] | [standalone]

#77298 — Re: suckitude classifications [was Re: Keeping python code and database in sync]

In article <mailman.13623.1409351910.18130.python-list@python.org>,
 Chris Angelico <rosuav@gmail.com> wrote:

> On Sat, Aug 30, 2014 at 5:02 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
> > Speaking of suckitude, we could classify technologies that way:
> >
> > xml: major suckitude
> >
> > rpc: no suckitude
> >
> > python: negative suckitude
> 
> I disagree with your last two qualifications. RPC still sucks, just
> not as much as some things do.

Are we talking the generic concept of Remote Procedure Calls, or the 
specific implementation of Sun RPC?

I'm surprised at the reaction to the word, as if it were something new.  
I thought the term has been in common parlance for many years.  Contrast 
with winitude.

Anyway, I think it is unfair to describe xml as major suckitude.  True, 
it is somewhat outdated, with more modern alternatives such as JSON, 
YAML, protobuffers, etc.  In its day, however, it was a breakthrough 
technology, even if only an incremental outgrowth of SGML.

[toc] | [prev] | [next] | [standalone]

#77300 — Re: suckitude classifications [was Re: Keeping python code and database in sync]

On Sat, Aug 30, 2014 at 9:19 AM, Roy Smith <roy@panix.com> wrote:
> In article <mailman.13623.1409351910.18130.python-list@python.org>,
>  Chris Angelico <rosuav@gmail.com> wrote:
>
>> On Sat, Aug 30, 2014 at 5:02 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
>> > Speaking of suckitude, we could classify technologies that way:
>> >
>> > xml: major suckitude
>> >
>> > rpc: no suckitude
>> >
>> > python: negative suckitude
>>
>> I disagree with your last two qualifications. RPC still sucks, just
>> not as much as some things do.
>
> Are we talking the generic concept of Remote Procedure Calls, or the
> specific implementation of Sun RPC?

I was thinking of XML-RPC, a particular protocol. Although the generic
concept of remotely calling something isn't entirely free of suckitude
- it's all very well in its way, but like everything else, it has its
flaws. In systems that completely hide the details and make it look
identical to local procedure calls, you have the same problem as
creating a Python property that does a lot of work ("wait, that call
actually goes out over the network??"), and in systems that make it
more clear that this is a network transaction, it's not really a
remote procedure call any more, it's a network action like any other.
But these are small nitpicks. Enough that I'd say "minor suckitude" or
"slight suckitude", but not "no suckitude".

> I'm surprised at the reaction to the word, as if it were something new.
> I thought the term has been in common parlance for many years.  Contrast
> with winitude.

It has. It's just that Skip's Polly has never heard the word before,
so we're teaching her.

> Anyway, I think it is unfair to describe xml as major suckitude.  True,
> it is somewhat outdated, with more modern alternatives such as JSON,
> YAML, protobuffers, etc.  In its day, however, it was a breakthrough
> technology, even if only an incremental outgrowth of SGML.

I've never used XML for anything that wasn't mandated by some other
end, and in every single one of those cases, JSON would have been a
much better fit for the data structure. XML is frequently shoehorned
into carrying array data like this:

<root>
<entry><heading>foo</heading><subentry>text</subentry><subentry>more
text</subentry><subentry>blah</subentry></entry>
<entry><heading>bar</heading><subentry>only one element</subentry></entry>
</root>

In JSON, this would be:

{"entry":[
    {"heading":"foo","subentry":["text","more text","blah"]},
    {"heading":"bar","subentry":["only one element"]}
]}

But in XML, there's no way to represent lists/arrays at all, and the
usual way of fitting them in doesn't allow you to distinguish
one-element lists from text strings (compare the heading entry).

So what's the value of XML for information storage? What's it being
compared against that makes it look good? And above all, why do people
still use it when JSON and other formats are available and well known?

ChrisA

[toc] | [prev] | [next] | [standalone]

#77303

[Multipart message — attachments visible in raw view] — view raw

On Aug 29, 2014 5:34 PM, "Chris Angelico" <rosuav@gmail.com> wrote:

> I'm not sure how suckitude is affected by bugs, exactly; possibly O(N
> log N), because each bug has a small probability of affecting another
> bug.

OTOH, bug fixes often have a fairly high probability of adding more bugs to
the system, especially if your test suite isn't up to snuff.

Skip

[toc] | [prev] | [next] | [standalone]

#77304

On 08/29/2014 04:47 PM, Skip Montanaro wrote:
> On Aug 29, 2014 5:34 PM, "Chris Angelico" wrote:
>
>> I'm not sure how suckitude is affected by bugs, exactly; possibly O(N
>> log N), because each bug has a small probability of affecting another
>> bug.
>
> OTOH, bug fixes often have a fairly high probability of adding more bugs to the system, especially if your test suite

Major suckitude !!  must be O(N**2) at least!

[Thus endeth my attempts to train Skip's Polly.  But I am curious -- if 'suckitude' is in immediate contact with 
punctuation such as just now, or at the end of a sentence, does it not count?  That would be suckitude indeed! ;) ]

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

#77309

On Fri, Aug 29, 2014 at 6:54 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
> Thus endeth my attempts to train Skip's Polly.  But I am curious -- if 'suckitude' is in immediate contact with punctuation such as just now, or at the end of a sentence, does it not count?  That would be suckitude indeed! ;)

Thank you all, "suckitude" made it into the corpus, or cruft, or
dustbin, whatever. :-)

? dict /usr/share/dict/words
accessing accounts adapted adding addressed adds adjusted adjusting
advantages advertised aired akumbo algorithms alister allen allowed
...
subscribed suckitude suffered suggested suggestions suggests suited
...

/usr/share/dict/words really isn't a very good dictionary. Note all
the words that are valid but which get flagged, mostly because they
have common suffixes applied to words. I'll fix that shortly.

Yes, "words" are skipped if they contain anything other than lower
case alphabetic characters. Really simple words = text.split(), then
discard words not meeting the criteria.

Skip

[toc] | [prev] | [next] | [standalone]

#77310

On Sat, Aug 30, 2014 at 12:14 PM, Skip Montanaro <skip@pobox.com> wrote:
> Yes, "words" are skipped if they contain anything other than lower
> case alphabetic characters. Really simple words = text.split(), then
> discard words not meeting the criteria.

Easy way to catch a few more: Just .strip() off a few common items of
punctuation (quotes (all types), full stop, comma, brackets (all
types), etc). If there are any inside the word, discard the word, but
those at one end or other aren't a problem.

ChrisA

[toc] | [prev] | [next] | [standalone]

#77318 — Re: suckitude classifications {taken to absurdity}

On Fri, 29 Aug 2014 12:02:36 -0700, Ethan Furman <ethan@stoneleaf.us>
declaimed the following:


>Polly want a cracker?  ;)

http://www.oxforddictionaries.com/us/definition/english/polly (definitely a
new meaning to me)

http://www.npr.org/blogs/codeswitch/2013/07/01/197644761/word-watch-on-crackers
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web