Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.databases.mysql > #596 > unrolled thread

Can MySql database store images?

Started by"Robert Crandal" <rcranz143101@gmail.com>
First post2011-04-23 23:47 -0700
Last post2011-04-25 16:46 +0200
Articles 20 on this page of 103 — 13 participants

Back to article view | Back to comp.databases.mysql


Contents

  Can MySql database store images? "Robert Crandal" <rcranz143101@gmail.com> - 2011-04-23 23:47 -0700
    Re: Can MySql database store images? "J.O. Aho" <user@example.net> - 2011-04-24 09:23 +0200
      Re: Can MySql database store images? "Robert Crandal" <rcranz143101@gmail.com> - 2011-04-24 01:19 -0700
        Re: Can MySql database store images? "J.O. Aho" <user@example.net> - 2011-04-24 11:13 +0200
    Re: Can MySql database store images? The Natural Philosopher <tnp@invalid.invalid> - 2011-04-24 11:17 +0100
    Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-24 13:06 +0100
    Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 08:42 -0400
      Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-24 14:45 +0100
        Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 10:27 -0400
          Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-24 19:04 +0100
            Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 14:44 -0400
              Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-24 20:51 +0100
                Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 17:03 -0400
                  Re: Can MySql database store images? Hans Castorp <REWYRLXHEGHO@spammotel.com> - 2011-04-24 23:18 +0200
                    Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 19:55 -0400
                  Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 08:30 +0100
                    Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 06:32 -0400
                      Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 12:04 +0100
                        Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 07:18 -0400
                          Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 13:32 +0100
                            Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 09:09 -0400
                              Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 14:17 +0100
                                Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 19:29 -0400
                        Re: Can MySql database store images? The Natural Philosopher <tnp@invalid.invalid> - 2011-04-25 12:30 +0100
                          Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 13:59 +0100
                            Re: Can MySql database store images? The Natural Philosopher <tnp@invalid.invalid> - 2011-04-25 18:10 +0100
          Re: Can MySql database store images? Norman Peelman <npeelman@cfl.rr.com> - 2011-04-24 16:40 -0400
            Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 17:05 -0400
              Re: Can MySql database store images? Norman Peelman <npeelman@cfl.rr.com> - 2011-04-25 02:57 -0400
                Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 08:59 +0100
                  Re: Can MySql database store images? Norman Peelman <npeelman@cfl.rr.com> - 2011-04-25 21:41 -0400
                    Re: Can MySql database store images? Norman Peelman <npeelman@cfl.rr.com> - 2011-04-26 06:20 -0400
                    Re: Can MySql database store images? dougatmilmacdotcom@example.com (Doug Miller) - 2011-04-26 14:10 +0000
                      Re: Can MySql database store images? Norman Peelman <npeelman@cfl.rr.com> - 2011-04-26 22:24 -0400
                        Re: Can MySql database store images? The Natural Philosopher <tnp@invalid.invalid> - 2011-04-27 08:24 +0100
                          Re: Can MySql database store images? Norman Peelman <npeelman@cfl.rr.com> - 2011-04-27 06:41 -0400
                            Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-27 12:32 +0100
                            Re: Can MySql database store images? The Natural Philosopher <tnp@invalid.invalid> - 2011-04-27 15:07 +0100
                          Re: Can MySql database store images? "Peter H. Coffin" <hellsop@ninehells.com> - 2011-04-27 07:11 -0500
                            Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-27 13:34 +0100
                              Re: Can MySql database store images? "Peter H. Coffin" <hellsop@ninehells.com> - 2011-04-27 09:25 -0500
                                Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-27 16:32 +0100
        Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 10:28 -0400
          Re: Can MySql database store images? Norman Peelman <npeelman@cfl.rr.com> - 2011-04-24 16:43 -0400
            Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 08:36 +0100
              Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 06:34 -0400
                Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 11:50 +0100
                  Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 07:20 -0400
            Re: Can MySql database store images? The Natural Philosopher <tnp@invalid.invalid> - 2011-04-25 12:42 +0100
      Re: Can MySql database store images? Luuk <Luuk@invalid.lan> - 2011-04-24 16:27 +0200
        Re: Can MySql database store images? The Natural Philosopher <tnp@invalid.invalid> - 2011-04-24 15:38 +0100
          Re: Can MySql database store images? Luuk <Luuk@invalid.lan> - 2011-04-24 16:41 +0200
          Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-24 19:07 +0100
            Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 14:46 -0400
              Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-24 21:01 +0100
                Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 17:10 -0400
                  Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 08:23 +0100
                    Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 06:39 -0400
                      Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 12:00 +0100
                        Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 07:27 -0400
                          Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 13:42 +0100
                            Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 09:12 -0400
                              Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 14:23 +0100
                                Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 19:31 -0400
                      Re: Can MySql database store images? "Mr. B-o-B" <mr.chew.baka@gmail.com> - 2011-04-25 20:30 -0500
            Re: Can MySql database store images? The Natural Philosopher <tnp@invalid.invalid> - 2011-04-25 12:21 +0100
              Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 13:46 +0100
        Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 13:23 -0400
          Re: Can MySql database store images? Luuk <Luuk@invalid.lan> - 2011-04-24 20:31 +0200
            Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 14:48 -0400
              Re: Can MySql database store images? Luuk <Luuk@invalid.lan> - 2011-04-24 21:11 +0200
      Re: Can MySql database store images? Axel Schwenke <axel.schwenke@gmx.de> - 2011-04-24 21:24 +0200
        Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 17:34 -0400
          Re: Can MySql database store images? Luuk <Luuk@invalid.lan> - 2011-04-25 12:04 +0200
            Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 06:40 -0400
              Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 11:48 +0100
                Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 07:28 -0400
                  Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 13:36 +0100
                    Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 09:17 -0400
                      Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 14:30 +0100
                        Re: Can MySql database store images? The Natural Philosopher <tnp@invalid.invalid> - 2011-04-25 18:13 +0100
                          Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 19:37 -0400
                        Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 19:35 -0400
      Re: Can MySql database store images? "Robert Crandal" <rcranz143101@gmail.com> - 2011-04-24 13:13 -0700
        Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 17:35 -0400
    Re: Can MySql database store images? Axel Schwenke <axel.schwenke@gmx.de> - 2011-04-24 21:29 +0200
      Re: Can MySql database store images? "Robert Crandal" <rcranz143101@gmail.com> - 2011-04-24 14:37 -0700
        Re: Can MySql database store images? Axel Schwenke <axel.schwenke@gmx.de> - 2011-04-25 00:41 +0200
          Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 20:16 -0400
            Re: Can MySql database store images? Axel Schwenke <axel.schwenke@gmx.de> - 2011-04-25 12:12 +0200
              Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 07:31 -0400
          Re: Can MySql database store images? Tim Watts <tw@dionic.net> - 2011-04-25 08:43 +0100
          Re: Can MySql database store images? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2011-05-01 09:59 +0200
        Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 20:03 -0400
      Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 17:39 -0400
        Re: Can MySql database store images? Axel Schwenke <axel.schwenke@gmx.de> - 2011-04-25 00:46 +0200
          Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-24 20:18 -0400
            Re: Can MySql database store images? Luuk <Luuk@invalid.lan> - 2011-04-25 12:13 +0200
              Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 06:43 -0400
                Re: Can MySql database store images? Luuk <Luuk@invalid.lan> - 2011-04-25 13:40 +0200
                  Re: Can MySql database store images? The Natural Philosopher <tnp@invalid.invalid> - 2011-04-25 12:43 +0100
                  Re: Can MySql database store images? Jerry Stuckle <jstucklex@attglobal.net> - 2011-04-25 09:19 -0400
                    Re: Can MySql database store images? Luuk <Luuk@invalid.lan> - 2011-04-25 16:46 +0200

Page 1 of 6  [1] 2 3 4 5 6  Next page →


#596 — Can MySql database store images?

From"Robert Crandal" <rcranz143101@gmail.com>
Date2011-04-23 23:47 -0700
SubjectCan MySql database store images?
Message-ID<npqdnTD4r-AUWi7QnZ2dnUVZ5qudnZ2d@giganews.com>
I just started learning about MySql in order to start building
a website that stores a database of user information.  I'm thinking
that I want my database to keep track of user accounts, user profiles,
and user information, etc.. etc...

I am planning to allow each user to upload as many photos as
they want into their personal account space, but my problem is
that I don't know WHERE to store an unknown number of images
in each user account.   Can I store a collection of images in the
MySql database somewhere?  How?  Also, what are some other
ways to do this?

Thank you!

[toc] | [next] | [standalone]


#597

From"J.O. Aho" <user@example.net>
Date2011-04-24 09:23 +0200
Message-ID<91i1fkFetdU1@mid.individual.net>
In reply to#596
Robert Crandal wrote:

> I am planning to allow each user to upload as many photos as
> they want into their personal account space, but my problem is
> that I don't know WHERE to store an unknown number of images
> in each user account.   Can I store a collection of images in the
> MySql database somewhere?  How?  Also, what are some other
> ways to do this?

There are different ways you can do this, the more traditional way would be to
have a directory where you create one directory per user (say the user name)
and then store that users images there.

You could also store the images in a database table

CREATE TABLE userimages(
	id int unsigned auto_increment primary key,
	userid int unsigned not null,
	name varchar(120) not null,
	comment text,
	image blob not null
);

The trickier part will be displaying the image, as you will need to have some
kind of script which fetches the image from the database and outputs the raw data.
I would recommend that the script fetches based on the id of the image, it may
include access checks too (if you want to allow the user to set different
access modes on images like "private", "public", "friend only").

If you are using a web hotel, then you may prefer to do the first option to
store directly to the disk, as you usually will have a lot less space to use
for your database than for your other files.

Keep in mind, don't do any "SELECT * FROM userimages" as you will fetch all
the image data too, if you want to create a list, then just use "SELECT id,
userid, name, comment FROM userimages", this will be lighter for your page,
specially if you have loads of images stored in the database.

-- 

  //Aho

[toc] | [prev] | [next] | [standalone]


#598

From"Robert Crandal" <rcranz143101@gmail.com>
Date2011-04-24 01:19 -0700
Message-ID<Kc2dnSiYuuhwQS7QnZ2dnUVZ5gWdnZ2d@giganews.com>
In reply to#597
"J.O. Aho" <user@example.net> wrote in message 
news:91i1fkFetdU1@mid.individual.net...
> Robert Crandal wrote:
>
> There are different ways you can do this, the more traditional way would 
> be to
> have a directory where you create one directory per user (say the user 
> name)
> and then store that users images there.
>
> You could also store the images in a database table
>

In your opinion or best guess, how do you think Facebook does this?
Do you think Facebook creates a directory per user where each user
can upload his personal photos?


[toc] | [prev] | [next] | [standalone]


#599

From"J.O. Aho" <user@example.net>
Date2011-04-24 11:13 +0200
Message-ID<91i7uiFr9fU1@mid.individual.net>
In reply to#598
Robert Crandal wrote:
> "J.O. Aho" <user@example.net> wrote in message
> news:91i1fkFetdU1@mid.individual.net...
>> Robert Crandal wrote:
>>
>> There are different ways you can do this, the more traditional way
>> would be to
>> have a directory where you create one directory per user (say the user
>> name)
>> and then store that users images there.
>>
>> You could also store the images in a database table
>>
> 
> In your opinion or best guess, how do you think Facebook does this?
> Do you think Facebook creates a directory per user where each user
> can upload his personal photos?

I would think the images are stored on disk and served by a static content web
service, where access is limited by configuration settings.

When I was working with an on-line web based game with 800k users, all images
were served by lighttpd while all dynamic content was served by apache. All
dynamic data was middle stored in memcached, so that pages would load far
faster as you didn't need to fetch the data from a database (and you can
settle with a cheaper database server solution).

Nowadays many of the big ones are using custom versions of apache, lighttpd
and mysql or even self developed applications to make the sites as fast as
possible.

Personally I prefer to have images on disk than in database, and I think there
will be a slight overhead with the database (I know Jerry will tell you
differently), but do your own tests and see what you think it faster and
easier to handle and use that.


-- 

  //Aho

[toc] | [prev] | [next] | [standalone]


#600

FromThe Natural Philosopher <tnp@invalid.invalid>
Date2011-04-24 11:17 +0100
Message-ID<ip0tba$ksh$1@news.albasani.net>
In reply to#596
Robert Crandal wrote:
> I just started learning about MySql in order to start building
> a website that stores a database of user information.  I'm thinking
> that I want my database to keep track of user accounts, user profiles,
> and user information, etc.. etc...
> 
> I am planning to allow each user to upload as many photos as
> they want into their personal account space, but my problem is
> that I don't know WHERE to store an unknown number of images
> in each user account.   Can I store a collection of images in the
> MySql database somewhere?  How?  Also, what are some other
> ways to do this?
> 
yes.

see BLOB data type, LOAD_FILE  SQL statement, and memory limit for what 
you have to tune.

As its a one to many relation, you need fields for ID, image, any 
descriptive stuff, and a field to say who the image belongs to.

> Thank you!
> 
> 

[toc] | [prev] | [next] | [standalone]


#601

FromTim Watts <tw@dionic.net>
Date2011-04-24 13:06 +0100
Message-ID<pagd88-pb5.ln1@squidward.dionic.net>
In reply to#596
Robert Crandal wrote:

> I just started learning about MySql in order to start building
> a website that stores a database of user information.  I'm thinking
> that I want my database to keep track of user accounts, user profiles,
> and user information, etc.. etc...
> 
> I am planning to allow each user to upload as many photos as
> they want into their personal account space, but my problem is
> that I don't know WHERE to store an unknown number of images
> in each user account.   Can I store a collection of images in the
> MySql database somewhere?  How?  Also, what are some other
> ways to do this?
> 
> Thank you!

You can store binary data (like your images) as BLOBS in the database, but 
quite frankly, I think this is a bad idea.

The way I would recommend is store the images on the filesystem, and store 
links (either unix path names or the url that your webserver presents) in 
the database.

Reasons:

If you server images out of the database, every time you display one, you 
will have to have a web handler that gets the BLOB from the database, and 
feeds it out via the web server. If you just have to provide a link, you can 
have a simple web server setup that hands out the images direct from disk 
which will be far simpler and more efficient.


-- 
Tim Watts

[toc] | [prev] | [next] | [standalone]


#602

FromJerry Stuckle <jstucklex@attglobal.net>
Date2011-04-24 08:42 -0400
Message-ID<ip15se$33s$1@dont-email.me>
In reply to#596
On 4/24/2011 2:47 AM, Robert Crandal wrote:
> I just started learning about MySql in order to start building
> a website that stores a database of user information. I'm thinking
> that I want my database to keep track of user accounts, user profiles,
> and user information, etc.. etc...
>
> I am planning to allow each user to upload as many photos as
> they want into their personal account space, but my problem is
> that I don't know WHERE to store an unknown number of images
> in each user account. Can I store a collection of images in the
> MySql database somewhere? How? Also, what are some other
> ways to do this?
>
> Thank you!
>
>

People have been using relational databases to store images for over 25 
years that I'm aware of -  we were doing it at IBM in the mid-80's. 
They store images just fine, and the more images you have, the better 
they do.  Just use a BLOB datatype.

The only downside is you can't serve the image directly from the 
database, but it just takes a short script do do so.  Not a real big thing.

There are a lot of advantages to using a database.  For instance, if you 
just store the filename in the database, what happens when someone 
deletes the image?  Or what if the filename is deleted from the database 
but the file still exists (orphaned file).

Also, file systems don't do well with thousands of users and hundreds of 
thousands of images - they just weren't made to handle that many 
individual files.  RDBs such as MySQL, OTOH, easily handles millions of 
users and hundreds of millions of images.

Remember- a file system is just a non-relational database.  It's good 
for some things but not everything.


-- 
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]


#603

FromTim Watts <tw@dionic.net>
Date2011-04-24 14:45 +0100
Message-ID<55md88-7pb.ln1@squidward.dionic.net>
In reply to#602
Jerry Stuckle wrote:


> Also, file systems don't do well with thousands of users and hundreds of
> thousands of images - they just weren't made to handle that many
> individual files.  RDBs such as MySQL, OTOH, easily handles millions of
> users and hundreds of millions of images.
> 
> Remember- a file system is just a non-relational database.  It's good
> for some things but not everything.

That rather depends on the filesystem.

XFS will happily handle 13TB with over 10 million files - that is from 
personal experience.

Obviously VFAT would be a bad idea, and putting *all* the files on one 
directory *may* be a bad idea (depends on directory hashing support in the 
FS) - but with a sensible layout (one dir per user, possibly with a further 
sublayer of user's first letters a-z) it will be fine.

Couple of points IMO - and I cannot speak for MySQL in particular - but what 
will it do for backups?

With a small database and a big filesystem of images, the DB dump will be 
fine and the FS can be backed up with an rsync type methodology.

With a huge number of images in a DB, I suspect the possibilities of doing 
incremental backups are vastly reduced and your DB dump file is ging to be 
huge.

Just my opinion, based on Postgresql - but the principles mostly apply to 
MySQL unless it has some particular features to help with the problems I 
posed.

Cheers,

Tim
-- 
Tim Watts

[toc] | [prev] | [next] | [standalone]


#604

FromJerry Stuckle <jstucklex@attglobal.net>
Date2011-04-24 10:27 -0400
Message-ID<ip1c0d$dde$1@dont-email.me>
In reply to#603
On 4/24/2011 9:45 AM, Tim Watts wrote:
> Jerry Stuckle wrote:
>
>
>> Also, file systems don't do well with thousands of users and hundreds of
>> thousands of images - they just weren't made to handle that many
>> individual files.  RDBs such as MySQL, OTOH, easily handles millions of
>> users and hundreds of millions of images.
>>
>> Remember- a file system is just a non-relational database.  It's good
>> for some things but not everything.
>
> That rather depends on the filesystem.
>
> XFS will happily handle 13TB with over 10 million files - that is from
> personal experience.
>
> Obviously VFAT would be a bad idea, and putting *all* the files on one
> directory *may* be a bad idea (depends on directory hashing support in the
> FS) - but with a sensible layout (one dir per user, possibly with a further
> sublayer of user's first letters a-z) it will be fine.
>

Such a layout is very hard to maintain and takes considerable care to 
keep from getting screwed up.  With a database, no special programming 
is required, other than a small (< 10 line) script to serve the image 
from the database.

> Couple of points IMO - and I cannot speak for MySQL in particular - but what
> will it do for backups?
>

Much better, because the files are backed up when the database is backed 
up.  Much less chance of referencing files from the database that don't 
exist, or having orphaned files (files not referenced by the database).

> With a small database and a big filesystem of images, the DB dump will be
> fine and the FS can be backed up with an rsync type methodology.
>

Databases can be backed up quite easily.  But rsync methodology does not 
solve the problems noted above.

> With a huge number of images in a DB, I suspect the possibilities of doing
> incremental backups are vastly reduced and your DB dump file is ging to be
> huge.
>

Not that bad, but if you really want, you could do incremental backups.

However, I always do full backups of both the file system and databases 
- incremental backups are fast and take less space, but are very tedious 
and error prone to restore because you have to go back to the last full 
backup then restore from the incremental backups in order. 
Additionally, incremental backups done with rsync usually end up 
restoring files which had been deleted, wasting more space.

Storage is cheap.

> Just my opinion, based on Postgresql - but the principles mostly apply to
> MySQL unless it has some particular features to help with the problems I
> posed.
>
> Cheers,
>
> Tim

Just my opinions based on 25 years of experience, starting with DB2 but 
also including MySQL, Oracle and SQL Server.

-- 
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]


#610

FromTim Watts <tw@dionic.net>
Date2011-04-24 19:04 +0100
Message-ID<p95e88-fve.ln1@squidward.dionic.net>
In reply to#604
Jerry Stuckle wrote:


> Not that bad, but if you really want, you could do incremental backups.

OK
 
> However, I always do full backups of both the file system and databases
> - incremental backups are fast and take less space, but are very tedious
> and error prone to restore because you have to go back to the last full
> backup then restore from the incremental backups in order.
> Additionally, incremental backups done with rsync usually end up
> restoring files which had been deleted, wasting more space.
> 
> Storage is cheap.

True - though I'm also factoring in time and network bandwidth. A backup 
that has to transfer 100% of a very large DB everytime over a weak offsite 
link may not be practical if that were the scenario.

But referencing your earlier points about rsync - I've always run reverse-
incrementals, ie the latest copy is always a full and the previous sets are 
what changed going backwards in time which is ery natural for rsync as you 
know so I don't see any issues with restoration from that.


>> Just my opinion, based on Postgresql - but the principles mostly apply to
>> MySQL unless it has some particular features to help with the problems I
>> posed.
>>
>> Cheers,
>>
>> Tim
> 
> Just my opinions based on 25 years of experience, starting with DB2 but
> also including MySQL, Oracle and SQL Server.
> 

That's fair enough - I will not answer for DB2 as I am aware it is a pretty 
powerful system with a lot of funky features, but I will stand by my 
personal preference that I prefer to keep RDBMSs "compact" with any lumpy 
stuff to one side. Note, I don't say it is the *only* way, but it's the way 
I default to advising my users.

On an aside, how would you do "incrementals" with MySQL - would you code 
something into the database with triggers, or do something along the lines 
of keep (in Postgres speak) the journals/WAL logs for the incrementals?

Cheers

Tim

-- 
Tim Watts

[toc] | [prev] | [next] | [standalone]


#613

FromJerry Stuckle <jstucklex@attglobal.net>
Date2011-04-24 14:44 -0400
Message-ID<ip1r2k$5pd$1@dont-email.me>
In reply to#610
On 4/24/2011 2:04 PM, Tim Watts wrote:
> Jerry Stuckle wrote:
>
>
>> Not that bad, but if you really want, you could do incremental backups.
>
> OK
>
>> However, I always do full backups of both the file system and databases
>> - incremental backups are fast and take less space, but are very tedious
>> and error prone to restore because you have to go back to the last full
>> backup then restore from the incremental backups in order.
>> Additionally, incremental backups done with rsync usually end up
>> restoring files which had been deleted, wasting more space.
>>
>> Storage is cheap.
>
> True - though I'm also factoring in time and network bandwidth. A backup
> that has to transfer 100% of a very large DB everytime over a weak offsite
> link may not be practical if that were the scenario.
>

Totally practical.  Just back it up locally then transfer whenever you 
want.  That's how I do filesystem backups, also.  That way the backup 
isn't being delayed by the network.

> But referencing your earlier points about rsync - I've always run reverse-
> incrementals, ie the latest copy is always a full and the previous sets are
> what changed going backwards in time which is ery natural for rsync as you
> know so I don't see any issues with restoration from that.
>
>

It has a tendency to leave files you want deleted.

>>> Just my opinion, based on Postgresql - but the principles mostly apply to
>>> MySQL unless it has some particular features to help with the problems I
>>> posed.
>>>
>>> Cheers,
>>>
>>> Tim
>>
>> Just my opinions based on 25 years of experience, starting with DB2 but
>> also including MySQL, Oracle and SQL Server.
>>
>
> That's fair enough - I will not answer for DB2 as I am aware it is a pretty
> powerful system with a lot of funky features, but I will stand by my
> personal preference that I prefer to keep RDBMSs "compact" with any lumpy
> stuff to one side. Note, I don't say it is the *only* way, but it's the way
> I default to advising my users.
>

Have you ever used a database for images?

> On an aside, how would you do "incrementals" with MySQL - would you code
> something into the database with triggers, or do something along the lines
> of keep (in Postgres speak) the journals/WAL logs for the incrementals?
>
> Cheers
>
> Tim
>

http://dev.mysql.com/doc/refman/5.5/en/backup-methods.html

Look down the page for incremental backups.


-- 
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]


#619

FromTim Watts <tw@dionic.net>
Date2011-04-24 20:51 +0100
Message-ID<jibe88-ud8.ln1@squidward.dionic.net>
In reply to#613
Jerry Stuckle wrote:


> Totally practical.  Just back it up locally then transfer whenever you
> want.  That's how I do filesystem backups, also.  That way the backup
> isn't being delayed by the network.

Yeah - I was kind of considering the case where the total dataset size 
exceeds the capacity of the link over the backup periodicy - which could be 
a problem for a very large dataset and a very weak link. But see below:

>> But referencing your earlier points about rsync - I've always run
>> reverse- incrementals, ie the latest copy is always a full and the
>> previous sets are what changed going backwards in time which is ery
>> natural for rsync as you know so I don't see any issues with restoration
>> from that.
>>
>>
> 
> It has a tendency to leave files you want deleted.

How so? This is a unidirectional sync - rsync is quite capable of fully 
synchronising the target with th source.

Bi-directional syncing is prone to such problems, but that doesn't apply 
here.

>>>> Just my opinion, based on Postgresql - but the principles mostly apply
>>>> to MySQL unless it has some particular features to help with the
>>>> problems I posed.
>>>>
>>>> Cheers,
>>>>
>>>> Tim
>>>
>>> Just my opinions based on 25 years of experience, starting with DB2 but
>>> also including MySQL, Oracle and SQL Server.
>>>
>>
>> That's fair enough - I will not answer for DB2 as I am aware it is a
>> pretty powerful system with a lot of funky features, but I will stand by
>> my personal preference that I prefer to keep RDBMSs "compact" with any
>> lumpy stuff to one side. Note, I don't say it is the *only* way, but it's
>> the way I default to advising my users.
>>
> 
> Have you ever used a database for images?

Yes - briefly when I was playing with some image aware DB bound widgets in a 
RAD tool. But if developing a web deplyment, I personally would feel it an 
easier task to keep the image data on the filesystem - even accounting for 
the double checking necessary.

I will admit that most of my opinions on BLOBs were drawn from pre-8.3 
postgres where a BLOB was a seperate storage entity and linkage with a 
column was weak (just an OID) and it was expected that an unmanaged DELETE 
would lead to orphaned BLOBs - at which point I could see no useful purpose 
to them.

Are MySQL BLOBs tightly tied to the table column they belong to - ie is a 
DELETE of a row with a BLOB guaranteed to destroy the BLOB too? In which 
case, I concede it is more of a sensible proposition.

>> On an aside, how would you do "incrementals" with MySQL - would you code
>> something into the database with triggers, or do something along the
>> lines of keep (in Postgres speak) the journals/WAL logs for the
>> incrementals?
>>
>> Cheers
>>
>> Tim
>>
> 
> http://dev.mysql.com/doc/refman/5.5/en/backup-methods.html
> 
> Look down the page for incremental backups.
> 
> 

Yep - that fits with what I said. 

Cheers,

Tim

-- 
Tim Watts

[toc] | [prev] | [next] | [standalone]


#624

FromJerry Stuckle <jstucklex@attglobal.net>
Date2011-04-24 17:03 -0400
Message-ID<ip236r$v3q$1@dont-email.me>
In reply to#619
On 4/24/2011 3:51 PM, Tim Watts wrote:
> Jerry Stuckle wrote:
>
>
>> Totally practical.  Just back it up locally then transfer whenever you
>> want.  That's how I do filesystem backups, also.  That way the backup
>> isn't being delayed by the network.
>
> Yeah - I was kind of considering the case where the total dataset size
> exceeds the capacity of the link over the backup periodicy - which could be
> a problem for a very large dataset and a very weak link. But see below:
>

No more so than for files.

>>> But referencing your earlier points about rsync - I've always run
>>> reverse- incrementals, ie the latest copy is always a full and the
>>> previous sets are what changed going backwards in time which is ery
>>> natural for rsync as you know so I don't see any issues with restoration
>>> from that.
>>>
>>>
>>
>> It has a tendency to leave files you want deleted.
>
> How so? This is a unidirectional sync - rsync is quite capable of fully
> synchronising the target with th source.
>
> Bi-directional syncing is prone to such problems, but that doesn't apply
> here.
>

And what happens when you delete a file - but later need to restore it 
from backup?

>>>>> Just my opinion, based on Postgresql - but the principles mostly apply
>>>>> to MySQL unless it has some particular features to help with the
>>>>> problems I posed.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Tim
>>>>
>>>> Just my opinions based on 25 years of experience, starting with DB2 but
>>>> also including MySQL, Oracle and SQL Server.
>>>>
>>>
>>> That's fair enough - I will not answer for DB2 as I am aware it is a
>>> pretty powerful system with a lot of funky features, but I will stand by
>>> my personal preference that I prefer to keep RDBMSs "compact" with any
>>> lumpy stuff to one side. Note, I don't say it is the *only* way, but it's
>>> the way I default to advising my users.
>>>
>>
>> Have you ever used a database for images?
>
> Yes - briefly when I was playing with some image aware DB bound widgets in a
> RAD tool. But if developing a web deplyment, I personally would feel it an
> easier task to keep the image data on the filesystem - even accounting for
> the double checking necessary.
>
> I will admit that most of my opinions on BLOBs were drawn from pre-8.3
> postgres where a BLOB was a seperate storage entity and linkage with a
> column was weak (just an OID) and it was expected that an unmanaged DELETE
> would lead to orphaned BLOBs - at which point I could see no useful purpose
> to them.
>
> Are MySQL BLOBs tightly tied to the table column they belong to - ie is a
> DELETE of a row with a BLOB guaranteed to destroy the BLOB too? In which
> case, I concede it is more of a sensible proposition.
>

Yes, in MySQL and most databases, the BLOB column is just another column 
in the table.  It might be stored in a separate place - but that is 
totally managed by the rdbms.  Deleting a row removes ALL data in that row.

PostGresSQL seems to be rather alone in this respect.


>>> On an aside, how would you do "incrementals" with MySQL - would you code
>>> something into the database with triggers, or do something along the
>>> lines of keep (in Postgres speak) the journals/WAL logs for the
>>> incrementals?
>>>
>>> Cheers
>>>
>>> Tim
>>>
>>
>> http://dev.mysql.com/doc/refman/5.5/en/backup-methods.html
>>
>> Look down the page for incremental backups.
>>
>>
>
> Yep - that fits with what I said.
>
> Cheers,
>
> Tim
>


-- 
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]


#627

FromHans Castorp <REWYRLXHEGHO@spammotel.com>
Date2011-04-24 23:18 +0200
Message-ID<91jicrFb4lU1@mid.individual.net>
In reply to#624
Jerry Stuckle wrote on 24.04.2011 23:03:
> Yes, in MySQL and most databases, the BLOB column is just another
> column in the table. It might be stored in a separate place - but
> that is totally managed by the rdbms. Deleting a row removes ALL data
> in that row.
>
> PostGresSQL seems to be rather alone in this respect.

PostgreSQL works just the same when you use the (recommended) bytea datatype instead of the "large objects" interface.

Thomas

[toc] | [prev] | [next] | [standalone]


#634

FromJerry Stuckle <jstucklex@attglobal.net>
Date2011-04-24 19:55 -0400
Message-ID<ip2d9m$r9q$1@dont-email.me>
In reply to#627
On 4/24/2011 5:18 PM, Hans Castorp wrote:
> Jerry Stuckle wrote on 24.04.2011 23:03:
>> Yes, in MySQL and most databases, the BLOB column is just another
>> column in the table. It might be stored in a separate place - but
>> that is totally managed by the rdbms. Deleting a row removes ALL data
>> in that row.
>>
>> PostGresSQL seems to be rather alone in this respect.
>
> PostgreSQL works just the same when you use the (recommended) bytea
> datatype instead of the "large objects" interface.
>
> Thomas

Thanks for the info.  I don't use PostgreSQL, so I wasn't aware of that.

-- 
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]


#641

FromTim Watts <tw@dionic.net>
Date2011-04-25 08:30 +0100
Message-ID<kikf88-1rv.ln1@squidward.dionic.net>
In reply to#624
Jerry Stuckle wrote:

> On 4/24/2011 3:51 PM, Tim Watts wrote:
>> Jerry Stuckle wrote:
>>
>>
>>> Totally practical.  Just back it up locally then transfer whenever you
>>> want.  That's how I do filesystem backups, also.  That way the backup
>>> isn't being delayed by the network.
>>
>> Yeah - I was kind of considering the case where the total dataset size
>> exceeds the capacity of the link over the backup periodicy - which could
>> be a problem for a very large dataset and a very weak link. But see
>> below:
>>
> 
> No more so than for files.

Yes more so for files.

Using any rsync wrapper of your choice, you may have 10TB of data total and 
find you only need to transfer 1-10GB of updates per day (real world figure 
from a university environment).

OK - you can keep shifting your binlogs from MySQL but sooner or later you 
are going to want to run another full backup and suddenly you have to shift 
all that data again. With files you always get "last night's full backup 
image" for the cost of the changed files.

> 
> And what happens when you delete a file - but later need to restore it
> from backup?

You copy it from one of the previous incrementales - specifically the one 
that was triggered due to the file deletion on the source end.

You manage how many incrementals you keep by deleting the ones you don't 
want, either consolidating the incremental set to the nearest weekly/monthly 
- or your use the "every incremental looks like a full but virtue of 
hardlinks" and just delete the ones you no longer want. Usually in some 
patter like having the last 7 dailys, 4 weeklys, 12 monthlys or whatever.

-- 
Tim Watts

[toc] | [prev] | [next] | [standalone]


#649

FromJerry Stuckle <jstucklex@attglobal.net>
Date2011-04-25 06:32 -0400
Message-ID<ip3ijr$f0q$1@dont-email.me>
In reply to#641
On 4/25/2011 3:30 AM, Tim Watts wrote:
> Jerry Stuckle wrote:
>
>> On 4/24/2011 3:51 PM, Tim Watts wrote:
>>> Jerry Stuckle wrote:
>>>
>>>
>>>> Totally practical.  Just back it up locally then transfer whenever you
>>>> want.  That's how I do filesystem backups, also.  That way the backup
>>>> isn't being delayed by the network.
>>>
>>> Yeah - I was kind of considering the case where the total dataset size
>>> exceeds the capacity of the link over the backup periodicy - which could
>>> be a problem for a very large dataset and a very weak link. But see
>>> below:
>>>
>>
>> No more so than for files.
>
> Yes more so for files.
>
> Using any rsync wrapper of your choice, you may have 10TB of data total and
> find you only need to transfer 1-10GB of updates per day (real world figure
> from a university environment).
>

That's really not a lot of data, especially in a university environment.

> OK - you can keep shifting your binlogs from MySQL but sooner or later you
> are going to want to run another full backup and suddenly you have to shift
> all that data again. With files you always get "last night's full backup
> image" for the cost of the changed files.
>

That's why I don't do incremental backups - of file systems or the database.

>>
>> And what happens when you delete a file - but later need to restore it
>> from backup?
>
> You copy it from one of the previous incrementales - specifically the one
> that was triggered due to the file deletion on the source end.
>

Ah, but you said use rsync - which does not create incrementals. 
Rather, it syncs files between two systems.


> You manage how many incrementals you keep by deleting the ones you don't
> want, either consolidating the incremental set to the nearest weekly/monthly
> - or your use the "every incremental looks like a full but virtue of
> hardlinks" and just delete the ones you no longer want. Usually in some
> patter like having the last 7 dailys, 4 weeklys, 12 monthlys or whatever.
>

And if you do need to restore the complete disk, you need to restore 
from the last full backup plus all incrementals since then.


-- 
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]


#658

FromTim Watts <tw@dionic.net>
Date2011-04-25 12:04 +0100
Message-ID<a31g88-6ii.ln1@squidward.dionic.net>
In reply to#649
Jerry Stuckle wrote:

> On 4/25/2011 3:30 AM, Tim Watts wrote:
>> Jerry Stuckle wrote:
>>
>>> On 4/24/2011 3:51 PM, Tim Watts wrote:
>>>> Jerry Stuckle wrote:
>>>>
>>>>
>>>>> Totally practical.  Just back it up locally then transfer whenever you
>>>>> want.  That's how I do filesystem backups, also.  That way the backup
>>>>> isn't being delayed by the network.
>>>>
>>>> Yeah - I was kind of considering the case where the total dataset size
>>>> exceeds the capacity of the link over the backup periodicy - which
>>>> could be a problem for a very large dataset and a very weak link. But
>>>> see below:
>>>>
>>>
>>> No more so than for files.
>>
>> Yes more so for files.
>>
>> Using any rsync wrapper of your choice, you may have 10TB of data total
>> and find you only need to transfer 1-10GB of updates per day (real world
>> figure from a university environment).
>>
> 
> That's really not a lot of data, especially in a university environment.
> 
>> OK - you can keep shifting your binlogs from MySQL but sooner or later
>> you are going to want to run another full backup and suddenly you have to
>> shift all that data again. With files you always get "last night's full
>> backup image" for the cost of the changed files.
>>
> 
> That's why I don't do incremental backups - of file systems or the
> database.
> 

So now you have to shift 10TB of data over the network every day? Even with 
the 10gigabit backboned network and a well specified backup host, it would 
take several days to shift the intial sync from cold. After that, 
incrementals would take a few hours typically, in parallel from a couple of 
dozen data storage hosts.

>>> And what happens when you delete a file - but later need to restore it
>>> from backup?
>>
>> You copy it from one of the previous incrementales - specifically the one
>> that was triggered due to the file deletion on the source end.
>>
> 
> Ah, but you said use rsync - which does not create incrementals.
> Rather, it syncs files between two systems.

I suggest you top talking now and "man rsync". It does sync filesystems. It 
also has the option to backup (divert to another directory tree) any file 
that it is about to delete or update on the target. There are your 
incrementals. I've been running that method for 10 years - it works.

> 
>> You manage how many incrementals you keep by deleting the ones you don't
>> want, either consolidating the incremental set to the nearest
>> weekly/monthly - or your use the "every incremental looks like a full but
>> virtue of hardlinks" and just delete the ones you no longer want. Usually
>> in some patter like having the last 7 dailys, 4 weeklys, 12 monthlys or
>> whatever.
>>
> 
> And if you do need to restore the complete disk, you need to restore
> from the last full backup plus all incrementals since then.

Which bit of "reverse incrementals" are you having difficulty with? The last 
backup is always a full. You only touch the incrementals if you want an 
*earlier* copy of the file.

Cheers

Tim

-- 
Tim Watts

[toc] | [prev] | [next] | [standalone]


#659

FromJerry Stuckle <jstucklex@attglobal.net>
Date2011-04-25 07:18 -0400
Message-ID<ip3la3$141$1@dont-email.me>
In reply to#658
On 4/25/2011 7:04 AM, Tim Watts wrote:
> Jerry Stuckle wrote:
>
>> On 4/25/2011 3:30 AM, Tim Watts wrote:
>>> Jerry Stuckle wrote:
>>>
>>>> On 4/24/2011 3:51 PM, Tim Watts wrote:
>>>>> Jerry Stuckle wrote:
>>>>>
>>>>>
>>>>>> Totally practical.  Just back it up locally then transfer whenever you
>>>>>> want.  That's how I do filesystem backups, also.  That way the backup
>>>>>> isn't being delayed by the network.
>>>>>
>>>>> Yeah - I was kind of considering the case where the total dataset size
>>>>> exceeds the capacity of the link over the backup periodicy - which
>>>>> could be a problem for a very large dataset and a very weak link. But
>>>>> see below:
>>>>>
>>>>
>>>> No more so than for files.
>>>
>>> Yes more so for files.
>>>
>>> Using any rsync wrapper of your choice, you may have 10TB of data total
>>> and find you only need to transfer 1-10GB of updates per day (real world
>>> figure from a university environment).
>>>
>>
>> That's really not a lot of data, especially in a university environment.
>>
>>> OK - you can keep shifting your binlogs from MySQL but sooner or later
>>> you are going to want to run another full backup and suddenly you have to
>>> shift all that data again. With files you always get "last night's full
>>> backup image" for the cost of the changed files.
>>>
>>
>> That's why I don't do incremental backups - of file systems or the
>> database.
>>
>
> So now you have to shift 10TB of data over the network every day? Even with
> the 10gigabit backboned network and a well specified backup host, it would
> take several days to shift the intial sync from cold. After that,
> incrementals would take a few hours typically, in parallel from a couple of
> dozen data storage hosts.
>

It's done every day by a large number of companies.  But your figures 
are way off - on a 10 GB network it would take about 3 hours.

>>>> And what happens when you delete a file - but later need to restore it
>>>> from backup?
>>>
>>> You copy it from one of the previous incrementales - specifically the one
>>> that was triggered due to the file deletion on the source end.
>>>
>>
>> Ah, but you said use rsync - which does not create incrementals.
>> Rather, it syncs files between two systems.
>
> I suggest you top talking now and "man rsync". It does sync filesystems. It
> also has the option to backup (divert to another directory tree) any file
> that it is about to delete or update on the target. There are your
> incrementals. I've been running that method for 10 years - it works.
>

OK, now the file gets replaced with a new one of the same name, edited 
and deleted again.  Happens all the time.  Does it still work?

And BTW - backups are nice.  But have you ever restored a disk from 
scratch using your method, up to but not including the last 3 
incrementals?

>>
>>> You manage how many incrementals you keep by deleting the ones you don't
>>> want, either consolidating the incremental set to the nearest
>>> weekly/monthly - or your use the "every incremental looks like a full but
>>> virtue of hardlinks" and just delete the ones you no longer want. Usually
>>> in some patter like having the last 7 dailys, 4 weeklys, 12 monthlys or
>>> whatever.
>>>
>>
>> And if you do need to restore the complete disk, you need to restore
>> from the last full backup plus all incrementals since then.
>
> Which bit of "reverse incrementals" are you having difficulty with? The last
> backup is always a full. You only touch the incrementals if you want an
> *earlier* copy of the file.
>
> Cheers
>
> Tim
>

And have you ever tried a complete restore to a blank disk of all but 
the last three incrementals?
-- 
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]


#669

FromTim Watts <tw@dionic.net>
Date2011-04-25 13:32 +0100
Message-ID<k76g88-rm7.ln1@squidward.dionic.net>
In reply to#659
Jerry Stuckle wrote:


> It's done every day by a large number of companies.  But your figures
> are way off - on a 10 GB network it would take about 3 hours.

My figures are based on reality with real hosts and other stuff going on on 
both hosts and networks.
 
>>>>> And what happens when you delete a file - but later need to restore it
>>>>> from backup?
>>>>
>>>> You copy it from one of the previous incrementales - specifically the
>>>> one that was triggered due to the file deletion on the source end.
>>>>
>>>
>>> Ah, but you said use rsync - which does not create incrementals.
>>> Rather, it syncs files between two systems.
>>
>> I suggest you top talking now and "man rsync". It does sync filesystems.
>> It also has the option to backup (divert to another directory tree) any
>> file that it is about to delete or update on the target. There are your
>> incrementals. I've been running that method for 10 years - it works.
>>
> 
> OK, now the file gets replaced with a new one of the same name, edited
> and deleted again.  Happens all the time.  Does it still work?

Yes. Of course it does. rsync uses one of two methods to decide to 
replace/update the file on the target:

1) (Cheap) The tuple (pathname,size,mtime) is used as a key - if source and 
target don't match then the target is updated - or more precisely, the 
target is unlinked and a new copy transfered.

This method is sufficient as the only way to break it is by perversly 
updating the mtime of you file to match and older version of the same size.

2) Do the above but include a checksum of the file as part of the tuple.

> And BTW - backups are nice.  But have you ever restored a disk from
> scratch using your method, up to but not including the last 3
> incrementals?
> 
>>>
>>>> You manage how many incrementals you keep by deleting the ones you
>>>> don't want, either consolidating the incremental set to the nearest
>>>> weekly/monthly - or your use the "every incremental looks like a full
>>>> but virtue of hardlinks" and just delete the ones you no longer want.
>>>> Usually in some patter like having the last 7 dailys, 4 weeklys, 12
>>>> monthlys or whatever.
>>>>
>>>
>>> And if you do need to restore the complete disk, you need to restore
>>> from the last full backup plus all incrementals since then.
>>
>> Which bit of "reverse incrementals" are you having difficulty with? The
>> last backup is always a full. You only touch the incrementals if you want
>> an *earlier* copy of the file.
>>
>> Cheers
>>
>> Tim
>>
> 
> And have you ever tried a complete restore to a blank disk of all but
> the last three incrementals?

No because that has never been necessary. A full restore would be required 
for a host failure or a screwup involving massive file deletion.

The incrementals are for users to go picking through when they make a booboo 
on a few of their own files generally.

But in principle, you'd restore the backup then copy on top, in time reverse 
order, the incrementals[1]. 

[1] Ok - this what you will get some surplus files that should not exist, 
but very few people complain of surplus files in the event of disaster 
recovery.

Or if you suspected this would be necessary in your world, and you don;t 
want surplus files, you would use a "cp -rl; rsync" methodology like 
rsnapshop does, that populates the incremental n-1 tree with a set of 
hardlinks to the current tree prior to running rsync. The net result is each 
"incremental" is actually a full snapshot on the day it was run. In which 
case, pick one and restore from it. This method costs inodes on the backup 
filesystem as well as directory overhead of a duplicate set of trees, but is 
not too inefficient for what it does.

Cheers,

Tim

-- 
Tim Watts

[toc] | [prev] | [next] | [standalone]


Page 1 of 6  [1] 2 3 4 5 6  Next page →

Back to top | Article view | comp.databases.mysql


csiph-web