Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #14835 > unrolled thread

Recovering an EXT3 FS - hopeless if inode list is overwritten?

Back to article view | Back to comp.os.linux.misc

Contents

  Recovering an EXT3 FS - hopeless if inode list is overwritten? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-05-22 08:05 +0000
    Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? Tim Watts <tw_usenet@dionic.net> - 2015-05-22 09:24 +0100
    Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? Michael Baeuerle <michael.baeuerle@stz-e.de> - 2015-05-22 08:38 +0000
      Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-05-22 09:58 +0000
      Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-07-12 12:54 +0000
        Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2015-07-12 12:47 -0400
          Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-07-13 07:40 +0000
            Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2015-07-13 14:30 -0400
        Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? Michael Baeuerle <michael.baeuerle@stz-e.de> - 2015-07-13 11:41 +0000
          Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-07-13 20:37 +0000
            Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-07-14 07:55 +0000
            Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? Michael Baeuerle <michael.baeuerle@stz-e.de> - 2015-07-14 14:35 +0000
              Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-07-14 15:20 +0000
    Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? Vilmos Soti <vilmos@soti.ca> - 2015-05-22 10:53 -0700
      Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-05-23 08:25 +0000
      Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? Unknown <dog@gmail.com> - 2015-05-25 06:40 +0000
    Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? lakshmi <lakshmipathi.g@gmail.com> - 2015-05-25 22:06 -0700
      Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? Jean-David Beyer <jeandavid8@verizon.net> - 2015-05-26 09:57 -0400
        Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? Robert Riches <spamtrap42@jacob21819.net> - 2015-05-27 02:58 +0000
          Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2015-06-04 06:17 +0000
            Re: Recovering an EXT3 FS - hopeless if inode list is overwritten? Charlie Gibbs <cgibbs@kltpzyxm.invalid> - 2015-06-04 18:44 +0000

Page 1 of 2  [1] 2  Next page →

#14835 — Recovering an EXT3 FS - hopeless if inode list is overwritten?

I'm trying to recover an ext3 fs I blew away by dd(1)-ing to a flash 
drive - but the device name got changed under me  :(

When I look at the alternate superblocks, it looks like the inode list 
starts at the 11th block of the fs...

Is it safe to say that if I overwrite the beginning of a very large 
filesystem, there's no way to recover any of the data, because the inodes 
will likely all be gone?

Or, is there also a redundant copy of the inode list somewhere?

TIA,
cts

[toc] | [next] | [standalone]

#14836

On 22/05/15 09:05, Charles T. Smith wrote:
> I'm trying to recover an ext3 fs I blew away by dd(1)-ing to a flash
> drive - but the device name got changed under me  :(
>
> When I look at the alternate superblocks, it looks like the inode list
> starts at the 11th block of the fs...
>
> Is it safe to say that if I overwrite the beginning of a very large
> filesystem, there's no way to recover any of the data, because the inodes
> will likely all be gone?
>
> Or, is there also a redundant copy of the inode list somewhere?

I think the inode table only has one copy.

There are multiple superblocks at intervals and recovering a copy of one 
of those might be worth a try - but I suspect you've blown the inode 
table away:

https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Layout

[toc] | [prev] | [next] | [standalone]

#14837

Charles T. Smith wrote:
> 
> I'm trying to recover an ext3 fs I blew away by dd(1)-ing to a flash 
> drive - but the device name got changed under me  :(
> 
> When I look at the alternate superblocks, it looks like the inode list 
> starts at the 11th block of the fs...
> 
> Is it safe to say that if I overwrite the beginning of a very large 
> filesystem, there's no way to recover any of the data, because the inodes 
> will likely all be gone?
> 
> Or, is there also a redundant copy of the inode list somewhere?

AFAIK there is no redundancy except for the superblock itself. But the
filesystem is divided into block groups and each such group has its
own metadata (including an inode table).

Because ext2 was designed with mechanical disks in mind, the associated
data is normally clustered inside the groups and not distributed over
the whole space to reduce seek times.

I think that there is still a chance to recover something useful from
the groups that were not overwritten.
I would try to copy the whole damaged filesystem to a recovery disk with
dd and then let fsck run on that copy. An alternative (not damaged copy)
of the superblock can be specified with the "-b" option.

===============

To look at the internal structure, use dumpe2fs on an intact ext2/3
filesystem to display data about the superblock and the block groups.
Example:
| 
| [Data from superblock]
| Group 0: (Blocks 0-32767)
|   Primary superblock at 0, Group descriptors at 1-1
|   Block bitmap at 2 (+2), Inode bitmap at 3 (+3)
|   Inode table at 4-512 (+4)
|   5 free blocks, 15476 free inodes, 52 directories
|   Free blocks: [...]
|   Free inodes: [...]
| Group 1: (Blocks 32768-65535)
|   Backup superblock at 32768, Group descriptors at 32769-32769
|   Block bitmap at 32770 (+2), Inode bitmap at 32771 (+3)
|   Inode table at 32772-33280 (+4)
|   91 free blocks, 9445 free inodes, 414 directories
|   Free blocks: [...]
|   Free inodes: [...]
| [...]

[toc] | [prev] | [next] | [standalone]

#14838

On Fri, 22 May 2015 08:38:52 +0000, Michael Baeuerle wrote:

> Charles T. Smith wrote:
>> 
>> I'm trying to recover an ext3 fs I blew away by dd(1)-ing to a flash
>> drive - but the device name got changed under me  :(
>> 
>> When I look at the alternate superblocks, it looks like the inode list
>> starts at the 11th block of the fs...
>> 
>> Is it safe to say that if I overwrite the beginning of a very large
>> filesystem, there's no way to recover any of the data, because the
>> inodes will likely all be gone?
>> 
>> Or, is there also a redundant copy of the inode list somewhere?
> 
> AFAIK there is no redundancy except for the superblock itself. But the
> filesystem is divided into block groups and each such group has its own
> metadata (including an inode table).
> 
> Because ext2 was designed with mechanical disks in mind, the associated
> data is normally clustered inside the groups and not distributed over
> the whole space to reduce seek times.
> 
> I think that there is still a chance to recover something useful from
> the groups that were not overwritten. I would try to copy the whole
> damaged filesystem to a recovery disk with dd and then let fsck run on
> that copy. An alternative (not damaged copy) of the superblock can be
> specified with the "-b" option.
> 
> ===============
> 
> To look at the internal structure, use dumpe2fs on an intact ext2/3
> filesystem to display data about the superblock and the block groups.
> Example:
> |
> | [Data from superblock]
> | Group 0: (Blocks 0-32767)
> |   Primary superblock at 0, Group descriptors at 1-1 |   Block bitmap
> at 2 (+2), Inode bitmap at 3 (+3) |   Inode table at 4-512 (+4)
> |   5 free blocks, 15476 free inodes, 52 directories |   Free blocks:
> [...]
> |   Free inodes: [...]
> | Group 1: (Blocks 32768-65535)
> |   Backup superblock at 32768, Group descriptors at 32769-32769 |  
> Block bitmap at 32770 (+2), Inode bitmap at 32771 (+3) |   Inode table
> at 32772-33280 (+4)
> |   91 free blocks, 9445 free inodes, 414 directories |   Free blocks:
> [...]
> |   Free inodes: [...]
> | [...]


Some good info, thank you.  Gives my hope ...

[toc] | [prev] | [next] | [standalone]

#15042

On Fri, 22 May 2015 08:38:52 +0000, Michael Baeuerle wrote:

> Charles T. Smith wrote:
>> 
>> I'm trying to recover an ext3 fs I blew away by dd(1)-ing to a flash
>> drive - but the device name got changed under me  :(
>> 
>> When I look at the alternate superblocks, it looks like the inode list
>> starts at the 11th block of the fs...
>> 
>> Is it safe to say that if I overwrite the beginning of a very large
>> filesystem, there's no way to recover any of the data, because the
>> inodes will likely all be gone?
>> 
>> Or, is there also a redundant copy of the inode list somewhere?
> 
> AFAIK there is no redundancy except for the superblock itself. But the
> filesystem is divided into block groups and each such group has its own
> metadata (including an inode table).
> 
> Because ext2 was designed with mechanical disks in mind, the associated
> data is normally clustered inside the groups and not distributed over
> the whole space to reduce seek times.
> 
> I think that there is still a chance to recover something useful from
> the groups that were not overwritten. I would try to copy the whole
> damaged filesystem to a recovery disk with dd and then let fsck run on
> that copy. An alternative (not damaged copy) of the superblock can be
> specified with the "-b" option.
> 
> ===============
> 
> To look at the internal structure, use dumpe2fs on an intact ext2/3
> filesystem to display data about the superblock and the block groups.
> Example:
> |
> | [Data from superblock]
> | Group 0: (Blocks 0-32767)
> |   Primary superblock at 0, Group descriptors at 1-1 |   Block bitmap
> at 2 (+2), Inode bitmap at 3 (+3) |   Inode table at 4-512 (+4)
> |   5 free blocks, 15476 free inodes, 52 directories |   Free blocks:
> [...]
> |   Free inodes: [...]
> | Group 1: (Blocks 32768-65535)
> |   Backup superblock at 32768, Group descriptors at 32769-32769 |  
> Block bitmap at 32770 (+2), Inode bitmap at 32771 (+3) |   Inode table
> at 32772-33280 (+4)
> |   91 free blocks, 9445 free inodes, 414 directories |   Free blocks:
> [...]
> |   Free inodes: [...]
> | [...]


Hello all,

I don't understand the s_blocks_count field.

I've been gnawing on this disk for awhile, in between work duties, etc.

I have a program that heuristically finds all the superblocks and dumps 
them formated, or just outputs a list of block numbers.  Now I'm trying 
to take the next step of finding the group descriptors.  The intent is to 
build a superblock that drops the destroyed block groups.

The problem is, my s_blocks_count field is 1965824 blocks long:

    1dff00:       s_blocks_count;         /* Blocks count */

This is the total count of all blocks in the FS, right?  1,965,824 4k 
blocks, or 8,052,015,104 bytes.  The problem is, it's 2Tb drive:

Disk /dev/sdc: 2000.4 GB, 2000365289472 bytes
64 heads, 32 sectors/track, 1907697 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x594e9f46

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1     1907697  1953481712   83  Linux

That's 2,000,365,289,472: 3 orders more.

I have 1,837 superblocks, all containing this value.  My mind keeps 
ending in the dead-end that s_blocks_count is really in mega-byte pages.
But there's gotta be a real explaination...

Does anybody know why my s_blocks_count is so small?

cts

[toc] | [prev] | [next] | [standalone]

#15043

On Sun, 12 Jul 2015 08:54:43 -0400, Charles T. Smith <cts.private.yahoo@gmail.com> wrote:

> Does anybody know why my s_blocks_count is so small?

It's a count of blocks, net sectors. See
http://cs.smith.edu/~nhowe/Teaching/csc262/oldlabs/ext2.html

Multiply s_blocks_count by s_log_block_size to get the number of bytes.

Regards, Dave Hodgins

-- 
Change nomail.afraid.org to ody.ca to reply by email.
(nomail.afraid.org has been set up specifically for
use in usenet. Feel free to use it yourself.)

[toc] | [prev] | [next] | [standalone]

#15045

On Sun, 12 Jul 2015 12:47:28 -0400, David W. Hodgins wrote:

> On Sun, 12 Jul 2015 08:54:43 -0400, Charles T. Smith
> <cts.private.yahoo@gmail.com> wrote:
> 
>> Does anybody know why my s_blocks_count is so small?
> 
> It's a count of blocks, net sectors. See
> http://cs.smith.edu/~nhowe/Teaching/csc262/oldlabs/ext2.html
> 
> Multiply s_blocks_count by s_log_block_size to get the number of bytes.
> 
> Regards, Dave Hodgins


Thank you, yeah that was the first mistake I'd made, but it's still 3 
orders of magnitude off.  1dff00 is 8 gigabytes but the disk (and FS) is 
2 tereabytes.

I'm pretty sure that the FS was ext3 (i.e. not ext4) - is it possible 
that this is just the lower part of a 64-bit block count?

[toc] | [prev] | [next] | [standalone]

#15050

On Mon, 13 Jul 2015 03:40:46 -0400, Charles T. Smith <cts.private.yahoo@gmail.com> wrote:

> On Sun, 12 Jul 2015 12:47:28 -0400, David W. Hodgins wrote:
>> Multiply s_blocks_count by s_log_block_size to get the number of bytes.
> Thank you, yeah that was the first mistake I'd made, but it's still 3
> orders of magnitude off.  1dff00 is 8 gigabytes but the disk (and FS) is
> 2 tereabytes.

1dff00 is 1,965,824 decimal. If that is the block count, then with a
block size of 1024 bytes, that is 2TB.

As per the code showing how to read the superblock on
http://cs.smith.edu/~nhowe/Teaching/csc262/oldlabs/ext2.html#struct
The block size in the superblock is not specified in bytes, but as
a count of how many bits to shift 1024 to get the number of bytes.

Regards, Dave Hodgins
-- 
Change nomail.afraid.org to ody.ca to reply by email.
(nomail.afraid.org has been set up specifically for
use in usenet. Feel free to use it yourself.)

[toc] | [prev] | [next] | [standalone]

#15047

Charles T. Smith wrote:
> 
> [...]
> I have a program that heuristically finds all the superblocks and dumps 
> them formated, or just outputs a list of block numbers.  Now I'm trying 
> to take the next step of finding the group descriptors.  The intent is to 
> build a superblock that drops the destroyed block groups.

Why want you do this manually? fsck should automatically recover what
is still intact if you point it to an undamaged copy of the superblock.

As an example I have created an ext3 FS in an image file for
demonstration:
------------------------------------------------------------------------
# dd if=/dev/zero of=./image.fs bs=1M count=10
# losetup /dev/loop0 ./image.fs 
# mke2fs -j -m0 /dev/loop0
[...]
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
2560 inodes, 10240 blocks
[...]
2 block groups
8192 blocks per group, 8192 fragments per group
1280 inodes per group
Superblock backups stored on blocks: 
        8193
[...]
# mount /dev/loop0 /mnt/test
# mkdir /mnt/test/dir1
# dd if=/dev/zero of=/mnt/test/dir1/file1 bs=1M count=4
# dd if=/dev/zero of=/mnt/test/dir1/file2 bs=1M count=3
# dumpe2fs /dev/loop0
[...]
Group 0: (Blocks 1-8192)
  Primary superblock at 1, Group descriptors at 2-2
  Reserved GDT blocks at 3-41
  Block bitmap at 42 (+41), Inode bitmap at 43 (+42)
  Inode table at 44-203 (+43)
  0 free blocks, 1269 free inodes, 2 directories
[Group 0 completely filled with data from file1 and file2]
  Free blocks: 
  Free inodes: 12-1280
Group 1: (Blocks 8193-10239)
  Backup superblock at 8193, Group descriptors at 8194-8194
  Reserved GDT blocks at 8195-8233
  Block bitmap at 8234 (+41), Inode bitmap at 8235 (+42)
  Inode table at 8236-8395 (+43)
  1591 free blocks, 1277 free inodes, 1 directories
                                      ^^^^^^^^^^^^^
[This directory is dir1]
  Free blocks: 8396-8400, 8654-10239
  Free inodes: 1284-2560
[The following PDF file goes completely into Group 1]
# cp /tmp/NC7S08_Fairchild.pdf /mnt/test/dir1/
# umount /dev/loop0
# losetup -d /dev/loop0
[The following will damage the first Superblock and Group 0]
# cp ./image.fs ./image_backup.fs
# dd if=/dev/zero of=./image.fs bs=1k count=1000 conv=notrunc
------------------------------------------------------------------------

This should be similar to your situation: The beginning of the FS and
the main superblock were overwritten with garbage (zeros in the demo
above).
But a superblock copy (Block 8193 in the demo above) is still available.
In addition a PDF file in an undamaged block group is waiting for
recovery.


Now I have tried whether e2fsck is able to recover the undamaged PDF
file automatically:
------------------------------------------------------------------------
# losetup /dev/loop0 ./image.fs
# mount /dev/loop0 /mnt/test
mount: you must specify the filesystem type
[As expected, because the FS is damaged]
# e2fsck -b 8193 /dev/loop0
[...]
Superblock has an invalid journal (inode 8).
Clear<y>? yes
*** ext3 journal has been deleted - filesystem is now ext2 only ***

/dev/loop0 was not cleanly unmounted, check forced.
Resize inode not valid.  Recreate<y>? yes
Pass 1: Checking inodes, blocks, and sizes
Root inode is not a directory.  Clear<y>? yes
Pass 2: Checking directory structure
Entry '..' in <2>/<1281> (1281) has deleted/unused inode 2.  Clear<y>? yes
Pass 3: Checking directory connectivity
Root inode not allocated.  Allocate<y>? yes
Unconnected directory inode 1281 (...)
[This is the subdirectory formerly named dir1 inside the overwritten
 root directory]
Connect to /lost+found<y>? yes
/lost+found not found.  Create<y>? yes
Pass 4: Checking reference counts
Inode 1281 ref count is 3, should be 2.  Fix<y>? yes
Pass 5: Checking group summary information
Block bitmap differences:  +(1--204) +(1247--8192)
Fix<y>? yes
Free blocks count wrong for group #0 (6944, counted=1040).
Fix<y>? yes
Free blocks count wrong for group #1 (1844, counted=1123).
Fix<y>? yes
Free blocks count wrong (8788, counted=2163).
Fix<y>? yes
Inode bitmap differences:  +1 +(3--10)
Fix<y>? yes
Free inodes count wrong for group #0 (1268, counted=1269).
Fix<y>? yes
Directories count wrong for group #0 (3, counted=2).
Fix<y>? yes
Free inodes count wrong for group #1 (1280, counted=1276).
Fix<y>? yes
Directories count wrong for group #1 (0, counted=1).
Fix<y>? yes
Free inodes count wrong (2548, counted=2545).
Fix<y>? yes
Recreate journal<y>? yes
Creating journal (1024 blocks):  Done.

*** journal has been re-created - filesystem is now ext3 again ***

/dev/loop0: ***** FILE SYSTEM WAS MODIFIED *****
/dev/loop0: 15/2560 files (6.7% non-contiguous), 9106/10240 blocks
# mount /dev/loop0 /mnt/test
# cmp /mnt/test/lost+found/#1281/NC7S08_Fairchild.pdf /tmp/NC7S08_Fairchild.pdf
[identical]
------------------------------------------------------------------------

The PDF file was recovered undamaged and I can view it again, the
grouping of the 3 files in one subdirectory was also recovered. Only the
root directory (and therefore the subdirectory name "dir1" and the
content of the files located in the damaged first block group are lost.

[toc] | [prev] | [next] | [standalone]

#15049

On Mon, 13 Jul 2015 11:41:43 +0000, Michael Baeuerle wrote:

> Charles T. Smith wrote:
>> 
>> [...]
>
> Why want you do this manually? fsck should automatically recover what is
> still intact if you point it to an undamaged copy of the superblock.
> 
> As an example I have created an ext3 FS in an image file for
> demonstration:


And indeed, you created a magnificent example, which I appreciate.

Let me explain why I want to do it manually (maybe I'm laboring under a 
simple misconception that can be cleared up readily):

I wrote this pgm that goes through the disk looking for superblocks - it 
found 1837 occurrences, example below.  The pgm prints out the sector 
number.  The first one found is 206983138:

$ dc
4k 206983138 8/p
25872892.2500

Interestingly all but 16 of the found superblocks are at a sector number 
that's divisible by 2 but not 4 or 8:

(echo 0; sed 's,$, 4k sa la 8/ 0k la 8/-+,'  0503/sdc1-5.0503.nums; echo 
'p4*p') | dc
457.2500
1829.0000


Now, I don't know for a fact that the block size is 4k, but I just used 
the kubuntu default when I originally created the FS.

Consequently, I don't know what number to put into e2fsck.  Here, check 
this out:

  # is the tool seeking to the right place?

  $ sudo strace dumpe2fs -o superblock=206983138 -o blocksize=512 /dev/
sdc1

  open("/dev/sdc1", O_RDONLY|O_LARGEFILE) = 3
  _llseek(3, 105975366656, [105975366656], SEEK_SET) = 0

  $ dc
  4k
  105975366656 206983138/p
  512.0000

  # i.e. it's seeking to the right place

  $ sudo dumpe2fs -o superblock=206983138 -o blocksize=512 /dev/sdc1

  dumpe2fs 1.41.11 (14-Mar-2010)
  dumpe2fs: Filesystem has unexpected block size while trying to open /
dev/sdc1
  Couldn't find valid filesystem superblock.

  # so, is there a superblock there?

  $ sudo dd if=/dev/sdc1 bs=512 count=1 skip=206983138 | ../../bin/blkdump
     78000:       s_inodes_count;         /* Inodes count */
    1dff00:       s_blocks_count;         /* Blocks count */
     17ff3:       s_r_blocks_count;       /* Reserved blocks count */
     f7099:       s_free_blocks_count;    /* Free blocks count */
     4c3da:       s_free_inodes_count;    /* Free inodes count */
         0:       s_first_data_block;     /* First Data Block */
         2:       s_log_block_size;       /* Block size */
         2:       s_log_frag_size;        /* Fragment size */
      8000:       s_blocks_per_group;     /* # Blocks per group */
      8000:       s_frags_per_group;      /* # Fragments per group */
      2000:       s_inodes_per_group;     /* # Inodes per group */
  5461f7a8:       s_mtime;                /* Mount time */
  5461f7a8:       s_wtime;                /* Write time */
         3:       s_mnt_count;            /* Mount count */
      ffff:       s_max_mnt_count;        /* Maximal mount count */
      ef53:       s_magic;                /* Magic signature */
         1:       s_state;                /* File system state */
         1:       s_errors;               /* Behaviour when detecting 
errors */
         0:       s_minor_rev_level;      /* minor revision level */
  5461e8b0:       s_lastcheck;            /* time of last check */
         0:       s_checkinterval;        /* max. time between checks */
         0:       s_creator_os;           /* OS */
         1:       s_rev_level;            /* Revision level */
         0:       s_def_resuid;           /* Default uid for reserved 
blocks */
         0:       s_def_resgid;           /* Default gid for reserved 
blocks */
         b:       s_first_ino;            /* First non-reserved inode */
       100:   s_inode_size;               /* size of inode structure */
         0:       s_block_group_nr;       /* block group # of this 
superblock */
        3c:       s_feature_compat;       /* compatible feature set */
       242:       s_feature_incompat;     /* incompatible feature set */
        7b:       s_feature_ro_compat;    /* readonly-compatible feature 
set */
         0:       s_algorithm_usage_bitmap; /* For compression */
         0:       s_prealloc_blocks;      /* Nr of blocks to try to 
preallocate*/
         0:       s_prealloc_dir_blocks;  /* Nr to preallocate for dirs */
  df010000:       s_padding1;
   8000000:       s_journal_inum;         /* inode number of journal file 
*/
         0:       s_journal_dev;          /* device number of journal 
file */
         0:       s_last_orphan;          /* start of list of inodes to 
delete */
  f806edbf:       s_hash_seed[4];         /* HTREE hash seed */
         1:       s_def_hash_version;     /* Default hash version to use 
*/
         1:       s_reserved_char_pad;
         0:       s_reserved_word_pad;
         c:       s_default_mount_opts;
         0:       s_first_meta_bg;        /* First metablock block group 
*/
  1407edbf:       s_reserved[190];        /* Padding to the end of the 
block */
  1+0 records in
  1+0 records out
  512 bytes (512 B) copied, 0.000568372 s, 901 kB/s

In conclusion, I have 1837 superblocks - they are all essentially 
identical - and unequal to the surrounding blocks, but they're at an
offset that I can't pass to any of the e2 programs.

That's why it occurred to me that there must be some other similar 
structures laying around, e.g. the Group Descriptor Tables to help me get 
oriented.

I also tried this:

$ sudo dumpe2fs -o superblock=$((206983138/2)) -o blocksize=1024 /dev/sdc1
dumpe2fs 1.41.11 (14-Mar-2010)
dumpe2fs: Filesystem has unexpected block size while trying to open /dev/
sdc1
Couldn't find valid filesystem superblock.

Thanks for your interest!
cts

[toc] | [prev] | [next] | [standalone]

#15052

A couple of notes to this superblock dump:

- the mtime/wtime values represent
  2014/11/11 12:48:56
  Definitely plausible

- the two arrays are not properly implemented by the dump program
(s_hash_seed and s_reserved).

On Mon, 13 Jul 2015 20:37:12 +0000, Charles T. Smith wrote:

>   $ sudo dd if=/dev/sdc1 bs=512 count=1 skip=206983138 |
>   ../../bin/blkdump
>      78000:       s_inodes_count;         /* Inodes count */
>     1dff00:       s_blocks_count;         /* Blocks count */
>      17ff3:       s_r_blocks_count;       /* Reserved blocks count */
>      f7099:       s_free_blocks_count;    /* Free blocks count */
>      4c3da:       s_free_inodes_count;    /* Free inodes count */
>          0:       s_first_data_block;     /* First Data Block */
>          2:       s_log_block_size;       /* Block size */
>          2:       s_log_frag_size;        /* Fragment size */
>       8000:       s_blocks_per_group;     /* # Blocks per group */
>       8000:       s_frags_per_group;      /* # Fragments per group */
>       2000:       s_inodes_per_group;     /* # Inodes per group */
>   5461f7a8:       s_mtime;                /* Mount time */
>   5461f7a8:       s_wtime;                /* Write time */
>          3:       s_mnt_count;            /* Mount count */
>       ffff:       s_max_mnt_count;        /* Maximal mount count */
>       ef53:       s_magic;                /* Magic signature */
>          1:       s_state;                /* File system state */
>          1:       s_errors;               /* Behaviour when detecting...
>          0:       s_minor_rev_level;      /* minor revision level */
>   5461e8b0:       s_lastcheck;            /* time of last check */
>          0:       s_checkinterval;        /* max. time between checks */
>          0:       s_creator_os;           /* OS */
>          1:       s_rev_level             /* Revision level */
>          0:       s_def_resuid;           /* Default uid for reserved...
>          0:       s_def_resgid;           /* Default gid for reserved...
>          b:       s_first_ino;            /* First non-reserved inode */
>        100:   s_inode_size;               /* size of inode structure */
>          0:       s_block_group_nr;       /* block group # of this sb */
>         3c:       s_feature_compat;       /* compatible feature set */
>        242:       s_feature_incompat;     /* incompatible feature set */
>         7b:       s_feature_ro_compat;    /* readonly-compatible fea...
>          0:       s_algorithm_usage_bitmap; /* For compression */
>          0:       s_prealloc_blocks;      /* Nr of blocks to try to ...
>          0:       s_prealloc_dir_blocks;  /* Nr to preallocate for ...
>   df010000:       s_padding1;
>    8000000:       s_journal_inum;         /* inode number of journal ...
>          0:       s_journal_dev;          /* device number of journal...
>          0:       s_last_orphan;          /* start of list of inodes ...
>   f806edbf:       s_hash_seed[4];         /* HTREE hash seed */
>          1:       s_def_hash_version;     /* Default hash version to ...
>          1:       s_reserved_char_pad;
>          0:       s_reserved_word_pad;
>          c:       s_default_mount_opts;
>          0:       s_first_meta_bg;        /* First metablock block gr...
>   1407edbf:       s_reserved[190];        /* Padding to the end of ...
>
>   1+0 records in
>   1+0 records out
>   512 bytes (512 B) copied, 0.000568372 s, 901 kB/s
> 
> In conclusion, I have 1837 superblocks - they are all essentially
> identical - and unequal to the surrounding blocks, but they're at an
> offset that I can't pass to any of the e2 programs.
> 
> That's why it occurred to me that there must be some other similar
> structures laying around, e.g. the Group Descriptor Tables to help me
> get oriented.
> 
> I also tried this:
> 
> $ sudo dumpe2fs -o superblock=$((206983138/2)) -o blocksize=1024
> /dev/sdc1 dumpe2fs 1.41.11 (14-Mar-2010)
> dumpe2fs: Filesystem has unexpected block size while trying to open
> /dev/ sdc1
> Couldn't find valid filesystem superblock.
> 
> Thanks for your interest!
> cts

[toc] | [prev] | [next] | [standalone]

#15054

Charles T. Smith wrote:
> 
> Let me explain why I want to do it manually (maybe I'm laboring under a 
> simple misconception that can be cleared up readily):
> 
> I wrote this pgm that goes through the disk looking for superblocks - it 
> found 1837 occurrences, example below.  The pgm prints out the sector 
> number.

I don't like the name "sector" here, because modern drives don't use
their sector size on the interface (e.g. use 4096 byte sectors on the
medium, but 512 byte logical blocks on the interface).

> The first one found is 206983138:

Therefore this should be a LBA (Logical Block Address) on the interface
of the disk

> $ dc
> 4k 206983138 8/p
> 25872892.2500
> 
> Interestingly all but 16 of the found superblocks are at a sector number 
> that's divisible by 2 but not 4 or 8:
> 
> (echo 0; sed 's,$, 4k sa la 8/ 0k la 8/-+,'  0503/sdc1-5.0503.nums; echo 
> 'p4*p') | dc
> 457.2500
> 1829.0000

The first question here is:
Are these numbers absolute LBAs (for the whole disk), or are they
relative LBAs with an offset (to the start of a partition)?

For the view from the FS side, the numbering scheme is relative to the
start of the FS (usually the start of a partition).

> Now, I don't know for a fact that the block size is 4k, but I just used 
> the kubuntu default when I originally created the FS.

The default ext2/3 block size for large FS is 4096 bytes on x86
architecture machines.
 
> Consequently, I don't know what number to put into e2fsck.  Here, check 
> this out:
> 
>   # is the tool seeking to the right place?
> 
>   $ sudo strace dumpe2fs -o superblock=206983138 -o blocksize=512 /dev/sdc1
                                                      ^^^^^^^^^^^^^
This is very likely wrong. This parameter specify the blocksize of the
FS and as noted above, the expected block size of the FS is 4096 bytes.


I have created a test FS again (this time with 4096 byte blocksize):
------------------------------------------------------------------------
# dd if=/dev/zero of=./image.fs bs=1M count=200
# losetup /dev/loop0 ./image.fs
# mke2fs -j -m 0 -b 4096 /dev/loop0
[...]
Block size=4096 (log=2)
Fragment size=4096 (log=2)
[...]
Maximum filesystem blocks=54525952
2 block groups
32768 blocks per group, 32768 fragments per group
25600 inodes per group
Superblock backups stored on blocks: 
        32768

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
------------------------------------------------------------------------

Test with feeding the number 32768 into the dump tools "as is":
------------------------------------------------------------------------
# dumpe2fs -o superblock=32768 -h /dev/loop0
[...]
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype sparse_super large_file
Filesystem flags:         unsigned_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         not clean
Errors behavior:          Continue
Filesystem OS type:       Linux
[...]
------------------------------------------------------------------------
This means the expected number is a the block number inside the FS.

This documentation:
<http://www.nongnu.org/ext2-doc/ext2.html#DISK-ORGANISATION>
specify that only the first superblock position has an offset of 1024
bytes from the start of block 0. The superblock backups all start on
boundaries of FS blocks.

The expected byte offset to the start of the FS therefore is expected to
be:

   32768 * 4096 = 134217728

Let's verify whether really something is read from this position:
------------------------------------------------------------------------
# strace dumpe2fs -o superblock=32768 -h /dev/loop0
[...]
open("/dev/loop0", O_RDONLY|O_LARGEFILE) = 3
stat64(0x10073838, 0xffc8e718)          = 0
ioctl(3, 0x2000127c, 0xffc8e788)        = 0
lseek(3, 134217728, SEEK_SET)           = 134217728
         ^^^^^^^^^
read(3, [...], 1024) = 1024
[...]
------------------------------------------------------------------------
Good (note that the marked offset for the lseek() system call is in
byte units). 

To get the offset in 512 byte logical blocks:

   134217728 / 512 = 262144
                     ^^^^^^
This should be the relative LBA to the start of the FS. In your case you
may additionally need to add the partition offset if you program
extracts absolute LBA numbers.

What you want is the reverse direction of what I described above:
- Take LBA from your search program
- Subtract partition offset from LBA (if any)
- Divide by 8 (4096 / 512) to get the FS block address
- Feed this value into the ext2 FS tool

Example for a partition offset of hypothetical 16 logical blocks:
- Superblock backup found on absolute LBA: 262160
- Subtract partition offset for relative LBA: 262160 - 16 = 262144
- Divide by 8 for FS block address: 262144 / 8 = 32768
- Feed this value into dumpe2fs


BTW:
If you don't have the partition table anymore, maybe take another disk
and create a similar layout. Same with the FS: Create a new empty FS
on the other disk to see what defaults are used by your OS. You can
also check your analysis tool with this undamaged OS and verify that
the block address calculations match.

[toc] | [prev] | [next] | [standalone]

#15056

On Tue, 14 Jul 2015 14:35:01 +0000, Michael Baeuerle wrote:

> Charles T. Smith wrote:
...
>> I wrote this pgm that goes through the disk looking for superblocks -
>> it found 1837 occurrences, example below.  The pgm prints out the
>> sector number.
> 
> I don't like the name "sector" here, because modern drives don't use
> their sector size on the interface (e.g. use 4096 byte sectors on the
> medium, but 512 byte logical blocks on the interface).


I just counted freads of 512 bytes.  I opened /dev/sdc1.  I recreated the 
partition table by simply using the fdisk defaults, assuming that that's 
what I used originally ...

Ah!  But that's a good place to start checking!!!  ;)


> ...
>>   # is the tool seeking to the right place?
>> 
>>   $ sudo strace dumpe2fs -o superblock=206983138 -o blocksize=512
>>   /dev/sdc1
>                                                       ^^^^^^^^^^^^^
> This is very likely wrong. This parameter specify the blocksize of the
> FS and as noted above, the expected block size of the FS is 4096 bytes.


Well, as noted above, the SB offset (206983138) is based on my 512 byte 
increments. ... oh, you mean, not allowed as an alternate blocksize ... 
but I did this:
>> $ sudo dumpe2fs -o superblock=$((206983138/2)) -o blocksize=1024
>> /dev/sdc1 dumpe2fs 1.41.11 (14-Mar-2010) dumpe2fs: Filesystem has
>> unexpected block size while trying to open /dev/ sdc1
>> Couldn't find valid filesystem superblock.


The rest I'll need a few hours to study ... "I'll be back!"


> I have created a test FS again (this time with 4096 byte blocksize):
> ------------------------------------------------------------------------
> # dd if=/dev/zero of=./image.fs bs=1M count=200 # losetup /dev/loop0
> ./image.fs
> # mke2fs -j -m 0 -b 4096 /dev/loop0
> [...]
> Block size=4096 (log=2)
> Fragment size=4096 (log=2)
> [...]
> Maximum filesystem blocks=54525952
> 2 block groups
> 32768 blocks per group, 32768 fragments per group 25600 inodes per group
> Superblock backups stored on blocks:
>         32768
> 
> Allocating group tables: done
> Writing inode tables: done
> Creating journal (4096 blocks): done
> Writing superblocks and filesystem accounting information: done
> ------------------------------------------------------------------------
> 
> Test with feeding the number 32768 into the dump tools "as is":
> ------------------------------------------------------------------------
> # dumpe2fs -o superblock=32768 -h /dev/loop0 [...]
> Filesystem magic number:  0xEF53
> Filesystem revision #:    1 (dynamic) Filesystem features:     
> has_journal ext_attr resize_inode dir_index filetype sparse_super
> large_file Filesystem flags:         unsigned_directory_hash Default
> mount options:    user_xattr acl Filesystem state:         not clean
> Errors behavior:          Continue
> Filesystem OS type:       Linux
> [...]
> ------------------------------------------------------------------------
> This means the expected number is a the block number inside the FS.
> 
> This documentation:
> <http://www.nongnu.org/ext2-doc/ext2.html#DISK-ORGANISATION> specify
> that only the first superblock position has an offset of 1024 bytes from
> the start of block 0. The superblock backups all start on boundaries of
> FS blocks.
> 
> The expected byte offset to the start of the FS therefore is expected to
> be:
> 
>    32768 * 4096 = 134217728
> 
> Let's verify whether really something is read from this position:
> ------------------------------------------------------------------------
> # strace dumpe2fs -o superblock=32768 -h /dev/loop0 [...]
> open("/dev/loop0", O_RDONLY|O_LARGEFILE) = 3 stat64(0x10073838,
> 0xffc8e718)          = 0 ioctl(3, 0x2000127c, 0xffc8e788)        = 0
> lseek(3, 134217728, SEEK_SET)           = 134217728
>          ^^^^^^^^^
> read(3, [...], 1024) = 1024
> [...]
> ------------------------------------------------------------------------
> Good (note that the marked offset for the lseek() system call is in byte
> units).
> 
> To get the offset in 512 byte logical blocks:
> 
>    134217728 / 512 = 262144
>                      ^^^^^^
> This should be the relative LBA to the start of the FS. In your case you
> may additionally need to add the partition offset if you program
> extracts absolute LBA numbers.
> 
> What you want is the reverse direction of what I described above: - Take
> LBA from your search program
> - Subtract partition offset from LBA (if any) - Divide by 8 (4096 / 512)
> to get the FS block address - Feed this value into the ext2 FS tool
> 
> Example for a partition offset of hypothetical 16 logical blocks: -
> Superblock backup found on absolute LBA: 262160 - Subtract partition
> offset for relative LBA: 262160 - 16 = 262144 - Divide by 8 for FS block
> address: 262144 / 8 = 32768 - Feed this value into dumpe2fs
> 
> 
> BTW:
> If you don't have the partition table anymore, maybe take another disk
> and create a similar layout. Same with the FS: Create a new empty FS on
> the other disk to see what defaults are used by your OS. You can also
> check your analysis tool with this undamaged OS and verify that the
> block address calculations match.

[toc] | [prev] | [next] | [standalone]

#14840

"Charles T. Smith" <cts.private.yahoo@gmail.com> writes:

> I'm trying to recover an ext3 fs I blew away by dd(1)-ing to a flash 
> drive - but the device name got changed under me  :(

I remember there is a program in a photo-related package which recovers
some files from a messed up partition. AFAIR it simply looks at the data
blocks and tries to figure out what kind of data it is. Of course, if
the FS is fragmented, then it won't help.

OK, I found it: http://www.cgsecurity.org/wiki/PhotoRec

Good luck.

Vilmos

[toc] | [prev] | [next] | [standalone]

#14842

On Fri, 22 May 2015 10:53:53 -0700, Vilmos Soti wrote:

> ity.org/wiki/PhotoRec

Thank you, I'll give that a try as well.

cts.

[toc] | [prev] | [next] | [standalone]

#14872

On Fri, 22 May 2015 10:53:53 -0700, Vilmos Soti wrote:

> "Charles T. Smith" <cts.private.yahoo@gmail.com> writes:
> 
>> I'm trying to recover an ext3 fs I blew away by dd(1)-ing to a flash
>> drive - but the device name got changed under me  :(
> 
> I remember there is a program in a photo-related package which recovers
> some files from a messed up partition. AFAIR it simply looks at the data
> blocks and tries to figure out what kind of data it is. Of course, if
> the FS is fragmented, then it won't help.
> 
> OK, I found it: http://www.security.org/wiki/PhotoRec
> 

I didn't succeeed with `testdisk` from security.org
and would like to know how other users experience it.

Let's see some logged results?

[toc] | [prev] | [next] | [standalone]

#14875

>Is it safe to say that if I overwrite the beginning of a very large 
>filesystem, there's no way to recover any of the data, 

If you corrupted only the beginning part of FS, you can recover *some*  data thanks to layout ext3fs. 

any idea how big your file system was and how much got overwritten?

I think you can try mounting with backup superblocks and see how it goes.

Can you tell us the exact command you used overwrite the FS? 

There exists forensic tools (which uses file's signature to recover things) photorec is one such tool. Since you might have reboot the machine, i think ext3grep won't help. I know some people have success with my tool extcarve.

But I would recommend you to try mounting with backup super block first before trying forensic recovery tools. 

[toc] | [prev] | [next] | [standalone]

#14877

>> Is it safe to say that if I overwrite the beginning of a very large 
>> filesystem, there's no way to recover any of the data, 
> 

Long ago, I was working at a large organization that had a computer
support group. I was in the middle of a task and compiled a program. I
then typed ./a.out and nothing happened. I did ls -l a.out and there was
no such file, so I tried to recompile it, but the source files were
gone. All my stuff was gone. I went to the administrator and he had done
a mkfs on the file system I was in. I asked him to restore it from
backup (they did backups every day), but he said he did not do backups
on that machine. I almost punched him in the face. Many months of work
gone. Instead, I just went home for the day. I was too pissed off to
work. I never did got those files restored and that was that.

Since then I do daily backups of the entire file system of my
computer(s). Then when that happens, it is a simple matter to restore
lost or damaged files, file systems, etc., from the backups.

I have friends who never do backups. People buy computers with no
hardware to do backups. What are they thinking of?

-- 
  .~.  Jean-David Beyer          Registered Linux User 85642.
  /V\  PGP-Key:166D840A 0C610C8B Registered Machine  1935521.
 /( )\ Shrewsbury, New Jersey    http://linuxcounter.net
 ^^-^^ 09:50:02 up 24 days, 17:41, 2 users, load average: 4.08, 4.26, 4.44

[toc] | [prev] | [next] | [standalone]

#14879

On 2015-05-26, Jean-David Beyer <jeandavid8@verizon.net> wrote:
>>> Is it safe to say that if I overwrite the beginning of a very large 
>>> filesystem, there's no way to recover any of the data, 
>> 
>
> Long ago, I was working at a large organization that had a computer
> support group. I was in the middle of a task and compiled a program. I
> then typed ./a.out and nothing happened. I did ls -l a.out and there was
> no such file, so I tried to recompile it, but the source files were
> gone. All my stuff was gone. I went to the administrator and he had done
> a mkfs on the file system I was in. I asked him to restore it from
> backup (they did backups every day), but he said he did not do backups
> on that machine. I almost punched him in the face. Many months of work
> gone. Instead, I just went home for the day. I was too pissed off to
> work. I never did got those files restored and that was that.
>
> Since then I do daily backups of the entire file system of my
> computer(s). Then when that happens, it is a simple matter to restore
> lost or damaged files, file systems, etc., from the backups.
>
> I have friends who never do backups. People buy computers with no
> hardware to do backups. What are they thinking of?

Somewhere, I read something similar to this:

     There are two kinds of people: 1) those who have lost
     important data; 2) those who will.

-- 
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)

[toc] | [prev] | [next] | [standalone]

#14891

On Wed, 27 May 2015 02:58:22 +0000, Robert Riches wrote:

> On 2015-05-26, Jean-David Beyer <jeandavid8@verizon.net> wrote:
>>>> Is it safe to say that if I overwrite the beginning of a very large
>>>> filesystem, there's no way to recover any of the data,
>>> 
>>> 
>> Long ago, I was working at a large organization that had a computer
>> support group. I was in the middle of a task and compiled a program. I
>> then typed ./a.out and nothing happened. I did ls -l a.out and there
>> was no such file, so I tried to recompile it, but the source files were
>> gone. All my stuff was gone. I went to the administrator and he had
>> done a mkfs on the file system I was in. I asked him to restore it from
>> backup (they did backups every day), but he said he did not do backups
>> on that machine. I almost punched him in the face. Many months of work
>> gone. Instead, I just went home for the day. I was too pissed off to
>> work. I never did got those files restored and that was that.
>>
>> Since then I do daily backups of the entire file system of my
>> computer(s). Then when that happens, it is a simple matter to restore
>> lost or damaged files, file systems, etc., from the backups.
>>
>> I have friends who never do backups. People buy computers with no
>> hardware to do backups. What are they thinking of?
> 
> Somewhere, I read something similar to this:
> 
>      There are two kinds of people: 1) those who have lost important
>      data; 2) those who will.


The corollary is, and both groups will do it again, because backups
are an incomplete solution.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.os.linux.misc

csiph-web