Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #14543 > unrolled thread

scanning HD data

Back to article view | Back to comp.os.linux.misc

Contents

  scanning HD data "Bill Cunningham" <nospam@nspam.invalid> - 2015-04-14 16:00 -0400
    Re: scanning HD data Vilmos Soti <vilmos@soti.ca> - 2015-04-14 13:14 -0700
      Re: scanning HD data "Bill Cunningham" <nospam@nspam.invalid> - 2015-04-14 16:18 -0400
    Re: scanning HD data Andreas Kohlbach <apr15.7.ankman@spamgourmet.net> - 2015-04-14 16:15 -0400
    Re: scanning HD data Eli the Bearded <*@eli.users.panix.com> - 2015-04-14 20:17 +0000
      Re: scanning HD data "Bill Cunningham" <nospam@nspam.invalid> - 2015-04-14 16:23 -0400
        Re: scanning HD data root <NoEMail@home.org> - 2015-04-14 22:00 +0000
        Re: scanning HD data Eli the Bearded <*@eli.users.panix.com> - 2015-04-14 23:20 +0000
          Re: scanning HD data "Bill Cunningham" <nospam@nspam.invalid> - 2015-04-14 19:59 -0400
            Re: scanning HD data Eli the Bearded <*@eli.users.panix.com> - 2015-04-15 00:43 +0000
              Re: scanning HD data "Bill Cunningham" <nospam@nspam.invalid> - 2015-04-14 21:09 -0400
              Re: scanning HD data "Bill Cunningham" <nospam@nspam.invalid> - 2015-04-15 14:35 -0400
        Re: scanning HD data Chick Tower <c.tower@deadspam.com> - 2015-04-15 17:09 +0000
          Re: scanning HD data The Natural Philosopher <tnp@invalid.invalid> - 2015-04-15 18:26 +0100
          Re: scanning HD data "Bill Cunningham" <nospam@nspam.invalid> - 2015-04-15 14:27 -0400
    Re: scanning HD data Baho Utot <baho-utot@columbus.rr.com> - 2015-04-14 18:42 -0400
      Re: scanning HD data "Bill Cunningham" <nospam@nspam.invalid> - 2015-04-14 19:28 -0400
    Re: scanning HD data James Moe <jimoeDESPAM@sohnen-moe.com> - 2015-04-15 11:24 -0700

#14543 — scanning HD data

    If I had a dump of a file. And then wanted to find it on the filesystem, 
could I use sed? I've never used this command and I am not familiar with it. 
Other than I know it exists. It is a "stream editor" whatever that is. If I 
had a dump of XX1. It's 500 MB long. Can I use sed and pipe to find it on 
the filesystem? I am running ext3 right now.

Bill

[toc] | [next] | [standalone]

#14544

"Bill Cunningham" <nospam@nspam.invalid> writes:

>     If I had a dump of a file. And then wanted to find it on the filesystem, 
> could I use sed? I've never used this command and I am not familiar with it. 
> Other than I know it exists. It is a "stream editor" whatever that is. If I 
> had a dump of XX1. It's 500 MB long. Can I use sed and pipe to find it on 
> the filesystem? I am running ext3 right now.

What do you want to do? Is this dump of a file a ... file on the
filesystem? Or is it the dump of a filesystem and you want to find a
file inside it?

Anyways, sed is most probably not what you are looking for. You can read
more about sed here: http://www.grymoire.com/Unix/sed.html

Vilmos

[toc] | [prev] | [next] | [standalone]

#14547

"Vilmos Soti" <vilmos@soti.ca> wrote in message 
news:lq3842sc1a.fsf@pia.msmri.medicine.ubc.ca...

> What do you want to do? Is this dump of a file a ... file on the
> filesystem? Or is it the dump of a filesystem and you want to find a
> file inside it?
>
> Anyways, sed is most probably not what you are looking for. You can read
> more about sed here: http://www.grymoire.com/Unix/sed.html

    I hexdump of a file. I want to scan the filesystem maybe with dd to find 
the hex representations of that file and its' location on the filesystem.

Bill

[toc] | [prev] | [next] | [standalone]

#14545

Bill Cunningham wrote on 14. April 2015:
>
>     If I had a dump of a file. And then wanted to find it on the filesystem, 
> could I use sed? I've never used this command and I am not familiar with it. 
> Other than I know it exists. It is a "stream editor" whatever that is. If I 
> had a dump of XX1. It's 500 MB long. Can I use sed and pipe to find it on 
> the filesystem? I am running ext3 right now.

Try

file -s

At least that works on hard disk nodes.

You could also try to "loop" mount the file. Something like

mount -o loop file /media/somwhere
-- 
Andreas

I use a Unix based operating system, which means I get laid almost as often as I
have to reboot my computer.

[toc] | [prev] | [next] | [standalone]

#14546

In comp.os.linux.misc, Bill Cunningham <nospam@nspam.invalid> wrote:
>     If I had a dump of a file. And then wanted to find it on the filesystem, 
> could I use sed? I've never used this command and I am not familiar with it. 
> Other than I know it exists. It is a "stream editor" whatever that is. If I 
> had a dump of XX1. It's 500 MB long. Can I use sed and pipe to find it on 
> the filesystem? I am running ext3 right now.

In the beginging there was 'ed'. It is an editor and can do all things
an editor needs to do. It doesn't do anything that you might like an
editor to also do. 'ed' can modify your file with a reg-exp search and
replace. But if you have any errors in that command, you get the
universal error message of 'ed': "?". 'ed' can copy or move lines around
in your file. But if you have any errors in those commands, you get the
universal error message of 'ed': "?". 'ed' can save files and open files. 
But if you have any errors in those commands, you get the universal
error message of 'ed': "?".

'sed' offers many of the features of 'ed', but dispenses with that whole
inter-active editing thing, and dispenses with that whole operate on
files on disk thing. 'sed' takes a set of editing commands and applies
them to STDIN and sends the output to STDOUT. You can get 'sed' to use a
list of files instead of STDIN, but the output is still STDOUT, not
changing those files named.

'ex' offers many of the features of 'ed', but with some more and with
actual error messages. 'ex' also has this novel concept of using the
whole screen in a mode called 'visual', usually just invoked with the
abbreviation 'vi'. It's all the power of 'ed', but with a friendlier
face and some new exciting features like named cut buffers and undo.

So anyway, no. You don't want 'sed' for your problem.

Elijah
------
have you considered 'find' and 'grep'?

[toc] | [prev] | [next] | [standalone]

#14548

"Eli the Bearded" <*@eli.users.panix.com> wrote in message 
news:eli$1504141617@qz.little-neck.ny.us...
> In comp.os.linux.misc, Bill Cunningham <nospam@nspam.invalid> wrote:
>>     If I had a dump of a file. And then wanted to find it on the 
>> filesystem,
>> could I use sed? I've never used this command and I am not familiar with 
>> it.
>> Other than I know it exists. It is a "stream editor" whatever that is. If 
>> I
>> had a dump of XX1. It's 500 MB long. Can I use sed and pipe to find it on
>> the filesystem? I am running ext3 right now.
>
> In the beginging there was 'ed'. It is an editor and can do all things
> an editor needs to do. It doesn't do anything that you might like an
> editor to also do. 'ed' can modify your file with a reg-exp search and
> replace. But if you have any errors in that command, you get the
> universal error message of 'ed': "?". 'ed' can copy or move lines around
> in your file. But if you have any errors in those commands, you get the
> universal error message of 'ed': "?". 'ed' can save files and open files.
> But if you have any errors in those commands, you get the universal
> error message of 'ed': "?".
>
> 'sed' offers many of the features of 'ed', but dispenses with that whole
> inter-active editing thing, and dispenses with that whole operate on
> files on disk thing. 'sed' takes a set of editing commands and applies
> them to STDIN and sends the output to STDOUT. You can get 'sed' to use a
> list of files instead of STDIN, but the output is still STDOUT, not
> changing those files named.
>
> 'ex' offers many of the features of 'ed', but with some more and with
> actual error messages. 'ex' also has this novel concept of using the
> whole screen in a mode called 'visual', usually just invoked with the
> abbreviation 'vi'. It's all the power of 'ed', but with a friendlier
> face and some new exciting features like named cut buffers and undo.
>
> So anyway, no. You don't want 'sed' for your problem.
>
> Elijah
> ------
> have you considered 'find' and 'grep'?

    Can you use find and grep to scan /dev/sda1 or so and match what comes 
through there to a hexdump of XX1 ? And get the address of the sector where 
that file has its' beginning and end?

HTH
Bill

[toc] | [prev] | [next] | [standalone]

#14549

Bill Cunningham <nospam@nspam.invalid> wrote:
> 
> "Eli the Bearded" <*@eli.users.panix.com> wrote in message 
> news:eli$1504141617@qz.little-neck.ny.us...
>> In comp.os.linux.misc, Bill Cunningham <nospam@nspam.invalid> wrote:
>>>     If I had a dump of a file. And then wanted to find it on the 
>>> filesystem,
>>> could I use sed? I've never used this command and I am not familiar with 
>>> it.
>>> Other than I know it exists. It is a "stream editor" whatever that is. If 
>>> I
>>> had a dump of XX1. It's 500 MB long. Can I use sed and pipe to find it on
>>> the filesystem? I am running ext3 right now.
>>
>> In the beginging there was 'ed'. It is an editor and can do all things
>> an editor needs to do. It doesn't do anything that you might like an
>> editor to also do. 'ed' can modify your file with a reg-exp search and
>> replace. But if you have any errors in that command, you get the
>> universal error message of 'ed': "?". 'ed' can copy or move lines around
>> in your file. But if you have any errors in those commands, you get the
>> universal error message of 'ed': "?". 'ed' can save files and open files.
>> But if you have any errors in those commands, you get the universal
>> error message of 'ed': "?".
>>
>> 'sed' offers many of the features of 'ed', but dispenses with that whole
>> inter-active editing thing, and dispenses with that whole operate on
>> files on disk thing. 'sed' takes a set of editing commands and applies
>> them to STDIN and sends the output to STDOUT. You can get 'sed' to use a
>> list of files instead of STDIN, but the output is still STDOUT, not
>> changing those files named.
>>
>> 'ex' offers many of the features of 'ed', but with some more and with
>> actual error messages. 'ex' also has this novel concept of using the
>> whole screen in a mode called 'visual', usually just invoked with the
>> abbreviation 'vi'. It's all the power of 'ed', but with a friendlier
>> face and some new exciting features like named cut buffers and undo.
>>
>> So anyway, no. You don't want 'sed' for your problem.
>>
>> Elijah
>> ------
>> have you considered 'find' and 'grep'?
> 
>     Can you use find and grep to scan /dev/sda1 or so and match what comes 
> through there to a hexdump of XX1 ? And get the address of the sector where 
> that file has its' beginning and end?

grep is used for finding text in files. If your hexdump looks like text
then you can use grep. Otherwise grep won't help.

First I would use the hexdump to find the file size. Then you can
use find to list files that have that size. That will *greatly* reduce
your task. The type of many files is indicated by bytes at the
beginning of a file. You might then see if you can match the start
of your hexdump with any of the known types. The command file
uses this method to identify files.

After that I think I would write a small program to check the
first N bytes of a file to see if they match your dump.





> 
> HTH
> Bill
> 
> 

[toc] | [prev] | [next] | [standalone]

#14553

In comp.os.linux.misc, Bill Cunningham <nospam@nspam.invalid> wrote:
> "Eli the Bearded" <*@eli.users.panix.com> wrote:
>> have you considered 'find' and 'grep'?
>     Can you use find and grep to scan /dev/sda1 or so and match what comes 
> through there to a hexdump of XX1 ? And get the address of the sector where 
> that file has its' beginning and end?

Sounds like you want this instead:

 	dd if=/dev/sda1 | grep -b

Elijah
------
dd syntax is odd

[toc] | [prev] | [next] | [standalone]

#14556

"Eli the Bearded" <*@eli.users.panix.com> wrote in message 
news:eli$1504141920@qz.little-neck.ny.us...
> In comp.os.linux.misc, Bill Cunningham <nospam@nspam.invalid> wrote:
>> "Eli the Bearded" <*@eli.users.panix.com> wrote:
>>> have you considered 'find' and 'grep'?
>>     Can you use find and grep to scan /dev/sda1 or so and match what 
>> comes
>> through there to a hexdump of XX1 ? And get the address of the sector 
>> where
>> that file has its' beginning and end?
>
> Sounds like you want this instead:
>
>  dd if=/dev/sda1 | grep -b

    Ok but where would the file come in. What you're saying is making sense. 
But if fileN is 250 MB and hexdump or xxd gives me a readout is just the 
first 512 bytes sufficent to search?

Bill

[toc] | [prev] | [next] | [standalone]

#14557

In comp.os.linux.misc, Bill Cunningham <nospam@nspam.invalid> wrote:
> "Eli the Bearded" <*@eli.users.panix.com> wrote:
> > Sounds like you want this instead:
> >  dd if=/dev/sda1 | grep -b
>     Ok but where would the file come in. What you're saying is making sense. 
> But if fileN is 250 MB and hexdump or xxd gives me a readout is just the 
> first 512 bytes sufficent to search?

I found you the byte offset. Everything else is filesystem dependent.
You can dig around and trace the entire fs data structures until you get
your answer.

Or you can use my original approach, find the file, get it's inode and
then where to look in the fs data structures will be more
straight-forward.

If you are doing data-forensics, the 'dd|grep' one is probably a better
path than 'find|grep', but suit yourself.

Elijah
------
a unix forensics toolkit would be a good thing to try for this problem

[toc] | [prev] | [next] | [standalone]

#14559

"Eli the Bearded" <*@eli.users.panix.com> wrote in message 
news:eli$1504142042@qz.little-neck.ny.us...

> I found you the byte offset. Everything else is filesystem dependent.
> You can dig around and trace the entire fs data structures until you get
> your answer.
>
> Or you can use my original approach, find the file, get it's inode and
> then where to look in the fs data structures will be more
> straight-forward.
>
> If you are doing data-forensics, the 'dd|grep' one is probably a better
> path than 'find|grep', but suit yourself.

    Ok yes. It is indeed data forensics.

Bill

[toc] | [prev] | [next] | [standalone]

#14566

"Eli the Bearded" <*@eli.users.panix.com> wrote in message 
news:eli$1504142042@qz.little-neck.ny.us...

> I found you the byte offset. Everything else is filesystem dependent.
> You can dig around and trace the entire fs data structures until you get
> your answer.
>
> Or you can use my original approach, find the file, get it's inode and
> then where to look in the fs data structures will be more
> straight-forward.
>
> If you are doing data-forensics, the 'dd|grep' one is probably a better
> path than 'find|grep', but suit yourself.
>
> Elijah
> ------
> a unix forensics toolkit would be a good thing to try for this problem

    Would sleuthkit be what you mean. It sounds like it might be. I hope to 
have nothing ever crash on me again now with "testdisk".

Bill

[toc] | [prev] | [next] | [standalone]

#14561

On 2015-04-14, Bill Cunningham <nospam@nspam.invalid> wrote:
>     Can you use find and grep to scan /dev/sda1 or so and match what comes 
> through there to a hexdump of XX1 ? And get the address of the sector where 
> that file has its' beginning and end?

Regardless of what tools you use, I think it would be very difficult to
do what you ask for a file that occupies multiple sectors, unless you just
happen to be lucky enough that the file occupies consecutive sectors.
Maybe I misunderstand you, but it sounds like you want to scan the data
on the hard drive while bypassing the file system, but the file system
is what knows where all the parts of a file are located on the disk.

As little as I know about hard drives and file systems, it may be that
each sector, in addition to the data it stores, also stores information
about which sector holds the next part of the file, if there is any more
to the file.  Whether it is the file system or the hard drive itself
that tracks the sectors used by each file, though, if you are trying to 
bypass the file system, it probably will take special forensic tools to 
do what you asked, just as someone suggested earlier.

You mentioned that you might want to scan for a match to a 250MB
hex-dump of a file.  Do you work for the MPAA or RIAA?    :)
-- 
                                 Chick Tower

For e-mail:  colm DOT sent DOT towerboy AT xoxy DOT net

[toc] | [prev] | [next] | [standalone]

#14563

On 15/04/15 18:09, Chick Tower wrote:
> On 2015-04-14, Bill Cunningham <nospam@nspam.invalid> wrote:
>>      Can you use find and grep to scan /dev/sda1 or so and match what comes
>> through there to a hexdump of XX1 ? And get the address of the sector where
>> that file has its' beginning and end?
>
> Regardless of what tools you use, I think it would be very difficult to
> do what you ask for a file that occupies multiple sectors, unless you just
> happen to be lucky enough that the file occupies consecutive sectors.
> Maybe I misunderstand you, but it sounds like you want to scan the data
> on the hard drive while bypassing the file system, but the file system
> is what knows where all the parts of a file are located on the disk.
>
> As little as I know about hard drives and file systems, it may be that
> each sector, in addition to the data it stores, also stores information
> about which sector holds the next part of the file, if there is any more
> to the file.  Whether it is the file system or the hard drive itself
> that tracks the sectors used by each file, though, if you are trying to
> bypass the file system, it probably will take special forensic tools to
> do what you asked, just as someone suggested earlier.
>
> You mentioned that you might want to scan for a match to a 250MB
> hex-dump of a file.  Do you work for the MPAA or RIAA?    :)
>
If the data IS in a file  and the file structure IS intact then it makes 
MORE sense to scan the filesystem as a filesystem.

With decent caching the overhead on speed wont be that long.



-- 
Everything you read in newspapers is absolutely true, except for the 
rare story of which you happen to have first-hand knowledge. – Erwin Knoll

[toc] | [prev] | [next] | [standalone]

#14565

"Chick Tower" <c.tower@deadspam.com> wrote in message 
news:mgm613$ta7$1@dont-email.me...

> Regardless of what tools you use, I think it would be very difficult to
> do what you ask for a file that occupies multiple sectors, unless you just
> happen to be lucky enough that the file occupies consecutive sectors.
> Maybe I misunderstand you, but it sounds like you want to scan the data
> on the hard drive while bypassing the file system, but the file system
> is what knows where all the parts of a file are located on the disk.
>
> As little as I know about hard drives and file systems, it may be that
> each sector, in addition to the data it stores, also stores information
> about which sector holds the next part of the file, if there is any more
> to the file.  Whether it is the file system or the hard drive itself
> that tracks the sectors used by each file, though, if you are trying to
> bypass the file system, it probably will take special forensic tools to
> do what you asked, just as someone suggested earlier.
>
> You mentioned that you might want to scan for a match to a 250MB
> hex-dump of a file.  Do you work for the MPAA or RIAA?    :)

    lol. I don't know those groups, but data analysis of filesystems is what 
I'm looking at. The question is does linux have the tools and ability t do 
it. I have a 512 byte or 1 sector volume boot record of an ntfs filesystem. 
Well we know that starts at /dev/sdaX for example. Which is /dev/sda1 on my 
machine. Begin scanning from /dev/sda. The beginning of the drive and stop 
where the "block" is. There may be utilities out there for this. Or 
something may have to be written up. But what I was wondering was "Does 
anyone know how with existing tools fi this can be done?"

Bill

[toc] | [prev] | [next] | [standalone]

#14551

On 04/14/15 16:00, Bill Cunningham wrote:
>      If I had a dump of a file. And then wanted to find it on the filesystem,
> could I use sed? I've never used this command and I am not familiar with it.
> Other than I know it exists. It is a "stream editor" whatever that is. If I
> had a dump of XX1. It's 500 MB long. Can I use sed and pipe to find it on
> the filesystem? I am running ext3 right now.
>
> Bill
>
>

Your best bet is to use a tuna fish, but it must be a tuna fish as 
mackerel just does not work for this.

Some thing here just stinks but I can not put my finger on it.

[toc] | [prev] | [next] | [standalone]

#14555

"Baho Utot" <baho-utot@columbus.rr.com> wrote in message 
news:5oh00c-lsu1.ln1@raspberry-pi.bildanet.com...

> Your best bet is to use a tuna fish, but it must be a tuna fish as 
> mackerel just does not work for this.
>
> Some thing here just stinks but I can not put my finger on it.

    It's hard for me to explain this. I know what I want to do, just no idea 
how to do it. But I think the others might understand. It's not an error.... 
IDK.

Bill

[toc] | [prev] | [next] | [standalone]

#14564

On 04/14/2015 01:00 PM, Bill Cunningham wrote:
> If I had a dump of a file. And then wanted to find it on the filesystem, 
> could I use sed?
>
  sed = "s"tream "ed"ditor
  So, no, sed is not useful here.
  If you want to search the filessytem, use "find".
  If you want to search files, use "grep".

-- 
James Moe
jmm-list at sohnen-moe dot com

[toc] | [prev] | [standalone]

Back to top | Article view | comp.os.linux.misc

csiph-web