Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #257618 > unrolled thread

sha256sum --text generating blank spaces and hyphens?

Back to article view | Back to linux.debian.user

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  sha256sum --text generating blank spaces and hyphens? Albretch Mueller <lbrtchx@gmail.com> - 2023-04-26 09:50 +0200
    Re: sha256sum --text generating blank spaces and hyphens? <tomas@tuxteam.de> - 2023-04-26 10:20 +0200
      Re: sha256sum --text generating blank spaces and hyphens? Albretch Mueller <lbrtchx@gmail.com> - 2023-04-26 16:40 +0200
        Re: sha256sum --text generating blank spaces and hyphens? Nicolas George <george@nsup.org> - 2023-04-26 16:50 +0200
          Re: sha256sum --text generating blank spaces and hyphens? Max Nikulin <manikulin@gmail.com> - 2023-04-28 04:30 +0200
            Re: sha256sum --text generating blank spaces and hyphens? Nicolas George <george@nsup.org> - 2023-04-28 10:10 +0200
              Re: sha256sum --text generating blank spaces and hyphens? Max Nikulin <manikulin@gmail.com> - 2023-04-28 18:50 +0200
                Re: sha256sum --text generating blank spaces and hyphens? Nicolas George <george@nsup.org> - 2023-04-29 11:00 +0200
                  Re: sha256sum --text generating blank spaces and hyphens? Max Nikulin <manikulin@gmail.com> - 2023-04-29 14:20 +0200
                    Re: sha256sum --text generating blank spaces and hyphens? Nicolas George <george@nsup.org> - 2023-04-29 14:30 +0200
        Re: sha256sum --text generating blank spaces and hyphens? Dan Ritter <dsr@randomstring.org> - 2023-04-26 17:10 +0200
        Re: sha256sum --text generating blank spaces and hyphens? David Wright <deblis@lionunicorn.co.uk> - 2023-04-26 17:10 +0200
        Re: sha256sum --text generating blank spaces and hyphens? "Thomas Schmitt" <scdbackup@gmx.net> - 2023-04-26 17:10 +0200
        Re: sha256sum --text generating blank spaces and hyphens? <tomas@tuxteam.de> - 2023-04-26 19:00 +0200
          Re: sha256sum --text generating blank spaces and hyphens? Albretch Mueller <lbrtchx@gmail.com> - 2023-04-26 22:40 +0200
            Re: sha256sum --text generating blank spaces and hyphens? Albretch Mueller <lbrtchx@gmail.com> - 2023-04-26 23:10 +0200
            Re: sha256sum --text generating blank spaces and hyphens? Andy Smith <andy@strugglers.net> - 2023-04-26 23:50 +0200
    Re: sha256sum --text generating blank spaces and hyphens? Andy Smith <andy@strugglers.net> - 2023-04-26 12:50 +0200
    Re: sha256sum --text generating blank spaces and hyphens? Jeffrey Walton <noloader@gmail.com> - 2023-04-26 18:00 +0200
    Re: sha256sum --text generating blank spaces and hyphens? David Christensen <dpchrist@holgerdanske.com> - 2023-04-27 00:50 +0200
      Re: sha256sum --text generating blank spaces and hyphens? Nicolas George <george@nsup.org> - 2023-04-27 00:50 +0200
        Re: sha256sum --text generating blank spaces and hyphens? David Christensen <dpchrist@holgerdanske.com> - 2023-04-27 06:10 +0200
          Re: sha256sum --text generating blank spaces and hyphens? Max Nikulin <manikulin@gmail.com> - 2023-04-27 08:00 +0200
            Re: sha256sum --text generating blank spaces and hyphens? Albretch Mueller <lbrtchx@gmail.com> - 2023-04-27 09:30 +0200
          Re: sha256sum --text generating blank spaces and hyphens? Nicolas George <george@nsup.org> - 2023-04-27 10:10 +0200
            Re: sha256sum --text generating blank spaces and hyphens? David Christensen <dpchrist@holgerdanske.com> - 2023-04-27 17:20 +0200
              Re: sha256sum --text generating blank spaces and hyphens? Albretch Mueller <lbrtchx@gmail.com> - 2023-05-02 02:10 +0200
      Re: sha256sum --text generating blank spaces and hyphens? Albretch Mueller <lbrtchx@gmail.com> - 2023-04-27 01:30 +0200
        Re: sha256sum --text generating blank spaces and hyphens? David Christensen <dpchrist@holgerdanske.com> - 2023-04-27 06:30 +0200

Page 2 of 2 — ← Prev page 1 [2]

#257646

David Christensen (12023-04-26):
> I suggest hashing the document content rather than the URL.  This would work
> nicely for static documents.

That will be very convenient to retrieve the document content from the
URL.

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#257651

On 4/26/23 15:48, Nicolas George wrote:
> David Christensen (12023-04-26):
>> I suggest hashing the document content rather than the URL.  This would work
>> nicely for static documents.
> 
> That will be very convenient to retrieve the document content from the
> URL.


My suggestion assumes that the URL => hash => content mapping is saved 
somehow.  For example, save the content in a file named after the hash 
and save the URL in a file whose name is the hash plus a suffix. 
Finding a document by URL then becomes a grep(1) invocation.


Things get more interesting when you approach the problem as a database. 
  Save the content wherever and put the metadata into a table -- content 
hash (primary key), URL, download timestamp, author, subject, title, 
keywords, etc..  Create fully inverted indexes.  Create a search engine. 
  Create a spider.  Implementation could range from a CSV/TSV flat-file 
and shell/P* scripts, to a desktop database/UI, to a LAMP stack, and 
beyond (NoSQL, N-tier).  There are distributed file sharing systems 
based on such ideas.


David

[toc] | [prev] | [next] | [standalone]

#257655

On 27/04/2023 11:02, David Christensen wrote:
> Things get more interesting when you approach the problem as a database. 
>   Save the content wherever and put the metadata into a table -- content 
> hash (primary key), URL, download timestamp, author, subject, title, 
> keywords, etc..  Create fully inverted indexes.  Create a search engine. 
>   Create a spider.  Implementation could range from a CSV/TSV flat-file 
> and shell/P* scripts, to a desktop database/UI, to a LAMP stack, and 
> beyond (NoSQL, N-tier).  There are distributed file sharing systems 
> based on such ideas.

I have never tried: "Open-source self-hosted web archiving"
https://github.com/ArchiveBox/ArchiveBox

This one allows to save selected part of a page:
https://github.com/danny0838/webscrapbook/

[toc] | [prev] | [next] | [standalone]

#257656

On 4/27/23, Max Nikulin <manikulin@gmail.com> wrote:
> I have never tried: "Open-source self-hosted web archiving"
> https://github.com/ArchiveBox/ArchiveBox
>
> This one allows to save selected part of a page:
> https://github.com/danny0838/webscrapbook/

 Thank you for keeping me busy! From their recommendations:

 https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community

 https://en.wikipedia.org/wiki/List_of_Web_archiving_initiatives

 https://coptr.digipres.org/index.php/Main_Page
~
 However, what I have in mind is definitely more than archiving, which
would be only the first phase of it.

// __ [Corpora-List] towards a "pan document format" (pun intended) . . .

 https://list.elra.info/mailman3/hyperkitty/list/corpora@list.elra.info/message/4AULI3UUQ7BQG5ANFYGEEL7FXQXIILYN/
~
 In particular, I am interested in a corpus of "universally appealing writers"

// __ list of authors and their work ...

 https://list.elra.info/mailman3/hyperkitty/list/corpora@list.elra.info/thread/5PFZUBNLRWW2FDHDWHPKZYOMAGZLOWXG/#4BTSFS5OCUFWVWU4ZDSBJ765DQFWWI7B/
~
 lbrtchx

[toc] | [prev] | [next] | [standalone]

#257657

David Christensen (12023-04-26):
> My suggestion assumes that the URL => hash => content mapping is saved
> somehow.

That is an assumption that needed to be made explicit from the start.

>	   For example, save the content in a file named after the hash and
> save the URL in a file whose name is the hash plus a suffix. Finding a
> document by URL then becomes a grep(1) invocation.

This is not very efficient.

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#257664

On 4/27/23 01:04, Nicolas George wrote:
> David Christensen (12023-04-26):
>> My suggestion assumes that the URL => hash => content mapping is saved
>> somehow.
> 
> That is an assumption that needed to be made explicit from the start.
> 
>> 	   For example, save the content in a file named after the hash and
>> save the URL in a file whose name is the hash plus a suffix. Finding a
>> document by URL then becomes a grep(1) invocation.
> 
> This is not very efficient.


Please see the OP, step (d).


You are free to propose better solutions.


On 4/26/23 21:02, David Christensen wrote:

 > Things get more interesting when you approach the problem as a database.
 >   Save the content wherever and put the metadata into a table -- content
 > hash (primary key), URL, download timestamp, author, subject, title,
 > keywords, etc..  Create fully inverted indexes.  Create a search engine.
 >   Create a spider.  Implementation could range from a CSV/TSV flat-file
 > and shell/P* scripts, to a desktop database/UI, to a LAMP stack, and
 > beyond (NoSQL, N-tier).  There are distributed file sharing systems
 > based on such ideas.


David

[toc] | [prev] | [next] | [standalone]

#257829

On 4/27/23, David Christensen <dpchrist@holgerdanske.com> wrote:
> Please see the OP, step (d).

>On 4/26/23, Albretch Mueller <lbrtchx@gmail.com> wrote:
>>  a) encode the string name as base64
>> b) calculate the sha256sum of §a
>>  c) use §b as file name (of course, leaving the original extension as it
>> is)
>>  d) include a "§b_file_name.txt" plain text file descriptor which only
>> content is the actual prehash name of that file.

 I do that because base64 would (must?) work on any OS and the
conversion from and to any other encoding is straightforward. As you
suggested, I am more friendly to the idea of including hashes of the
data payload, even though I think it is not that important, because
the actual big problem that corpora research people have is files with
exactly the same look and feel and the same content which have
different hashes (for example, pdf files). I have been thinking about
a way to compute hashes which resemble more faithfully, both,
structural and content similarity among files. Do you know of any way
to do such thing? The structural aspect should be "easy". It could be
handled as DAGs of some sort of XPaths.

  I was actually going to show to you what I meant, but I was happy to
see "I was wrong". I even waited to try it from some other access
point. I have used this one liner to show how
google/youtube/NSA/"Vladimir Putin"/... was watermarking files for
whatever reason, but it worked fine when I was trying to show it to
you ;-)

_YT_URI=EngW7tLk6R8; _OFL="${_YT_URI}_"$(date +%Y%m%d%H%M%S)".mp4";
./yt-dlp --verbose --format "mp4" --output "${_OFL}" -- "${_YT_URI}";
ls -l "${_OFL}"; file --brief "${_OFL}"; time sha256sum "${_OFL}"


-rwxrwxrwx 1 user user 828540 Aug 15  2022 EngW7tLk6R8_20230501185618.mp4
ISO Media, MP4 v2 [ISO 14496-14]
0b950b88667b5fec35f3dd54005c16e5e742c703a0c776ec6da11b60a4775ae6
EngW7tLk6R8_20230501185618.mp4

-rwxrwxrwx 1 user user 828540 Aug 15  2022 EngW7tLk6R8_20230501185657.mp4
ISO Media, MP4 v2 [ISO 14496-14]
0b950b88667b5fec35f3dd54005c16e5e742c703a0c776ec6da11b60a4775ae6
EngW7tLk6R8_20230501185657.mp4

 Max Nikulin (12023-04-28):
> And you will quickly face servers that sends incorrectly Content-Type or
> intentionally put application/octet-stream with no sniff header to force
> browser to save the file instead of opening it e.g. in built-in PDF
> reader.

 Even if not totally syntactic (so you can't functionally solve it
with some code), this is a relatively manageable problem, you would:

 a) take notice of the sites that do such things;
 b) sniff not only the http headers, but notice the file extension of
the file; and
 c) safe the file to a temp repository for the Linux util "file" to be
run on it ...

 Out of those heuristics you should be able to strategize around such problems.

 lbrtchx

[toc] | [prev] | [next] | [standalone]

#257647

On 4/26/23, David Christensen <dpchrist@holgerdanske.com> wrote:
> I suggest hashing the document content rather than the URL.  This would
> work nicely for static documents.

 What do you mean by "hashing the document content"?

 How would that help when what you are trying to do is cleanse and
canonize texts as best as you could to find relationships among their
text segments?

 lbrtchx

[toc] | [prev] | [next] | [standalone]

#257652

On 4/26/23 16:21, Albretch Mueller wrote:
> On 4/26/23, David Christensen <dpchrist@holgerdanske.com> wrote:
>> I suggest hashing the document content rather than the URL.  This would
>> work nicely for static documents.
> 
>   What do you mean by "hashing the document content"?


2023-04-26 21:03:08 dpchrist@taz ~
$ touch foo

2023-04-26 21:03:12 dpchrist@taz ~
$ sha256sum foo
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  foo


In this case, the content is an empty string and the hexadecimal 
encoding of the the SHA256 hash is 
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855".

> 
>   How would that help when what you are trying to do is cleanse and
> canonize texts as best as you could to find relationships among their
> text segments?
> 
>   lbrtchx


* Each unique text would be stored once regardless of how many URL's 
link to it.

* If the content at a URL changes, the new content will have a new hash. 
  So, the new content will be saved and the old content will be 
preserved (instead of the new content overwriting the old content).

* With regard to my response to the post by Nicolas George, a database 
of metadata could benefit analysis regardless of the scheme used to name 
content files.


David

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.user

csiph-web