Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #32825 > unrolled thread

FORTH BLOCKS and scale-able DASD File system for full-text searches

Back to article view | Back to comp.lang.forth

Contents

  FORTH BLOCKS and scale-able DASD File system for full-text searches mfanyikazi@gmail.com - 2014-08-02 04:24 -0700
    Re: FORTH BLOCKS and scale-able DASD File system for full-text searches "Rod Pemberton" <dont_use_email@xnothavet.cqm> - 2014-08-02 16:27 -0400
    Re: FORTH BLOCKS and scale-able DASD File system for full-text searches albert@spenarnc.xs4all.nl (Albert van der Horst) - 2014-08-19 12:19 +0000
    Re: FORTH BLOCKS and scale-able DASD File system for full-text searches Steve Graham <jsgrahamus@yahoo.com> - 2014-08-19 09:25 -0700

#32825 — FORTH BLOCKS and scale-able DASD File system for full-text searches

I was wondering if anyone has attempted to create a scale-able file system using FORTH blocks.

I am aware of the many short comings of direct access protocols but I desire to apply the block file system for fixed-length data. I wish to create a simple indexed DASD file system that has full-text search capabilities and that scales well into trillions of mega bytes.

I am designing an application that aims to work much like a search engine but only crawls regional-data , it's my simple attempt at organizing the web , seeing that Google is not doing a very good job where I am right now.

I believe that It's more relevant to have people find resources on a regional-based policy than a global one, since fine-grained searches always tend to be more helpful than generalized results that most searches return.

I have been looking for methods to organize crawled data and I found mainframe data-sets to be quite useful for my small application. If anyone knows of someone with a similar project which does not depend on a database , please share. Oh and don't state the obvious , (GFS , HADOOP etc) I need to know how I can use a DASD to organize the data with a full-text search solution.

Thanks.

[toc] | [next] | [standalone]

#32829

On Sat, 02 Aug 2014 07:24:33 -0400, <mfanyikazi@gmail.com> wrote:

> I was wondering if anyone has attempted to create a scale-able file  
> system using FORTH blocks.

...

> I am aware of the many short comings of direct access protocols but I  
> desire to apply the block file system for fixed-length data. I wish to  
> create a simple indexed DASD file system that has full-text search  
> capabilities and that scales well into trillions of mega bytes.

Have you talked to guys on operating system development forums,
or alt.os.development, about the merits, advantages, and design
of filesystems?  E.g., a recent link came up on alt.os.development
on virtual filesystems and their implementation.


Rod Pemberton

[toc] | [prev] | [next] | [standalone]

#32966

In article <f2d733fe-aa15-4f2b-8e30-09aade6350fe@googlegroups.com>,
 <mfanyikazi@gmail.com> wrote:
>I was wondering if anyone has attempted to create a scale-able file system using FORTH blocks.
>
>I am aware of the many short comings of direct access protocols but I desire to apply the block
>file system for fixed-length data. I wish to create a simple indexed DASD file system that has
>full-text search capabilities and that scales well into trillions of mega bytes.
>
>I am designing an application that aims to work much like a search engine but only crawls
>regional-data , it's my simple attempt at organizing the web , seeing that Google is not doing
>a very good job where I am right now.
>
>I believe that It's more relevant to have people find resources on a regional-based policy than
>a global one, since fine-grained searches always tend to be more helpful than generalized
>results that most searches return.
>
>I have been looking for methods to organize crawled data and I found mainframe data-sets to be
>quite useful for my small application. If anyone knows of someone with a similar project which
>does not depend on a database , please share. Oh and don't state the obvious , (GFS , HADOOP
>etc) I need to know how I can use a DASD to organize the data with a full-text search solution.

I've been thinking about this too. If you call it a scale-able file system using
SD-cards (which is about the same) you will have a much larger audience/feedback.

You could base it on the FREE ALLOCATE system from ciforth.
If you want to try this you can mail me for a more heavily commented version than
the 2 screens in ciforth 5.1 (see my sig.)
You would start each item on a block boundary, a simple change. Then
reserve a fixed amount at the start of each block for administration,
instead of the single pointer to the free space after the
space-in-use. This again should be simple.

This should be speedy for storing streamed data, much better than the
FAT based cards, at the expense of estimating up front how much space
you need.

>
>Thanks.

Groetjes Albert
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

[toc] | [prev] | [next] | [standalone]

#32973

mfanyikazi@gmail.com wrote:
> I was wondering if anyone has attempted to create a scale-able file
 > system using FORTH blocks.
>
> I am aware of the many short comings of direct access protocols but
 > I desire to apply the block file system for fixed-length data. I
 > wish to create a simple indexed DASD file system that has full-text
 > search capabilities and that scales well into trillions of mega bytes.
>
> I am designing an application that aims to work much like a search
 > engine but only crawls regional-data , it's my simple attempt at
> organizing the web , seeing that Google is not doing a very good job
> where I am right now.
>
> I believe that It's more relevant to have people find resources on a
> regional-based policy than a global one, since fine-grained searches
> always tend to be more helpful than generalized results that most
 > searches return.
>
> I have been looking for methods to organize crawled data and I found
> mainframe data-sets to be quite useful for my small application. If
 > anyone knows of someone with a similar project which does not depend
 > on a database , please share. Oh and don't state the obvious , (GFS ,
 > HADOOP etc) I need to know how I can use a DASD to organize the data
 > with a full-text search solution.
>
> Thanks.
>

The language I've used for most of my career, MUMPS, uses a b-tree for 
organizing data.  Very good for searching.  I started years ago to code 
such in Forth and for a competent Forth programmer, I don't think it 
would take much time to do it.


Steve

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.forth

csiph-web