#Public domain text?
10 messages in this thread
Albert,
There are fundimental differences between conventional database requirements
and full text data/indexing systems. First is the obvious part, conventional
db's have (sometimes) limited fields with ranges of values. Full text just isnt
as well chunked. Then there is the indexing problems and index/data size. A
full text index needs a number of more things per item like the location of
instance. (Actually, this depends highly on the type of index you want. Some
systems only need keep a count of the # of occurances of the item in the
chunk.)
Data Size is one of the big failings of most low cost db's. They work quite
well when you have say 1-32,768 records, but when you want to go to a cdrom and
have millions of records, they just cant handle it.
Another problem is the Relational DB concept of breaking records down into
normalized tables. This is really hard on CDROM users 'cause each of those
tables is put in a different place on the disk. When you want to put the data
back together, you got to scan all over the place to assemble the data.
Jerry
(who as a freelance programmer would love to take on some of those interface &
database jobs people mentioned in this thread <g>)
Jerry,
I'm still waiting for someone to write one (full text retrevial) and place it
in public domain. Are you interested? <grin>
— Albert
Albert,
At one time there was a project to develop a public domain index
format, which would then let people just distribute the databases on CDROMs and
let the customers pick their own retrieval engines. Now that I'm interested I
of course can't find any more info.
Frank
Frank,
Its never too late to start. What kind of specifications would you need in pd
indexing program that would make you use it?
Albert,
Just a couple nasties<g> we need phrase and proximity searches, and
can't have stopwords messing up the phrases. (Basically, stopwords are
incompatible with phrase search–f'rinstance, the legalese for 'technical term'
is 'term of art.' If 'of' is a stopword, that phrase will never hit)
We could probably survive with "same sentence" and "same paragraph"
searching, but it would mean educating customers who can just about handle
turning on the machine.
The other big deal is _simple_: we're talking lawyers here. A menu
system to choose the database to search, lots of on-screen help etc. Honestly,
I wouldn't mind a PD toolkit, and I'll take it from there.
Frank
Frank,
It sounds like fun. When do we start writing the code?
— Albert
Frank:
You may be thinking about an effort started by SIGCAT a few years ago, and
headed by a guy in the Air Force (his name will come to me). I think they were
unable to come up with a protocol for the index itself, and became a group
trying to implement a generic SGML description.
In theory, a generic index structure isn't hard to conceive. One of the blocks
these guys ran into, however, was the tendency of software companies to use
proprietary compression/encryption schemes on their indices, and they are very
protective of those schemes. As you might guess, there wasn't a lot of
enthusiasm from the commercial marketplace.
Best – Rich
Some messages in the Dr. Dobbs Journal forum (GO DDJFOR) had a message about
some SMGL source code in the DTP forum. Also internet is supposed to have a
newsgroup as well.
Rich,
Have you heard of the AppleSearch technology? It allows one to have
unstructured text on a server. It sounds very exciting. I read about it in
on-line today (GO OLT-90).
Rich,
That sounds quite likely. I'm not surprised to hear about what
happened. Now that the subject is being kicked around, I have been cogitating
on what we could use, and it is getting pretty sophisticated. Basic function
looks pretty straightforward,(I could probably program it in a year or so<g>)
but getting everything to run at a decent speed would be a real would push it
out even farther. Hmm. If the fractint crew can do it…..
Frank