Mineral database
58 messages in this thread
Greetings all,
For several years, I have been creating a database of minerals for my own use,
and began toying with the idea of putting it in a format of use to others.
After going online nearly a year ago, it became obvious that there were many
people interested in such a database at an affordable price.
My database has all known (accepted by IMA) minerals (soon will have anyway,
still entering data for the new minerals from a few years ago when I let it sit
for a while). Included data is Name, Formula, System, Group, Color, Opacity,
Luster, Streak, Hardness, Density (S.G.), No. of Cleavages, Quality and
direction of Cleavages, Fracture and tenacity, Habit, and Info (things such as
fluorescence, polymorphism, etc.).
What do you want–what would be most useful to you? Please let me know, in this
forum or by private e-mail, what would be the most useful to you.
I am considering a few options. One being to provide the database only with the
data as given by the mineral describer. For searches, sometimes this is a
problem–i.e. the color given may be "rose-red" but if you search for "pink",
then you won't find a "rose-red" mineral, which most people would consider
pink. Similarly, in the habit, the description will often not fit "simple
common terms" such as "equant" or "tabular."
So, to make the database more useful for searches, I have the option of adding
descripters which have a tendency to "generalize" data, but also make it more
useful to some. So I have thought of releasing the database in two formats: 1)
all the minerals with only the data as given by the describer and 2) a
shortened list with only about 1,000 of the more "common" minerals with
"standardized" terms added to aid in searches.
Also, I have not decided if it is best to make it available as an text file
only, you put it in your database program and search/use it the way you wish,
or make it available only as a run-time database with the interface already
designed and setup (either way it will be for Dos/Windows and Mac.)
And by the way–I am in the the mineral-related publishing business to make
money to live, so this database is not free. I am thinking of a price of around
$25 for the shortened version (if it should be made) and $40 for the complete
list.
Let me know what you think, and what your needs are.
Thanks.
Lanny
Your database is most valuable, it's ripe for improvement with a friendly
interface and a good database searching engine, customized for the study of
minerals. I might be able to help with the color question, in terms of
programming. There's a technique I use in my programs called the "color naming
system." In programmer's terms, it's a subroutine that can take a limited set
of color descriptions "light yellow-green" and turn it into an RGB value,
suitable for a computer screen. It also perform the reverse, mapping colors to
English color names with great specificity. Given the wide range of actual
colors of a specific mineral, and the latitude we must extend to humans when it
comes to color naming, I'd say your database needs to be quite flexible when it
comes to color handling. I can imagine an interface that would let the user
set a color on-screen, with sliders, and then the program would use the
resulting RGB value to find matches within a certain range of that value.
>> I can imagine an interface that would let the user set a color
>> on-screen, with sliders, and then the program would use the resulting
>> RGB value to find matches within a certain range of that value.
Nice idea, John! My first inclination was to vote for the raw database, but if
the search engine has goodies like that, I like it. I think there also needs to
be a flexible synonym facility.
Al
Well, you've stated a very short and succinct description of all those
techie-words I used: it's a color synonym system. Minerals vary in color, and
it would be a necessity.
Hi, John–
The colors of minerals vary so erratically (some come in all colors of the
rainbow, relatively few have only one distinctive color), and the descriptions
Lanny must be finding in the literature have so many different notions of how
to describe them, that I suspect it is best to just give the user dozen or so
color buttons ("red", "yellow", etc.) so the user will not waste too much time
fine-tuning RGB slides which would get something more precise than the data
supports. Of course, the user may just spend a lot of time agonizing over the
choice between buttons anyway…
I hope the database application would be able to override any custom color
palettes to be sure to show the colors intended. I wonder how dependent color
display is on monitor choice?
Folks, in choosing between a raw database and one coupled with a manager
program, remember a manager can make some things easy, like finishing a word or
offering a pick list as soon as you type a few letters.
–Doug
>> waste too much time fine-tuning RGB slides
How about interfacing to a color hand scanner <G>? Automatic color input.
>> I wonder how dependent color display is on monitor choice?
It's dependent on both the monitor and the display adapter, but would
undoubtedly be good enough for the purpose. Graphics artists and others who
need precise color matching (to match standard print colors, for example) use
color standards and calibration programs.
>> choosing between a raw database and one coupled with a manager program,
>> remember a manager can make some things easy
OTOH, a manager coupled with a proprietary data base can make it impossible to
do anything that isn't built in. As long as *I'm* not doing the work, I vote
for an xbase file AND a search engine <g>.
Al
Hi, Al–
Scanning in the color – I like it! I wonder if somehow it could decide luster
also from that scan…
Right, the proprietary data base format is one reason I never bought the Aleph
Enterprises database of minerals (there were $500 other reasons). It forbids
too many possible uses for the data.
–Doug
Doug,
>> wonder if somehow it could decide luster also from that scan…
Hmmm… There should be enough information there, but you'd have to get so
close to the hardware it would probably be different for every brand. A scanner
has a lot of stuff not needed, too. I wonder if anybody markets a pocket
colorimeter? Should be able to make it about the size of a fontain pen. Use
fixed-frequency diode lasers for the color light source. On-board memory to
hold a couple of hundred measurements and a serial port to dump the data into
your desktop PC. Anybody want to invest a couple million in a new business I
just invented?
Al
Hi, Al–
As long as we are designing a specialized device, soup up that laser so it can
vaporize a tiny bit of the specimen and spectroanalyze the vapor. Now let me
start digging around on my desk, I must have left a couple of million here
someplace…
–Doug
Doug,
With the souped up laser, we could cut the specimen out of the mountain it came
in, too. I suspect the power pack is getting a bit heavy at this point.
Al
Hi, Al
> With the souped up laser, we could cut the specimen out of the mountain it
> came in, too. I suspect the power pack is getting a bit heavy at this
> point.
Rats! I'm fresh out of dilithium crystals. <G>
-Guy
No doubt a simple interface would serve many users… of course, the underlying
code techniques could still use more of a fuzzy, friendly search for ranges of
color. Of course, there's always the possibility that this database program
could someday serve as a way to gather information as well as dispense it,
recording the user's RGB values for their specimen of leverite, etc.
John,
Just curious – have you looked at the named color ID/matching stuff in
WinImages?
Ben
No, I've never seen WinImages. Did you use the CNS stuff I sent you once upon
a time?
John,
> No, I've never seen WinImages. Did you use the CNS stuff I sent you once
upon a time?
Uh-uh. Barry Chalmers, our lead programmer, developed this on his own. All I
know about it is the feel and UI – how it performs. But I still have your email
on the Amiga… 🙂
It does both way matching while the user bumps around in the palette, color
wheels, wells, luma scrollers and so on. You can add your own names, we provide
a decent set to start with. No Pantone.. we talked to them, and they wanted a
_lot_ for the color naming… not worth it, I decided. Easy to do, hard to
afford (and impossible to justify). End users can add it themselves, though…
easy as pie.
Ben
Hi, John–
Leave lots of room – someday, they will read in the color with a
color-calibrated scanner for coupling to the inboard GPS reading. If color
alone is too widely variable, maybe a narrower range of colors recorded for
each site can be a stronger identifier.
–Doug
Doug,
Remembe though, that the database serves two function–one being a method to
help determine what a mineral species is from its characteristics, the other
being like an encyclopedia to provide the information/characteristics of a
mineral.
Lanny
Hi, Lanny–
Hopefully, the database will have a lot more than 2 uses. Show me all barium
minerals for use in my report on barium; given a mineral name, add formula and
xtal system to the automatic label printout; find me a mineral in my collection
which I can use for rat poison; and so on beyond imagination.
This calls into my mind another field that could be handy someday: the precise
crystallographic spec of the mineral. Perhaps a program could automatically
generate/find crystal drawings based on that spec, or discard candidates based
on angle measurements using that spec. Without said programs, that is best
left for the pro databases, but keep it in mind for a someday project.
–Doug
Doug,
I believe your suggested uses in the first paragraph all fit nicely under my
second use–as an "encyclopedia" or source of data on a particular mineral or
minerals. But yes, hopefully it would get lots of use. It is amazing what one
comes up with when asking it to do something like "display all minerals with
Be," or whatever. (actually it chokes on "whatever").
I have been thinking of what you say. Ultimately it could be a hypertext type
of database that would link to crystal drawings, crystal form data, optic data,
photos, etc. And from there to type localities, localities, major specimens in
museums… I wonder when CD-ROMs will hold about 5x what they do now!
My original thought along this was something I proposed in Mineral News about
3-4 years ago-a photographic encyclopedia of minerals with many photos of each
mineral (as appropriate–some only occur as one grain in a polished section for
instance), and expand that idea to the hypertext type tool. The CD-ROM is the
best way and actually, George Gerhold at Western Washington Univ. has beat me
to the punch on that (sort of) he has nearly completed such a critter for the
"common" minerals.
Lanny
Hi, Lanny–
If you develop an application to go with the database, it could likely have
hooks for integrating something like an image database sold by someone else.
I have been hearing about a process soon out that will multiply the capacity of
CDs made with the process.
Any idea when that CD of mineral pix might be out and how much it will sell
for?
–Doug
The nice part is the ability to colormatch on screen, with the program fitting
the result into a range of possibilities in the DB. Within the limits of color
reproduction on the average screen, of course. OTOH, even that can be
calibrated, as I understand it (I think Pantone is one such system?).
I was also thinking of the need to provide synonyms of other mineral attributes
as well.
Pantone is for the printed page, which is a different ball of wax than
providing calibration of monitor colors…
Hi, John,
I thought there was a Pantone program to make monitor colors match the printed
page, but I may be thinking of another product. Sort of a WYSIWYG for colors.
Al
Yes, for example, illustration packages often include pre-made palettes of
monitor equivalents (in RGB) of Pantone (paper) colors. This is a complicated
subject! It's not easy to calibrate two different monitors, but that's yet
another problem. The latest Macintosh operating system software includes
special methods for calibrating monitors in order to make some of these methods
possible; so far, the Windows world isn't that sophisticated.
>> This is a complicated subject!
I can believe that!. Do you have any feel for whether the typical monitor (VGA,
1MB video RAM) is "good enough" for the rather fuzzy matching we're talking
about here?
Al
Al,
A flexible synonym facility, now we are starting to make this into real work
(so far it has only been several hundred hours, let's go for several hundred
more).
I'm not sure I want to continue reading this thread. These ideas are sounding
pretty good, but what I thought was going to be a "simple" project, do it and
its done, could become a nearly "Never Ending Story"!
A thought just came to mind on John's color idea–considering the variances in
color screens (and maybe in people's color vision/interpretation) would this
cause enough of a difficulty to be a significant problem?
Thanks for the synonym idea,
Lanny
Lanny,
I think most modern computers are fairly good, but even though I'm a computer
consultant, this isn't my area of expertise. I used to design logic circuits
for Bausch & Lomb – that's where the color-blindness experts are. Maybe we
could ask for help from the engineering section?
>> "Never Ending Story"
All the best software is <g>. We'll all be grateful for the simple version and
the bells and whistles can come later.
Al
Al,
Well as a publisher reading certain magazines, etc., all I continually see is a
lot of whining and complaining that there is no comparison in color from one
monitor to the next (and to the printer's output etc.). Maybe for our needs,
unless a user's monitor is way out of whack the colors would be close enough.
Lanny
Lanny,
I think (and from other feedback here) that most monitors would be close
enough. Most people (like me) are not really trained to see differences the way
a publisher or graphic artist would. I've heard people complain about horrible
gross mismatches that I couldn't even see. Unneccessary precision is a waste.
Al
Yeah,
As a publisher I suppose I should see some of it too, but honestly I am the
same way. I just don't see where the subtle difference makes or breaks a
photograph or design, if I see the difference at all!
As for minerals, sometimes the descriptions makes one wonder. How many people
really know what carmine red is, or verdis gris green, or even cobalt blue?
Lanny
>>don't see where the subtle difference makes or breaks a photograph or design
I'm especially embarrassed about fonts. I'm told that each of the thousands has
their use, and it's vitally important to use the right ones, but other than a
few which strike me as particulary horrible, I usually don't even notice them.
>> How many people really know what carmine red is, or verdis gris green,
>> or even cobalt blue?
I don't have any problem, as long as the labels aren't worn off the crayons.
Al
John,
That is an interesting approach I hadn't thought of. First, I think I will work
on getting a basic packing competed and available, then spend the time to do
some things like this.
I hadn't thought about applying the potential of the computer to this project
quite like your idea. There probably are some other things that could be done
too.
Thanks for the offer to help too. For the next round we shall see!
Lanny
Lanny,
The most useful item I can think of is "in what place(s), _specifically_, can I
find this stuff". 🙂
Ben
>> most useful item I can think of is "in what place(s), _specifically_
Seconded.
Al
Ben,
Of course, and with GPS data included! But that is another project (actually
been in the works for several years too). Unfortunately for some, and for the
convenience of dispensing data to "everyone" the size of that database is going
to be huge (or at least bigger than a sportscar).
Lanny
>> with GPS data included
Lanny, stop that! you're making me drool! Keep me in mind for help on the next
round, too. Since it's a public service, I won't even charge my usual
exorbitant fees.
Al
Al,
Thanks, you couldn't charge me your usual exorbitant fees anyway. I believe it
probably takes money to pay them!
But these ideas and any future help are welcome. If the first version actually
sells enough to make more than a dollar sixty nine, then the expanion of it to
the next level may take some major input/ideas and possibly some real technical
help.
Thanks to you and all who have already helped.
Lanny
Hi, Lanny and folks–
Many of us eagerly await this database!
In addition to (1) all minerals with original descriptions and (2) a thousand
minerals with standardized and original descriptions, there are other
variations possible, like (3) all minerals with standardized descriptions only,
and others.
As a text file, it could be used clumsily with a word processor as well as
spreadsheets and database programs, and is more ready for use in
spell-checking; as an Xbase (standard database) file, it could be used with
most spreadsheet and database programs. If it is in some less common database
format, options like these would disappear, and it might be constrained for use
only with the interface program Lanny speaks of. Which database manager did
you have in mind, Lanny?
Well, spell-checking could be handled by the McDonald database of names and
formulae, already available via anonymous FTP, but not compatibly with CIS FTP.
I hope to gain permission for upload or find another FTP source.
Another question relates to subscripts and other exotic characters in chemical
formulae. Lanny could keep them as unsubscripted ASCII numerals (so users
would need no exotic fonts), or maybe could be persuaded to set them up for use
with CHEMFO.EXE (in Science forum library 3, chemistry) which offers a font for
using formula characters in Windows (but not Macs, which Lanny uses), or a 3rd
system I have recently heard of, not yet generally available, which would cover
Windows and Mac.
–Doug
P.S. Lanny, my last CISmail message had been sitting in my out-basket for a
week; once in a while I forget to finish up with the "Send" button…
>> as an Xbase (standard database) file, it could be used with most
>> spreadsheet and database programs.
Actually, most modern word processors can handle Xbase files, as well, so there
isn't a serious tradeoff.
But please do make it Xbase.
Al
Al,
Being a Mac user myself, and not heavy into databases, I have forgotten what
xbase is.
For the Mac, it is so simple to transfer any data from any database or text to
any database program, just by exporting it as ASCII with a return separating
records and a tab separating fields (as I understand ad a CR for DOS). I
believe DOS is about the same, so if it was raw data, the user could set up his
database to run it any way desired. Or am I forgetting something?
Lanny
Lanny,
Xbase refers to the Dbase, Rbase, etc bunch of database programs. They are by
different publishers but use the same data structures, hence have been lumped
into the "X"base category, at least colloquially.
>> exporting it as ASCII with a return separating records and a tab
>> separating fields
True. Every database I know of can import and export ascii files with various
delimiting schemes (PC as well as Mac). There are reasons for using other
formats, including efficiency and the ability to include relationships between
items in the DB. Before any DB gurus jump on me, I know that's putting it
crudely.
However, many useful relationships can be done by selection on the ASCII info,
and it IS universal. I just don't want to see a proprietary format which nobody
can read except the provided search engine.
Al
OK,
And if I really wanted to sell it, it would have to be in a format that people
could easily get, load, etc. So I either do it right, or I'm stupid and broke
forever (and I should be able to change at least one of these two).
Once I exported it in Dbase format and it went from the 1.3 megs to something
like 1.8 (its about .864 as text).
Anyway, that is big, and already inconvenient. Actually the final product will
be somewhat smaller, I have some data in there that I am not releasing yet.
I think I might buy FoxPro and see what it can do.
Lanny
Lanny,
Foxpro's a good package. I think the text form would certainly be good enough
for the first version. Once you test the market, you have a better idea of how
much resources you want to invest.
Al
Yeah, I think it is going to be FoxPro or Cause and Effect. I guess I will know
this week, because it is time to do something!
Hi, Lanny–
In compressed form, that xBase file may be hardly larger than the text file.
That still makes it more of a pain to keep on hard disk, but any user is likely
to want to set the text version up in some database format, which would still
enlarge it.
–Doug
Hi, Lanny–
Xbase is the standard that is/was used by dBase, which was ruled public
domain. What it could do that text databases cannot do is keep the field names
and definitions, so you do not have to rename/resize/etc each time you import,
and allow the database to be multiple related tables. One table could be the
IMA-named minerals with their invariant properties, another table could relate
synonyms to the IMA-names, another could list sites and site-specific
properties of the mineral at that site, and so on. A relational database
manager could do automatic lookups (you enter a nonstandard name, it
automatically shifts to the IMA name, and that sort of thing).
–Doug
That does sound good.
BTW Doug, I am going to send you the name-formula file to run through your
formula checker again.
Lanny
Doug,
There probably isn't really much need for all minerals with standardized
descriptions. Once you get past the 600 or 1,000 "common" minerals, you can't
really identify them with the hand specimen physical data anyway, and most of
the people who will need a reference to keithconnite or boggsite will be after
the actual scientific data. (I think)
I haven't decided yet. It will have to be one that works on both Mac and
DOS/Windows and should. And, considering my incredible wealth, not cost too
much to own a copy of each and have a version to produce stand alone run time
product. I am open to consideration of opinions and suggestions. One thing to
consider, is that I would like the database to sell below $50, considering the
relatively limited mineral collect/mineralogist market, one can't spend too
much time setting up the front end, so it has to be easy to program. Not
something where one has to basically write code.
Yeah, subscripts is a problem. One solution I considered was to sell it with a
particular font designed for it.
Lanny
Hi, Lanny–
Getting an exotic-character-set font to work gracefully for many users is a
lot harder than it seems – that is what I was doing for my two years with a
super-small company. It could easily blow either your budget or your customer
satisfaction levels, unless you get it "off the shelf" (from one of the two I
mentioned, perhaps).
–Doug
Actually Doug, I already have one. Designed it a couple years ago. I would
think that a utility could change it to whatever needs to be done so it would
work with DOS/Windows and not just Mac.
Then again, I haven't even decided how it would be best to implement it for
users of this database.
Lanny
Hi, Lanny–
It is possible Windows has been around long enough for font-making utilities
to have added the needed accessories for stand-alone installation, but I would
not assume it. Perhaps by now it is enough to cover Windows rather than the
tower-of-Babel world of DOS applications. The real work is not in the font
itself but in the keyboard management and font installation. There are also
lots of tricks in choosing the character code assignments.
I have a DOS screen/keyboard/dot matrix font and driver set for super/subscript
that I hotrodded from a version of my company's product, but it is not fit for
ordinary users – that is the hardest part (besides, I do not have the right to
sell it).
Does your font depend on selecting a different font to be able to type the
super/subscripts? That is generally harder to use than a font with the exotic
characters in the extended (upper 128) character codes coupled with a keyboard
manager, and the latter allows for global search-and-replace, database queries,
and such things to distinguish between super/subscripts and regular numerals.
–Doug
Hi Lenny,
I find your ideia excelent, and I am a candidate to be your "customer".
Refering standardization of terms, why don't you allow the user of the database
to edit some of the words in a way he fancies best ? You could always provide a
read only table with your original terms (in case the user changes is mind).
Cheers from Portugal
Joao
Hi Joao,
Great, a new customer and it isn't even finished!
Yeah, good idea. Something that will have to be allowed for if presented as a
finished application. But, I may let the first version go as "data only" in
which case the user can set up his version anyway desired.
Thanks,
Lanny
Hi, Joao–
Welcome to the forum/section!
This is an excellent way to handle the question of whether to standardize or
stick to original data – if Lanny has both in separate fields, he can give us
one database with, say, the standardized terms, and a supplementary file with
just mineral names (or keys) and original info – the extra file is small and
the user can easily set it aside if it is not wanted.
–Doug
Your database, from CIM subscribers only?
No, it is to be marketed mostly outside of CIS for any and all mineral
collectors, mineralogists or others who have a need for a database of physical
characteristics of minerals at a reasonable price (under $50).
Lanny
Lanny,
Thanks for your response. How do I get the database? Sounds great and the
price is ok.
John