Question on ARRAYs
15 messages in this thread
I probably don't understand this as well as I should, but I can
declare an array of virtually any size in Basic limited only be available
memory. I believe the same is true in C. I'm trying to envisage the
programming problem when I'm trying to invert a large matrix in a stat
program where I need three large arrays. In Basic the solution is
relatively simple to do, but I get the idea that in Modula it becomes a
quite complex chore, relatively speaking. Although I've never tried this
in C, it seems to me that I recall that even in C one can declare an array
of almost any size within constraints of memory. Not being a Modula
programmer, I guess I may be seeing more in this problem than there is, but
why would any language need to impose such constraints especially for a
target machine which has large memory capacity. In my old CP/M machine it
was understandable because typical systems rarely had more than 32k of
available memory. I can see the problem on MSDOS machines because of the
64k page limitations of the processor, but even there most compilers
provide for a large code and data area to overcome the limitation. I'm
trying to understand this and maybe I just don't see the forest for the
trees, so I appreciate any insights. I'd like to learn Modula because it
seems a bit less cryptic than C and yet possesses all the programming power
of C on the Amiga. I'm not being critical, here, but simply trying to
understand.
It don't know why the 32K array limit is present. Large matrix inversion
is kind of a pain but what about this? Create a 2D array of pointers to
arrays of your data elements. If everything, including addresses, is a 2
byte value your virtual array can be a max of 8,192 elements on a side or
more than 67 million total elements. Now, my matrix math is rusty but…to
invert the whole matrix you invert each sub-array then invert the array of
array pointers. Will that work? — Art
Art,
In theory it will work, of course. But the programming to do it
becomes a bit horny. I guess I know now why most stat programs written in
Pascal or Modula generally use a disk virtual file, so it can page 32K into
and out of memory! I checked through my Lattice docs and it stated that
there was no limit on the size of an array in C other than that imposed by
memory, data, and storage type. However, I checked in some books on Modula
and apparently the 32K limit isn't uncommon to those language
implementations. I can't imagine why 'ol what's his name would have
created a language and imposed such an unnecessary limitation. I
understand the limitation on module size, but that certainly poses no
serious limitation on program implementation since it's highly unlikely
that one needs to write a single code module greater than 32K in even the
largest program. As I think of it, tho, I remember a number of Basic
implementations which had 32K limits on such things as max string length
and code size, but I think most of those were implementations with their
origins in CP/M. I guess that the practical solution is to page 32K of
data into memory at a time, then page it back out to disk. That is no more
complicated than the pointer solution and has the added feature of allowing
a datafile size limited only by disk storage. That way one could, in
theory, have a data file of extremely large size, while at the same time
not stepping over the 32K array boundary. Aha, an elegant solution with a
feature out of a "bug"!
Steve
I believe the reason for the 32K limit is not from the language but from
the implimentation using 16 bit integers as a standard. With a 16 integer,
you are limited to 32K (signed).
Bill,
Now that makes a lot of sense. Hadn't given that much thought
because the Amiga isn't limited to 16 bit integers. Still raises the
interesting question of why a need to retain the limitation for 32 bit
implementations? Oh well, it's all academic anyway!
Steve
Steve, note that swapping to disk is NOT a requirement of my scheme unless
you run out of memory – the same thing is true for C.
TYPE
ElementType = INTEGER;
LittleArray = ARRAY[0..99,0..99] OF ElementType;
LittleArrayPtr = POINTER TO LittleArray;
BigArray = ARRAY[0..99,0..99] OF LittleArrayPtr;
PROCEDURE GetElement(VAR Big : BigArray; X,Y : CARDINAL) : ElementType; VAR
LittleX, LittleY;
BigX, BigY : CARDINAL;
BEGIN
BigX := X DIV 100;
BigY := Y DIV 100;
LittleX := X MOD 100;
LittleY := Y MOD 100;
RETURN Big[BigX,BigY]^[LittleX,LittleY]; END GetElement;
Get the idea? — Art
Art,
Yep, I got the idea. However, combining your idea with the disk
swapping not only resolves the 32k limitation, but also provides for
dealing with a data file too large to fit in memory. Your code, with minor
adaptation, provides the mechanism for reading in parts of a disk file at a
time, then paging it back to disk — virtual memory. This has been one of
the features lacking in most Amiga programs, that is unless the file can
fit in memory it cannot be manipulated. Although building virtual memory
into the program is not the most elegant way to resolve this problem, it's
necessary until the OS incorporates a virtual memory handler. Even that
alone will not do the trick unless the machine has an MMU, a feature only
available on the 68020/030 boards. Of course even the disk swapping idea
is easier if we didn't need the extra code for handling the large arrays,
but that can't be done in Modula-2. I have this sick feeling that I may
have to resort to learning C on the Amiga or sticking with Basic to avoid
all this extra programming. Either that, or I spend my sabbatical this
year trying to become a decent programmer in Modula. By the time I
accomplish that, my leave will be over and I will still have an incomplete
project! In the long run, I'd be real happy if someone else would write a
good stat package for Ami. I'd rather buy one than write one!
Steve
Well I won't try to sway you toward C or Modula. In any case, VM is an
entirely new issue. Neither C nor Modula is inherently more suited to
imlementing it. I will say that if you write pseudo-VM routines (in any
language) they should be as generic as possible (i.e., independent of your
array management). — Art
Art,
What would be best, I think, would be for VM to be an inherent part
of the Amiga OS; that would provide a great benefit to all software running
on the Amiga, don't you think? Unfortunately, from what I understand, it
would only prove feasible with an MMU, which is missing from any systems
not using accelerator cards with the MMU on board. Alas! Well, as for my
problem, I'm not sure what the best direction is. Modula and C both have
positive and negative characteristics, but software maintenance and
modularity seems to weigh more favorably with Modula. The array concerns
are less important with C. Both languages represent, for me, a substantial
time commitment for learning, Modula more so than C, but the differences
are not substantial. I have the next year off for my sabbatical, so
perhaps the learning problem is an appropriate task for my leisure time!
Steve
Just a note that I agree that having VM as part of the OS sounds great. It
works much better with HW support of a MMU, so the 500 and 2000 machines
without accelerators would have problems with VM. I think C= could do what
IBM never did, two track the OS. One version with VM for systems with
accelerators and 3000's, one for other systems with the only difference
being VM. We have often wondered why IBM/MS did not do this for AT
systems.
>> Gary <<
Gary,
Well, probably cause they're still trying to figure out
multitasking! I just read that IBM will come out with a "light" version of
OS/2. It only requires a 286/386 platform and 2 meg of memory instead of 4
meg! I guess that makes the Amiga OS a dieter's delight!
Steve
Steven,
That sounds about par for Incompatable Buisness Machines. The easy
thing to do will never creap into their Ity Bitty Mindes. My respect for
IBM, MicroSoft and INTEL has gone from low down as my experience with thier
products advances. The problem is that I have to keep about 200 PC's going
at work. They are not the only ones with problems either. Our 64 user
UNISYS computer is not VM! The hardware has always been in it, but the OS
(UNIX) does not use the hardware. The MAC's are little better than OS/2
for memory consumption, and are not even multitasking! I just hope C=
continues to do the great job it has done with the AMIGA and VM would be a
great way to continue!
>> Gary <<
What kind of Unisys machine are you using? Back in the old days of 1974, I
worked for Burroughs (part of "The Power of ^2"), and they had a VM (not
the "standard" as defined by IBM) system going on machines from the B7500
down to the B700. I am not overly fascinated by UNIX at all, and cannot
see any advantage of it over the MCP that was available at that time on the
Burroughs machines. Or do you have a machine from the Univac side of the
family?
Gary,
I agree. Rumor was flying around that VM might become a reality
with the new OS once CBM finished AMIX. NOt that it would be present when
2.0 came out, but that it might be later on. The rumor seemed to suggest
that as AMIX became a reality, so might VM.
The unfortunate thing about the Amiga as a competitor with IBM and
Mac is the lack of a substantial supply of very professional and highly
polished software products with truly uniform user interfaces and full use
of data interchange. This has only been a partial accomplishment. Most
Amiga software is not as professional or extensive as the IBM or Mac
counterparts. Data interchange has never been well established through the
clipboard, as it is on the Mac. However, as time goes by, if the 3000
makes a real dent in the media market, then a growing user base should
eventually solve these problems. Now it's truly difficult to make the
Amiga a viable alternative for most business users, and that's where the
big bucks are!
Steve
This solution is just the basic solution. You must include error code to
insure that the BigArray pointers are properly initialized. This would be
a good scheme, though, where the arrays are sparse and clustered. You
would allocate a LittleArray when you have data to store in that section,
and leave the majority unallocated. Another solution might be to make
BigArray an array of character strings which contain the names of files,
and the procedure GetElement would handle all the I/O, memory loading, and
unloading. This is the strong point of modula. You can make these
changes, and do very little recompilation. It might be better to make
LittleArrayPtr a private type, then none of the client modules would ever
have to be recompiled, no matter how the problem was solved. This module
could be put in a library here, and distributed to anyone who wanted to use
it.