#sound sync.
30 messages in this thread
Hi Gary:
Is it possible to trigger sound through boards like soundblaster from within
3ds using an ipas??!!
I am talking to a few people for developing a "sound Sync" Ipas( you can see
the string of messages in Ipas wishlist lib).
No one i have talked to so far, really knows if it is possible to play music
from within 3ds!! This is somewhat important for the functionality of the ipas,
but if push comes to shove, it can do without!
Please let me know what you think.
Thanks
ara
You can basically do anything with your computer from inside of an IPAS routine
that you can do from DOS.
But I'm curious as to why you'd want to TRIGGER a sound card from within 3DS.
Since 3DS isn't a real-time app, that doesn't make sense. I'd think you'd
actually want to trigger 3DS (or make keys) from a sound source (either via
amplitude, frequency, or midi data).
– G
Yes, that is exactly what i had in mind!
The Ipas will read a .wav file and will show a graph of the sound wave.
At this point the user has the choice of either automatically (through setting
tolorances) mark the peak points or a range of points of the wave file or can
manually mark the needed points on the wave file.
Each point or range can be given a name with a limited number of characters,
say 10, (ex. "A" or "D"… or abriviate a certain event name like "explosion
start" or "explosion range"…etc).
The user also specifies a range of frames for which the sound is being
analyzed.
Once this is done, the Ipas will super impose the name of the wave points on
the corisponding frame/ frames in the KF.(No marks for frames with no wave
point)
This will give a direct visual ref. to both music beat and/or a particular
event in the music to which the user will adjust his/hers keyframing for
perfect sync.
This is the main application of the Ipas. If it can do this, then we are 80%
there!!
Where playing the music from 3ds becomes important is when you want to identify
certain points in the music.
The user will play the music through the ipas and watch a vertical bar pass
over the wave, showing the particular point on the wave that corisponds to the
sound being heard.
The user with a click on a button in the ipas(lets say this button is called
"mark") will automatically pick the point on the wave he was intrested in.
At this point the ipas pauses and opens up a window allowing the user to name
the point. Once the point is named the ipas resumes with the rest of the
music/sound.
So through realtime music play back one can manually mark and identify the
needed points in the music!!!
The second use for realtime sound playback which i think is also very
interesting is as follows:
Once the above procedure is complete and the keyframing is done, one can go
back to the ipas and take a look at a preview. The Ipas will playback, in real
time, the segment of the animation worked on and also play the music in sync
with the segment!! This will allow double checking the motion with the sound in
real time!!!
The quality of the rendering for the preview is secondary. Ofcourse phong
shaded smooth preview is the best! But wireframe will do just as good.
I believe 3ds is capable of playing previews in real time given the window of
the playback is not in full screen??!!!(a small 3x2in window will do).
There it is, Thats the Ipas! It will both allow lip sync through point
identification and music/sound sync through beat and range identification!
I strongly believe this will be a very useful Ipas and i am sure there will be
a great deal of demand for it.
Unfortunatly I have Zero knowledge about programing with C and the Ipas kit, or
i would have done it myself for sure. (i wish it was possible to learn these
languages easly on partime base)!
What do you think?!
ara
ara
It sounds like you need to start taking C programming classes! (or make
friends with an IPAS developer 🙂
Good luck!
– G
I will try if it is practical for me to do that or i will hire someone!
How difficult is learning c to a level to program Ipas for someone with my
general background???:
I have a Mech. engineering BS degree and an MBA.
For the last twelve years i have been envolved in product
development.
I am really interested in your opinion of the parcticality of such undertaking
(learning C) on part time bases??
Please let me know
thanks
ara
It all depends upon how much of a gearhead you are. If you've got a proclivity
towards building sand-castles in your mind, you can do it. It's best to find a
local school and take an into-to-C course — that'll at least give you a
first-hand feel for whether or not you want to go farther.
– G
Gary:
I am going to check UCLA extention courses for intro to c.
I am for sure going to at least do the 1st course as you suggested.
I will let you know how it goes.
This is going to be exciting!!!!
Should I order my ipas dev. kit now or wait until i have gone throught the 1st
course??!
Thanks
Ara
Thanks
Actually, with the scripting language that comes with R4, you'll be able to
write your own KXP routines without the IPAS SDK. So get started with that
course at UCLA!
Ahh… another gearhead in the making.
– G
Right on!!!!
Here i goooooo…….
Ara
Ara, the person to write to about IPAS and C programming is Jonas. I think with
time and diligence you could master it. If you haven't had any programming
experience you might want to consider taking C in a community college. Then you
need to get MetaWare C and a Phar Lap bundle. The Developers library is
continually growing with source code and of course there is a developer's kit
to assist you. Definitely worth checking into if you have the time.
John
I am going to take an extention course at UCLA for Intro to C.
Thanks for the suggestion!
This is going to be exciting
Thanks
Ara
Ara,
I envy you, UCLA has got excellent resources. I hope you really enjoy it.
Programming can be a lot of fun, and the logic and syntax will help you with
scripting in 3DS as well.
John
Thanks a lot for the support.
I hope programing will turn out to be fun for me as well.
But i know developing an ipas concept is for sure going to be both fun and
exciting.
Thanks
Ara
Fantastic idea, Ara! There is DEFINITELY a market out there…I'll be the 1st
in line. Go for it! The only way I know of now is to dub in sound through an
audio mixer when recording from computer to SVHS in real time. The obvious
problem with this approach is that it has to be done again with each SVHS
recording, and the timing has to be right or you've wasted a tape — not cheap.
Maybe you can find an investment partner (a C programmer) to work with you on
this venture…Good luck.:)
Regards, Connie
Thanks
I am going to give it my best shot.
Wish me luck!! (:-(
If i cant get it done, i am sure someone else will. So i think one way or
another we will see something like this!!
I hope I am first though.
ara
Hi Ara,
Since we talked on the phone about this… and since it seems to be a popular
idea… we here at Digimation are going to help you look into this sound idea.
I've kinda wanted to look into this ever since I saw the Weldon Pond demo at
the Alias boothe at Siggraph. So if you would like to help us bring this to
market start by putting together an outrageous wish list of everything you
would like it to do. Sketch out an interface and send it down to me. The timing
could be good because of the Visual Keyframer we're designing now that we talked
about. I can't promise anything at this point since we're deep into production
of Bones and some other goodies but let's put our heads together. Anybody else
out there, please feel free to chime in. Thanks. David – Digimation
David / ALL,
I've been very loosely following this thread, but since an IPAS developer such
as yourself has requested ideas (unusual in itself), I guess it's time to speak
up. I'm trying to type this as I think about it (in a hurry), but when I'm
done – if you or anyone else here would like to go into more specific details –
I'd be glad to provide them. I think the basic idea can be made more do-able,
but also needs to be expanded in other related ways:
First of all, there's obviously nothing new about any of this, it's been done
before, & the techniques go back about 3/4 century or so – using a sound
editor, mag tracks, wax pencils, stopwatches, metrenomes, bar sheets, &
exposure or "dope" sheets……and these techniques & materials work pretty
well. And of course anyone having created sound-synced animation without native
tools available, has done all of this manually with some of the above
materials, pencil & paper. The fundemental process of creating/adjusting
animation to the beat and/or various accents of a music track, or for
lip-syncing is well-established. Beats/accents is pretty quick & easy.
Analyzing & adjusting for speech-track syncing isn't so easy. Plus, almost no
notable feature animation is precisely matched to an audio sync track – ON
PURPOSE. A long time ago, folks at Disney determined that, for whatever reason,
the visual accent should lead the sound event, but varies with context.
The fundamental GUI layout therfore is pretty easy & should mimic those
materials & techniques historically used in feature cel animation. To try to
attemp to support all the audio h/w AND make a phonetic breakdown from a
composite audio signal is bound to be de-railing in nature wrt the
project…and really, unnecessary. Unless a discrete analog or MIDI track is
fed to the sampler, I imagine just extracting a reliable beat & accents would
be difficult…although I know it is being done.
What's REALLY needed is an all-encompassing interface template for visually
relating ALL elements of an animation to one another, with the ability to
easily adjust any parameter – INCLUDING a reference to audio tracks. This
ALONE, would greatly enhance 3DS's animation capabilities. This should consist
of a near full-screen dialog with additional pop-up dialogs, comprising an
adaptation of vertically/horizontally scrolling phonetic bar sheet &
beat/accent tracks at the top….followed by spline-graph controlled channels
for EVERY animatable parameter, including VP stuff if possible…..along with
text notation areas where appropriate & even thumbnail w/f's or renderings of
the scene/item at that frame coulmn entry. As I said, each animatable parameter
should be displayed as & controlled by a spline curve. I suppose Bezier would
be good as it's consistent with 3DS, the curve passes through the control
points, & it can be easily shaped interactively with familiar Bezier
adjustments.
NOW, this *could* have the first (top) channel read & display a sampled
composite or discrete waveform over time, which it would read from the output
of the sampling h/w & s/w. – if that would be helpful to some. Then followed by
the phonetic breakdown bar sheet channel which you can analyze & create
manually & plug in the data (text). Followed by beat/accent track that again,
you analyze manually and perhaps can be computer-assisted here with consistent
rhythms & accents. Of course, if it's not too much to ask some h/w / s/w combo
to analyze speech/music tracks for breakdown and insertion here – GREAT –
incorporate it. Or design hooks for reading this data at some future point in
time with an internal or external program.
Augmenting the interactive dialog, should be a full complement of
timeline/graph editing functions for inserting, moving, sliding, cutting,
pasting, adding, deleting single or ranges of frames….as well as controls
relating to the control curves. Also, the main dialog should contain a preview
viewport with at least a wireframe display capable of being driven manually, or
locked to a set frame/field rate. And there should be pointers to dynamically
point to the various channel displays over time – as the preview is running
under computer or manual control.
To top it all off, (and I know little about modern sound cards), if possible –
let the s/w sync with & trigger the audio playback in real-time or
proportionally-scaled near-real-time to accompany the preview. The PV can be
pre-calculated of course, if necessary. If this *IS* possible, don't worry
about supporting all of the sound hardware….pick a good one & the user will
just have to get it to use the system. If you or anyone can pull off this
control/display system, I don't think too many folks will squabble over having
to replace their card. And when considering the above description, especially
excluding the options, I really don't think this would be terribly difficult to
code. That's about it, without getting into all the fine details & specifics.
Regards,
BILL
Bill,
Get your duffle bag packed, son.
After reading your suggestions on the sound sync project, I am going to get
this project declared a National Emergency Software Development Program and you
should be very quickly receiving orders from Washington sending you TDY down to
Digimation.
Get to work. Your country needs you.
Alec Jason
Please, let's keep the government out of it <g>. I've done all that I can do,
other than get into specifics & details & testing….Can't program the thing.
There's really nothing special or fancy or original about what I described,
though. It's just the logical way to control an animated project. From what
I've seen so far, it seems very do-able via IPAS with 3DS. Actually, I'm
surprised it hasn't been done. Tempo and core 3DS stuff scratches the surface.
Thanks a lot for the comments.
BILL
Thanks, Bill! Another comprehensive and informative contribution. It goes in
my files.
Regards, Connie
You're welcome. Thnak you for the compliment.
BILL
Great!
Connie
David:
First let me thank you for the upgrade!! I just downloaded it and have not
tried it yet. I will let you know how it goes.
I will be more than happy to send you my ideas on the functionality and
inteface of this Ipas. This is great!!! I am very excited about it!!
I am going to take some courses at UCLA for programing in C right away.
But i am sure it is going to be a while before i am good enough to write Ipas
programs. So rather than trying to develope this concept on my own, i believe
it is better for someone who is already involved in programing Ipas to give it
a shot.
In the meanwhile I would be more than happy to be involved at any level. It
would be a start and a very helpful experiance for me.
Once i am good enough in C, i will hopefully be able to develope some Ipas
programs on my own. 🙂
I will send you the stuff this week and i am sure we will talk soon.
Thanks and take care
Ara
I don't think a simple amplitude display would be sufficient. Within other
sound editors, is the amplitude display enough to find the beat?
John:
I am no expert in digital sound, but I have been using windows based sound
editors for simple sound editing functions and from what i have seen a lot can
be extracted from a sound wave graph.
I am sure that one does not need to reinvent the wheel for this Ipas. Simply by
studing what has been done with some of the funkiest sound softwares out there,
a lot of puzzzels can be solved.
But there is one thing i am sure of; if the wave graph and realtime sound
playback are combined and the ipas has the capability of marking the wave on
the fly….. A lot of information can be extracted and marked by the user
including beat and speach letters.
It is the automatic non-user interactive mode of the ipas that is a little more
complicated.
Honestly, i belive that the manual mode is going to be the most useful mode of
this Ipas.
Most definitly someone with a good digital sound background is going to be
needed to complete this project.
Ara
You would be better off leaving the communication with the hardware up to some
other app. It would be too much trouble trying to support all different cards
out there. Just record the audio segment using some Windows tool and save it
out as wave or MIDI file depending what the source is. You would later load
just the file and process within 3D Studio. Processing MIDI files is a piece of
cake. Audio files, on the other hand, is a bit trickier. I've written some
software in the past to analise radio spectrum (from a home brew radio
interferometer) and it could be used for this type of analysis. The idea is to
identify certain patterns and build a vocabulary of sorts. This is a huge
database of Fourier equations. It takes quite a bit of time to process this
data though. On the other hand, a much smaller sampling rate could be used for
this purpose saving on the bandwidth. Food for thought!
Gus:
In light of all the interest and input we have seen on this string of messages,
I belive it is better that the Masters(like you) tackle this one first!!
I am just about to begin taking cources in C and then comes Ipas developers
kit! This should take a while.
So for this one I will only get involved at the concept level!
I agree that there is no need for this ipas to digitize the sound!
My suggestion was that the ipas have the ability to read predigitized wav. or
midi files and somehow give the user a visual referance to the wave graph for
marking the key points.
I also think it is important for this Ipas to have the ability to play back the
digitized sound from within 3ds, again for referancing the key events in the
music and sync previews!!
Thanks a lot for the input and information.
Ara
(p.s. A while ago i sent you a message on two problems i was having with the
PAR driver, I believe you wrote the driver. I never heard from you regarding
this issue! If you never got the message, please let me know, i will send it to
you again.)
Thanks
Ara,
<< My suggestion was that the ipas have the ability to read predigitized wav.
or midi files and somehow give the user a visual referance to the wave graph
for marking the key points. >>
wav. or midi? – I sure hope you mean wav and midi, right?? I don't think
I could use the routine if wav support isn't included.
Mark Yankee
Yes. For sure!!!
This was a typing error. I meant (*.wav) files.
sorry.
Ara
Thanks for clearing that up Ara.
Mark Yankee