Abstract Motion Capture Assisted Animation Texturing and Synthesis

KatherinePullenStanfordUniversityChristophBreglerStanfordUniversityAbstract

2Keyframed Data0−2−4−6−8−10−1200.20.40.60.811.21.41.61.82translation in inchesWediscussamethodforcreatinganimationsthatallowstheanima-tortosketchananimationbysettingasmallnumberofkeyframesonafractionofthepossibledegreesoffreedom.Motioncapturedataisthenusedtoenhancetheanimation.Detailisaddedtode-greesoffreedomthatwerekeyframed,aprocesswecalltexturing.Degreesoffreedomthatwerenotkeyframedaresynthesized.Themethodtakesadvantageofthefactthatjointmotionsofanartic-ulatedfigureareoftencorrelated,sothatgivenanincompletedataset,themissingdegreesoffreedomcanbepredictedfromthosethatarepresent.

CRCategories:I.3.7[ComputerGraphics]:Three-DimensionalGraphicsandRealism—Animation;J.5[ArtsandHumantities]:performingartsKeywords:synthesis

animation,motioncapture,motiontexture,motion

(a)Motion Capture Data10−1−2−3−4−500.20.40.60.811.21.41.61.82(b)1Introduction

time in secondsAstheavailabilityofmotioncapturedatahasincreased,therehasbeenmoreandmoreinterestinusingitasabasisforcreatingcom-puteranimationswhenlife-likemotionisdesired.However,therearestillanumberofdifficultiestoovercomeconcerningitsuse.Asaresult,mosthighqualityanimationsarestillcreatedbykeyfram-ingbyskilledanimators.

Animatorsusuallyprefertousekeyframesbecausetheyallowprecisecontrolovertheactionsofthecharacter.However,creatingalife-likeanimationbythismethodisextremelylaborintensive.Iftoofewkeyframesareset,themotionmaybelackinginthedetailweareusedtoseeinginlivemotion(figure1).Thecurvesthataregeneratedbetweenkeyposesbycomputerareusuallysmoothsplinesorotherformsofinterpolation,whichmaynotrepresentthewayalivehumanoranimalmoves.Theanimatorcanputinasmuchdetailasheorshewants,eventothepointofspecifyingthepositionateverytime,butmoredetailrequiresmoretimeandeffort.Asecondreasonkeyframingcanbeextremelylaborintensiveisthatatypicalmodelofanarticulatedfigurehasover50degreesoffreedom,eachofwhichmustbepainstakinglykeyframed.

Motioncapturedata,ontheotherhand,providesallthedetailandnuanceoflivemotionforallthedegreesoffreedomofachar-acter.However,ithasthedisadvantageofnotprovidingforfull

Figure1:Comparisonofkeyframeddataandmotioncapturedataforrootytranslationforwalking.(a)keyframeddata,withkeyframesindicatedbyreddots(b)motioncapturedata.Inthisex-ample,thekeyframeddatahasbeencreatedbysettingtheminimumpossiblenumberofkeystodescribethemotion.Noticethatwhileitisverysmoothandsinusoidal,themotioncapturedatashowsir-regularitiesandvariations.Thesenaturalfluctuationsareinherenttolivemotion.Aprofessionalanimatorwouldachievesuchdetailbysettingmorekeys.Copyright © 2002 by the Association for Computing Machinery, Inc.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212-869-0481 or e-mail permissions@acm.org. © 2002 ACM 1-58113-521-1/02/0007 $5.00

controloverthemotion.Motioncapturesessionsarelaborinten-siveandcostly,andiftheactordoesnotdoexactlywhattheanima-torhadinmindorifplanschangeafterthemotioncapturesession,itcanbedifficultandtimeconsumingtoadaptthemotioncapturedatatothedesiredapplication.Amoresubtleproblemwithmotioncapturedataisthatitisnotanintuitivewaytobeginconstructingananimation.Animatorsareusuallytrainedtousekeyframes,andwilloftenbuildananimationbyfirstmakingaroughanimationwithfewkeyframestosketchoutthemotion,andaddcomplexityanddetailontopofthat.Itisnoteasyorconvenientforananimatortostartcreatingananimationwithadetailedmotionheorshedidnotcreateandknoweveryaspectof.Weproposeamethodforcombiningthestrengthsofkeyframeanimationwiththoseofusingmotioncapturedata.Theanimatorbeginsbycreatingaroughsketchofthesceneheorsheiscreatingbysettingasmallnumberofkeyframesonafewdegreesoffree-dom.Ourmethodwillusetheinformationinmotioncapturedatatoadddetailtothedegreesoffreedomthatwereanimatedifde-sired,aprocesswecalladding“texture”tothemotion.Degreesoffreedomthatwerenotkeyframedatallaresynthesized.Theresultisananimationthatdoesexactlywhattheanimatorwantsitto,buthasthenuanceoflivemotion.501

2RelatedWork

Therehasbeenagreatdealofpastresearchinanumberofdifferentareasthatarerelatedtoourproject.Wedividethisworkintofourmaincategoriesthataredescribedbelow.

Wealsoareinterestedinadaptingmotioncapturedatatodiffer-entsituations.However,ratherthanstartingwiththelivedata,westartwithasketchcreatedbytheanimatorofwhatthefinalresultshouldbe,andfitthemotioncapturedataontothatframework.Asaresult,itcanbeusedtocreatemotionssubstantiallydifferentfromwhatwasintheoriginaldata.

2.1VariationsinAnimationManyotherresearchersbeforeushavemadetheobservationthatpartofwhatgivesatextureitsdistinctivelook,beitinclothorinmotion,arevariationswithinthetexture.Thesevariationsareof-tenreferredtoasnoise,andoneoftheearliestpaperstoaddressthistopicwasinimagetexturesynthesis,whererandomvariabilitywasaddedtotextureswiththePerlin-noisefunction[Perlin1985].Theseideaswerelaterappliedtoanimations[PerlinandGoldberg1996].Otherresearchershavecreatedmotionofhumansrunningusingdynamicalsimulations[Hodginsetal.1995]andappliedhandcraftednoisefunctions[Bodenheimeretal.1999].Statisticalvari-ationsinmotionwereextracteddirectlyfromdatabysamplingkernel-basedprobabilitydistributionsin[PullenandBregler2000].Herewealsocreateanimationsthatexhibitnaturalvariations,inthiscasebecausetheyareinherenttothefragmentsofmotioncap-turedatathatweuseintexturingandsynthesis.

2.4Styleandsynthesis2.2SignalProcessingThereareanumberofearlierstudiesinwhichresearchersinbothtexturesynthesisandmotionstudieshavefoundittobeusefultolookattheirdatainfrequencyspace.Inimagetexturesynthesis,oneoftheearliestsuchapproachesdividedthedataintomulti-levelLaplacianpyramids,andsyntheticdatawerecreatedbyahistogrammatchingtechnique[HeegerandBergen1995].Thisworkwasfur-therdevelopedbyDeBonet[1997],inwhichthesynthesistakesintoaccountthefactthatthehigherfrequencybandstendtobecondi-tionallydependentuponthelowerfrequencybands.

Weincorporateasimilarapproach,butappliedtomotiondata.Inanimation,Unumaetal.[1995]usefourieranalysistomanipulatemotiondatabyperforminginterpolation,extrapolation,andtransi-tionaltasks,aswellastoalterthestyle.BruderlinandWilliams[1995]applyanumberofdifferentsignalprocessingtechniquestomotiondatatoallowediting.LeeandShin[2001]developamul-tiresolutionanalysismethodthatguaranteescoordinateinvarianceforuseinmotioneditingoperationssuchassmoothing,blending,andstitching.Ourworkrelatestotheseanimationpapersinthatwealsousefrequencybandsasausefulfeatureofthedata,butweusethemtosynthesizemotiondata.

2.3MotionEditingManytechniqueshavebeenproposedthatstartwithexistingmo-tions,oftenobtainedfrommotioncapturedata,andvarythemo-tionstoadapttodifferentconstraintswhilepreservingthestyleoftheoriginalmotion.WitkinandPopovic[1995]developedamethodinwhichthemotiondataiswarpedbetweenkeyframe-likeconstraintssetbytheanimator.ThespacetimeconstraintsmethodoriginallycreatedbyWitkinandKass[1988]wasdevelopedtoal-lowtheanimatortospecifyconstraintssuchasfeetpositionsofacharacter,andthensolvefortheseconstraintsbyminimizingthedifferencefromtheoriginaldata[Gleicher1997].

Infurtherwork,thismethodwasappliedtoadaptasetofmotiondatatocharactersofdifferentsizes[Gleicher1998],andcombinedwithamultiresolutionapproachforinteractivecontrolofthere-sult[LeeandShin1999].PhysicswereincludedinthemethodofPopovicandWitkin[1999],inwhichtheeditingisperformedinareduceddimensionalityspace.

Numerousotherprojectsbesidesourshaveaddressedtheproblemofsynthesizingmotionsoralteringpre-existingmotionstohaveaparticularstyle.AMarkovchainmontecarloalgorithmwasusedtosamplemultipleanimationsthatsatisfyconstraintsforthecaseofmulti-bodycollisionsofinanimateobjects[ChenneyandForsyth2000].Inworkwithsimilargoalstooursbutappliedtoimage-basedgraphics,otherresearchers[Schodletal.2000]developtheconceptofavideotexture,whichenablesausertobeginwithashortvideoclipandthengenerateaninfiniteamountofsimilarlookingvideo.Montecarlotechniquesareusedtoaddressthestochasticnatureofthetexture,andappropriatetransitionsarefoundinthemotiontocreatealoop.Themethodwasappliedtoexamplemo-tionsthatcontainbotharepetitiveandstochasticcomponent,suchasfireoraflagblowinginthewind.

Inotherprojects,acommonmethodofrepresentingdatahasbeentousemixturesofGaussiansandhiddenMarkovmodels.Bre-gler[1997]hasusedthemtorecognizefullbodymotionsinvideosequences,andBrand[1999]hasusedthemtosynthesizefacialan-imationsfromexamplesetsofaudioandvideo.BrandandHertz-mann[2000]havealsousedhiddenMarkovmodelsalongwithanentropyminimizationsproceduretolearnandsynthesizemotionswithparticularstyles.Ourapproachdiffersinthatwestrivetokeepasmuchoftheinformationinthemotioncapturedataintactaspos-sible,bydirectlyusingfragmentsofrealdataratherthangeneral-izingitwithrepresentationsthatmaycausesomeofthefinedetailtobelost.Inotherinterestingwork,Chiandhercolleagues[Chietal.2000]presentedworkwithsimilargoalstoours,inthattheywereseekingtocreateamethodthatallowsanimatorstoenhancethestyleofpre-existingmotionsinanintuitivemanner.TheymadeuseoftheprinciplesofLabanMovementAnalysistocreateanewinterfaceforapplyingparticularmovementqualitiestothemotion.Morerecently,therehavebeenanumberofprojectsaimedto-wardallowingananimatortocreatenewanimationsbasedonmo-tioncapturedata.Forexample,intheworkofLietal.[2002],thedatawasdividedintomotiontextons,eachofwhichcouldbemodelledbyalineardynamicsystem.Motionsweresynthesizedbyconsideringthelikelihoodofswitchingfromonetextontothenext.Otherresearchersdevelopedamethodforautomaticmotiongenerationatinteractiverates[ArikanandForsyth2002].Heretheanimatorsetshighlevelconstraintsandarandomsearchalgorithmisusedoffindappropriatepiecesofmotiondatatofillinbetween.Incloselyrelatedwork,theconceptofamotiongraphisdefinedtoenableonetocontrolacharacters’slocomotion[Kovaretal.2002].Themotiongraphcontainsoriginalmotionandautomaticallygen-eratedtranslations,andallowsausertohavehighlevelcontroloverthemotionsofthecharacter.Intheworkof[Leeetal.2002],anewtechniqueisdevelopedforcontrollingacharacterinrealtimeusingseveralpossibleinterfaces.Theusercanchoosefromfromasetofpossibleactions,sketchapathonthescreen,oractoutthemotioninfrontofavideocamera.Animationsarecreatedbysearchingthroughamotiondatabaseusingaclusteringalgorithm.Anyoftheabovetechniqueswouldbemoreappropriatetousethanoursinthecasewheretheuserhasalargedatabaseofmotionsandwantshighlevelcontrolovertheactionsofthecharacter.Ourprojectisgearedmoretowardananimatorwhomayhavealimitedsetofdataofaparticularstyle,andwhowantstohavefinecontroloverthemotionusingthefamiliartoolsofkeyframing.

502

Correlation Between Joint Angles20Keyframed DataMotion Capture DataAnkle angle in degrees100−10−20−30−40−30−20−100102030405060Hip angle in degreesFigure2:Correlationbetweenjointangles.Shownistheankleangleversusthehipangleforhumanwalkingdata.Thefactthatthisplothasadefiniteformdemonstratesthattheanglesarerelatedtoeachother.

Figure3:Frequencyanalysis.Shownarebands2-7ofaLaplacianPyramiddecompositionofthelefthipanglefordancemotionsfrombothkeyframingandmotioncapture.Oneband,shownwithareddashedline,ischosenforthematchingstep3Methods

Inhumanandanimalmotion,therearemanycorrelationsbetweenjointactions.Thesecorrelationsareespeciallyclearforarepetitivemotionlikewalking.Forexampleastherightfootstepsforward,theleftarmswingsforward,orwhenthehipanglehasacertainvalue,thekneeangleismostlikelytofallwithinacertainrange.Wecanseethosecorrelationsgraphicallywithaplotsuchasthatshowninfigure2,whereweplottheankleangleasafunctionofhipangleforsomehumanwalkingdata.Thefactthattheplothasaspecificshape,askewedhorseshoeshapeinthiscase,indicatesthatthereisarelationshipbetweentheangles.

Theserelationshipsholdtrueformorecomplexmotionsaswell,butmaybemorelocalintime,specifictoaparticularactionwithinamotiondataset.Inourmethodwetakeadvantageoftheserela-tionshipstosynthesizedegreesoffreedomthathavenotbeenani-mated.Similarly,wecanadddetailtoadegreeoffreedomthathasbeenanimatedbysynthesizingonlythehigherfrequencybands,aprocesswerefertoastexturing.

Theanimatormustprovidethefollowinginformation:(1)whichjointanglesshouldbetextured(2)whichjointanglesshouldbesynthesized(3)whichjointanglesshouldbeusedtodrivethemo-tionineachcase.Forexample,supposeananimatorsketchesoutawalkbyanimatingonlythelegsandwantstosynthesizetheupperbodymotions.Agoodchoicefortheanglestodrivetheanimationwouldbethehipandkneexangles(wherewedefinethexaxisashorizontal,perpendiculartothedirectionofwalking)becausetheydefinethewalkingmotion.Thesedataarebrokenintofragments,andusedtofindfragmentsofthemotioncapturedatawithhipandkneexanglessimilartowhathasbeencreatedbykeyframing.Thecorrespondingfragmentsofmotioncapturedatafortheupperbodymotioncanthenbeusedtoanimatetheupperbodyofthecomputercharacter.

Toachievethistask,werequireamethodtodeterminewhatcon-

stitutesamatchingregionofdata.Theproblemiscomplicatedbythefactthatthekeyframeddatamaybeofadifferenttimescalefromtherealdata.Inaddition,theendsofthefragmentswechoosemustjointogethersmoothlytoavoidhighfrequencyglitchesinthemotion.Weaddresstheseissuesinourmethod,whichwedivideintothefollowingsteps:(1)frequencyanalysis(2)matching(3)pathfindingand(4)joining.

Inthefollowingexplanation,wewillusetheexampleofusingthelefthipandleftkneexanglestosynthesizeupperbodymo-tions.Thesedegreesoffreedomwillbereferredtoasthematchingangles.Alsonotethatwedefine“keyframeddata”asthedataatev-erytimepointthathasbeengeneratedintheanimationaftersettingthekeyframes.Anexampleofsuchdataisinfigure1a.

3.1FrequencyAnalysisInordertoseparatedifferentaspectsofthemotion,thefirststepistodividethedata(bothkeyframedandmotioncapture)intofrequencybands(figure3).Forajointthathasalreadybeenanimated,wemayonlywanttoalterthemidtohighfrequencyrange,leavingtheoverallmotionintact.Foradegreeoffreedomthathasnotbeenanimated,wemaywishtosynthesizeallofthefrequencybands.

3.2MatchingMatchingisattheheartofourmethod.Itistheprocessbywhichfragmentsofdatafromthekeyframedanimationarecomparedtofragmentsofmotioncapturedatatofindsimilarregions.Tobeginthisstep,alowfrequencybandofoneofthejointsischosen,inourexamplethelefthipangle.Theresultsarenothighlydependentuponwhichfrequencybandischosen,aslongasitislowenoughtoprovideinformationabouttheoverallmotion.Forexampleinfigure3weillustratechosingband6oftheLaplacianpyramid,butchosingband4or5alsoyieldsgoodresults.Band7istoolow,ascanbeseenbythelackofstructureinthecurve,andband3istoohigh,asitdoesnotreflecttheoverallmotionwellenough.

Wefindthelocationsintimewherethefirstderivativeofthechosenbandofoneofthematchingangleschangessign.Therealandkeyframeddataofallofthematchinganglesofthatband(the

503

Keyframed Data(a)Motion Capture Data(b)(a)210−1−2−3210−1−2−3(b)1020304020406080(c)2210−1−2−3(d)Angle in degrees(c)(d)10−1−2−31020304010203040Time in Seconds X 24Figure4:Breakingdataintofragments.Thebandsofthekeyframeddataandmotioncapturedatashownwithreddashedlinesinfigure3arebrokenintofragmentswherethesignofthefirstderivativechanges.(a)keyframeddata.(b)motioncapturedata.(c)keyframeddatabrokenintofragments.(d)motioncap-turedatabrokenintofragments.

lefthipandleftkneexangles,inourexample)arebrokenintofrag-mentsatthoselocations(figure4).Notethatinthefiguresweil-lustratetheprocessforjustoneofthematchingangles,thehip,butactuallytheprocessisappliedtoallofthematchinganglessimul-taneously.Wealsomatchthefirstderivativeofthechosenbandofeachoftheseangles.Includingthefirstderivativesinthematchinghelpschoosefragmentsofrealdatathataremorecloselymatchednotonlyinvaluebutindynamicstothekeyframeddata.Notethatthesignchangeofthefirstderivativeofonlyoneoftheanglesisusedtodeterminewheretobreakallofthedatacorrespondingtothematchinganglesintofragments,sothatallarebrokenatthesamelocations.

Allofthefragmentsofkeyframeddatainthechosenfrequencybandandtheirfirstderivativesaresteppedthroughonebyone,andforeachweaskwhichfragmentofrealdataismostsimilar(fig-ure5).Toachievethiscomparison,westretchorcompresstherealdatafragmentsintimebylinearlyresamplingthemtomakethemthesamelengthasthekeyframedfragment.Inthemotioncapturedata,thereareoftenunnaturalposesheldforrelativelylongperiodsoftimeforcalibrationpurposes.Toavoidchosingthesefragments,anyrealfragmentthatwasoriginallymorethan4timesaslongasthefragmentofkeyframeddatabeingmatchedisrejected.Wefindthesumofsquareddifferencesbetweenthekeyframedfragmentbeingmatchedandeachoftherealdatafragments,andkeeptheKclosestmatches.Aswesavefragmentsofthematchingangles,wealsosavethecorrespondingfragmentsoforiginalmotioncap-turedata(notjustthefrequencybandbeingmatched)foralloftheanglestobesynthesizedortextured(figure6).

Atthispoint,itissometimesbeneficialtoincludeasimplescalefactor.LetAbethem×nmatrixofvaluesinthekeyframeddatabeingmatched,wheremisthenumberofmatchinganglesandnisthelengthofthosefragments.LetMbethem×nmatrixofoneoftheKchoicesofmatchingfragments.Thentoscalethedata,welookforthescalefactorsthatminimizes󰀂Ms−A󰀂.Thefactorsisthenmultipliedbyallofthedatabeingsynthesized.Inpracticesuchascalefactorisusefulonlyinalimitedsetofcases,becauseitassumesalinearrelationshipbetweenthemagnitudeofthematch-

Figure5:Matching.Eachkeyframedfragmentiscomparedtoallofthemotioncapturefragments,andtheKclosestmatchesarekept.Shownistheprocessofmatchingthefirstfragmentshowninfig-ure4(c).(a)Thekeyframedfragmenttobematched.(b)Thekeyframedfragment,showninathickblueline,comparedtoallofthemotioncapturefragments,showninthinblacklines.(c)Sameas(b),butthemotioncapturedfragmentshavebeenstretchedorcompressedtobethesamelengthasthekeyframedfragment.(d)Sameas(c),butonlythe5closestmatchesareshown.

inganglesandthemagnitudeoftherestoftheangles,whichisnotusuallylikelytobetrue.However,itcanimprovetheresultinganimationsforcasesinwhichthekeyframeddataissimilartothemotioncapturedata,andtheactionisfairlyconstrained,suchaswalking.

Morefragmentsthanjusttheclosestmatcharesavedbecausethereismoretoconsiderthanjusthowclosethedatafragmentistotheoriginal.Wemusttakeintoconsiderationwhichfragmentscomebeforeandafter.Wewouldliketoencouragetheuseofcon-secutivechunksofdataasdescribedinthenextsection.

3.3PathfindingNowthatwehavetheKclosestmatchesforeachfragment,wemustchooseapaththroughthepossiblechoicestocreateasingledataset.Theresultinganimationisusuallymorepleasingiftherearesectionsintimewherefragmentsthatwereconsecutiveinthedataareusedconsecutivelytocreatethepath.Asaresult,ouralgorithmconsiderstheneighborsofeachfragment,andsearchesforpathsthatmaximizetheuseofconsecutivefragments.

Foreachjoinbetweenfragments,wecreateacostmatrix,theijthcomponentofwhichgivesthecostforjoiningfragmentiwithfragmentj.Ascoreofzeroisgivenifthefragmentswereconsec-utiveintheoriginaldata,andoneiftheywerenot.Wefindallofthepossiblecombinationsoffragmentsthatgothroughthepointsofzerocost.

Thistechniqueiseasiesttoexplainusinganexample,whichisdiagrammedinfigure7.Supposewehad4fragmentsofsyntheticdatatomatch,andsaved3nearestmatches.Intheillustrationweshowthatforfragment1ofthekeyframeddata,thebestmatchesweretofragments4,1,and3oftherealdata,andforfragment2ofthekeyframeddatatheclosestmatchesweretofragments5,7,and2oftherealdata,andsoon.Wehavedrawnlinesbetweenfragmentstoindicatepathsofzerocost.Heretherearethreebest

504

210−1−2050100150200250(a)KeyframedFragment1413011110111257211011111139461111101114285Angles in Degrees403020100−100403020100−1005010015020025050100150200250(b)Matching DataFragments(c)CostMatriciesTime in Seconds X 24Figure6:Matchingandsynthesis.(a)Thefiveclosestmatchesforaseriesoffragmentsofkeyframeddataisshown.Thekeyframeddataisshownwithathickblueline,thematchingmotioncapturefragmentsareshownwiththinblacklines.(b)Anexampleofoneoftheanglesbeingsynthesizedisshown,thelowestspinejointanglerotationaboutthexaxis.Thefivefragmentsforeachsectioncomefromthespinemotioncapturedatafromthesamelocationintimeasthematchinghipanglefragmentsshowninplot(a).(c)Anex-ampleofapossiblepaththroughthechosenspineanglefragmentsisshownwithathickredline.

choices.Oneisfragment4,5,6,and2fromtherealdata.Inthiscasewechoosefragment2oftherealdatatomatchthefourthfrag-mentofkeyframeddataratherthan8or5becauseitwasoriginallytheclosestmatch.Asecondpossiblepathwouldbe4,5,4,and5,andathirdwouldbe1,2,4,5.Allthreewouldyieldtwoinstancesofzerocost.Anexampleofanactualpathtakenthroughfragmentschosenbymatchingisshowninfigure6c.

Notethatforzinstancesofzerocost,therecanbenogreaterthanzpathstoconsider,andinfactwillusuallybefarlessbecausetheinstancescanbelinkedup.Inourexample(figure7)therewerefourinstancesofzerocost,butonlythreepossiblepathsthatmin-imizethecost.ThePbestpaths(wherePisaparametersetbytheanimator)aresavedfortheanimatortolookat.Allarevalidchoicesanditisanartisticdecisionwhichisbest.

Inpracticewefoundthatsavingroughly1/10thetotalnumberoffragmentsproducedgoodresults.Savingtoomanymatchesre-sultedinmotionsthatwerenotcoordinatedwiththerestofthebody,andsavingtoofewdidnotallowforsufficienttemporalco-herencewhenseekingapaththroughthefragments.

Figure7:Choosingapathbymaximizingtheinstancesofcon-secutivefragments.Inthetableweshowahypotheticalexampleofacasewherefourkeyframedfragmentswerematched,andtheK=3closestmatchesofmotioncapturefragmentswerekeptforeachkeyframedfragment.Thematchesatthetopsofthecolumnsaretheclosestofthe3matches.Bluelinesaredrawnbetweenfrag-mentsthatwereconsecutiveinthemotioncapturedata,andthecostmatriciesbetweeneachsetofpossiblematchesareshownbelow.

Thenextstepistoskewthefragmenttopassthroughthenewendpoints.Toachievethiswarping,wedefinetwolines,onethatpassesthroughtheoldendpoints,andonethatpassesthroughthenewendpoints.Wesubtractthelinethatpassesthroughtheoldendpointsandaddthelinethatpassesthroughthenewendpointstoyieldtheshiftedfragment.Theprocessisdiagramedinfigure8.Inordertofurthersmoothanyremainingdiscontinuity,aquadraticfunctionisfittothejoinregionfromNpointsawayfromthejointpointtowithin2pointsofthejoinpoint,whereNisaparameter.AsmallervalueofNkeepsthedatafrombeingalteredtoogreatlyfromwhatwasinthemotioncapturedata,andalargervaluemoreeffectivelyblendsbetweendifferentfragments.Inprac-ticewefoundaNfrom5-20tobeeffective,correspondingto0.2-0.8seconds.Theresultingquadraticisblendedwiththeoriginaljoineddatausingasinesquaredfunctionasfollows.Definetheblendfunctionfas

f=(cos

πt

+1)22N(1)whereNishalfthelengthoftheshortestofthefragmentsthataretobejoinedandtisthetime,shiftedtobezeroatthejoinregion.Ifwedefineqasthequadraticfunctionweobtainedfromthefit,andmasthedataaftermatching,thenthedatasaftersmoothingiss(t)=f(t)q(t)+(1−f(t))m(t).(2)3.4JoiningAnexampleofthisprocessisshowninfigure9.Nowthatwehavethebestpossiblepaths,theendsmaystillnotquitelineupincaseswherethefragmentswerenotoriginallycon-secutive.Forexample,infigure6cweshowanexampleofdataaftermatchingandchoosingthepaths.Totakecareofthesedis-continuities,wejointheendstogetherbythefollowingprocess.Forfragmenti,wedefinethenewendpointsasfollows.Thenewfirstpointwillbethemeanbetweenthefirstpointoffragmentiandthelastpointoffragmenti−1.(Notethatthereisoverlapbetweentheendsofthefragments;ifthelastpointoffragmentiisplacedattimet,thefirstpointoffragmenti+1isalsoattimet.)Thenewlastpointoffragmentiwillbethemeanbetweenthelastpointoffragmentiandthefirstpointoffragmenti+1.

4ExperimentsWetestedourmethodonseveraldifferentsituations,threeofwhicharedescribedbelow.Alloftheseexamplesarepresentedontheaccompanyingvideotape.4.1Walking

Ashortanimationoftwocharacterswalkingtowardeachother,slowingtoastop,stomping,andcrouchingwascreatedusing505

302520151050−52040608010012030(a)2520151050−5(b)204060801001203030Angle in Degrees2520151050−5(c)2520151050(d)Figure10:Exampleframesfromthewalkinganimations.Onthetoprowaresomeframesfromthekeyframedsketch,andonthebottomrowarethecorrespondingframesafterenhancement.

2040608010012020406080100120−5Time in Seconds x 24Figure8:Joiningtheendsofselectedfragments.(a)Fourfragmentsofspineangledatathatwerechoseninthematchingstepareshown.Notethisgraphisacloseupviewofthefirstpartofthepathillus-tratedinfigure6c.Therearesignificantdiscontinuitiesbetweenthefirstandsecondfragments,aswellasbetweenthethirdandfourth.(b)Theoriginalendpointsofthefragmentsaremarkedwithblackcircles,thenewendpointsaremarkedwithbluestars.Thesecondandthirdfragmentswereconsecutiveinthemotioncapturedata,sothenewandoldendpointsarethesame.(c)Foreachfragment,thelinebetweentheoldendpoints(blackdashes)andthelinebetweenthenewendpoints(bluesolidline)areshown.(d)Foreachfrag-ment,thelinebetweentheoldendpointsissubtracted,andthelinebetweenthenewendpointsisadded,toyieldthecurveofjoinedfragments.Thenewendpointsareagainmarkedwithbluestars.

keyframes.Keyframesweresetonlyonthepositionsoftheroot(nottherotations)andfeet.Inversekinematicswereusedonthefeetattheanklejoint,asiscustomaryinkeyframeanimationofar-ticulatedfigures.Thejointanglesforthehipsandkneeswerereadoutafterwardsforuseintexturingandsynthesis.

Eachcharacter’smotionwasenhancedusingadifferentmotioncapturedatasetofwalkingmotion.Thetwodatasetseachcon-sistedofwalkingwithroughlyasinglestepsize,buteachexhibitedaverydifferentstyleofwalking.Onewasarelatively“normal”walk,butratherbouncy,andtheotherwasofapersonimitatingadragqueenandwasquitestylized,containingunusualarmandheadgestures.Thelengthofeachdatasetwas440timepointsat24framespersecond,orabout18secondsworthofdata.ALapla-cianpyramidwasusedforthefrequencyanalysis.The4thhighestbandwasusedformatching.Fortexturing,bands2-3weresynthe-sized,andforsynthesis,allbands2andlower.Theveryhighestfrequencybandtendedtoaddonlyundesirablenoisetothemotion.Theupperbodydegreesoffreedomcouldsuccessfullybesyn-thesizedusinganumberofdifferentcombinationsforthematchingangles.Forexample,bothhipxangles;thelefthipxandleftkneexangle;ortherighthipxandrightkneexallgavegoodresults.Themostpleasingresultswereobtainedbyusingdatafromthelefthipandleftkneexanglesduringthestomp(thecharacterstompshisleftfoot)anddatafrombothhipsfortherestoftheanimation.Scalingaftermatchingalsoimprovedtheresultsinthiscase,forexamplewhenthecharacterslowsdownandcomestoastop,scal-ingcausedthemotionofthebodyandarmmotionstoreduceincoordinationwiththelegs.

Themethoddoesnotdirectlyincorporatehardconstraints,soweusedthefollowingmethodtomaintainthefeetcontactwiththefloor.Firstthepelvisandupperbodymotionsweresynthesized.Sincealteringthepelvisdegreesoffreedomcauseslargescalemo-tionsofthebody,inversekinematicconstraintsweresubsequentlyappliedtokeepthefeetinplaceonthefloor.Thisnewmotionwasusedfortexturingthelowerbodymotionduringtimesthefeetwerenotincontactwiththefloor.

Themotionofthecharacterswasmuchmorelife-likeafteren-hancement.Theupperbodymovedinarealisticwayandrespondedappropriatelytothevaryingstepsizesandthestomp,eventhoughthesemotionswerenotexplicitinthemotioncapturedata.Inad-dition,thestyleofwalkingforeachcharacterclearlycamefromthedatasetusedfortheenhancement.Someexampleframesareshowninfigure10.

4(a)20−2−4510152025303540Joined DataQuadratic Fit4Angle in Degrees(b)20−2−4510152025303540Joined DataSmoothed DataTime in Seconds x 24Figure9:Smoothingatthejoinpoint.Acloseupofthejoinbe-tweenfragments1and2fromfigure8isshownwitharedsolidline.(a)Thequadraticfitusingthepointsoneithersideofthejoinpoint(asdescribedinthetext)isshownwithablackdashedline.(b)Thedataafterblendingwiththequadraticfitisshownwithabluedashedline.

506

Figure11:Exampleframesfromanimationsoftheottercharac-ter.Onthetoprowaresomeframesfromtheoriginalkeyframedanimation,whileonthebottomarethecorrespondingframesaftertexturing.

Figure12:Exampleframesfromthedanceanimations.Thebluecharacter,ontheleftineachimage,representsthekeyframedsketch.Thepurplecharacter,ontherightineachimage,showsthemotionafterenhancement.

didnotactuallydo.Someexampleframesareshowninfigure12.Thebestresultswereobtainedbyusingallofthehipandkneean-glesasthematchingangles,butsomegoodanimationscouldalsobecreatedusingfewerdegreesoffreedom.Intheseexperiments,theeffectsofchoosingdifferentpathsthroughthematcheddatabe-cameespeciallynoticeable.Becauseofthewidevariationwithinthedata,differentpathsyieldedsignificantlydifferentupperbodymotions,allofwhichwerewellcoordinatedwiththelowerbody.

4.2OtterCharacterAlthoughwehavefocussedontheideaoffillinginmissingdegreesoffreedombysynthesisoraddingdetailbytexturing,themethodcanalsobeusedtoalterthestyleofanexistinganimationthatal-readyhasalargeamountofdetailinit.

Totestthispossibility,weusedanottercharacterthathadbeenanimatedbykeyframeanimationtorun.Usingthemotioncapturesetsofwalkingdescribedabove,wecouldaffectthestyleofthecharacter’srunbytexturingtheupperbodymotions,usingthehipandkneeanglesasthematchingangles.Theeffectwasparticularlynoticeablewhenusingthedragqueenwalkfortexturing,theottercharacterpickedupsomeoftheheadbobsandasymmetricalarmusageofthemotioncapturedata.Someexampleframesareshowninfigure11.

5ConclusionandFutureWork

4.3ModernDanceInordertoinvestigateawiderrangeofmotionsthanthoserelatedtowalkingorrunning,weturnedtomoderndance.Unlikeotherformsofdancesuchasballetorotherclassicalforms,moderndoesnothaveasetvocabularyofmotions,andyetitusesthewholebodyatitsfullrangeofmotion.Thusitprovidesasituationwherethecorrelationsbetweenjointswillexistonlyextremelylocallyintime,andastringenttestofourmethod.

Amoderndancephrasewasanimatedbysketchingonlythelowerbodyandrootmotionswithkeyframes.Motioncapturedataofseveralphrasesofmoderndancewascollected,andatotalof1097timepoints(24framespersecond)from4phraseswasused.Theupperbodywassynthesized,andthelowerbodytextured.Thesamemethodformaintainingfeetcontactwiththefloorthatwasdescribedaboveforthewalkingexperimentswasusedhere.Thefrequencyanalysiswasthesameasforthewalking,exceptthatthe6thhighestbandwasusedformatching.Alowerfrequencybandwasusedbecausethelargemotionsinthedancedatasettendedtohappenoverlongertimesthanthestepsinwalking.

Theresultswerequitesuccessfulhere,especiallyforsynthesisoftheupperbodymotions.Themotionswerefullandwellcoor-dinatedwiththelowerbody,andlookedlikesomethingthedancerwhoperformedforthemotioncapturesessioncouldhavedone,but

Presentlythetwomainmethodsbywhichthemotionsforcomputeranimationsarecreatedarebyusingkeyframesandbyusingmotioncapture.Themethodofkeyframesislaborintensive,buthasthead-vantageofallowingtheanimatorprecisecontrolovertheactionsofthecharacter.Motioncapturedatahastheadvantageofprovidingacompletedatasetwithallthedetailoflivemotion,buttheanimatordoesnothavefullcontrolovertheresult.Inthisworkwepresentamethodthatcombinestheadvantagesofbothmethods,byallowingtheanimatortocontrolaninitialroughanimationwithkeyframes,andthenfillinmissingdegreesoffreedomanddetailusingthein-formationinmotioncapturedata.Theresultsareparticularlystrik-ingforthecaseofsynthesis.Onecancreateananimationofonlythelowerbody,andgivensomemotioncapturedata,automaticallycreatelife-likeupperbodymotionsthatarecoordinatedwiththelowerbody.

Onedrawbackofthemethodasitcurrentlystandsisthatitdoesnotdirectlyincorporatehardconstraints.Asaresultthetextur-ingcannotbeappliedtocaseswherethefeetaremeanttoremainincontactwiththefloor,unlessitwerecombinedwithaninversekinematicsolverintheanimationpackagebeingused.Currentlyweareworkingtoremedythisdeficiency.

Anotheractiveareaofresearchistodetermineamorefunda-mentalmethodforbreakingthedataintofragments.Inthisworkweusedthesignchangeofthederivativeofoneofthejointan-glesusedformatching,becauseitissimpletodetectandoftenrep-resentsachangefromonemovementideatoanother.Theexactchoiceofwheretobreakthedataintofragmentsisnotasimportantasitmayseem.Whatisimportantisthatboththekeyframedandrealdataarebrokenatanalogouslocations,whichisclearlythecasewithourmethod.Themethodcouldbemademoreefficientbyde-tectingmorefundamentalunitsofmovementthatmayyieldlargerfragments.However,duetothecomplexityofhumanmotion,this

507

problemisachallengingone,andanongoingtopicofresearch.Onthesurface,anotherdrawbackofthemethodmayappeartobetheneedtochooseaparticularfrequencybandforthematchingstep.However,thechoiceisnotdifficulttomake,andinfactthere-sultsarenothighlydependentuponthechoice.Anylowfrequencybandthatprovidesinformationabouttheoverallmotionwillpro-videgoodresults.Theresultinganimationssometimesvaryfromoneanotherdependinguponwhichfrequencybandischosenasslightlydifferentregionsofdataarematched,butmoreoftentheyarequitesimilar.Iftoohighabandischosen,however,theresult-inganimationhasanuncoordinatedlooktoit,astheoverallmotionisnotaccuratelyrepresentedinthematchingstep.

Similarly,anotherdrawbackofthemethodmayappeartobethattheanimatormustspecifywhichdegreesoffreedomtouseasthematchingangles.However,ifonehasspentsometimekeyframingacharacter,itisquiteeasyinpracticetospecifythisinformation.Themostsimplisticapproachistosimplyuseallofthedegreesoffreedomthattheanimatorhassketchedoutthemotionforwithkeyframes.Inmanycases,however,fewerdegreesoffreedomcanbespecified,andequallygoodresultscanbeobtained.Ifthemotionhaslessvariation,suchaswalking,theresultswillstillbepleasingiffeweranglesarechosenasthematchingangles.Infactitisfas-cinatinghowcorrelatedthemotionsofthehumanbodyare.Givenonlythedataintwoangles,suchasthehipandkneexanglesofoneleg,onecansynthesizetherestofthebody.However,foramotionwithmorevariationinit,suchasdancing,itisbettertoincludemoreangles,toensuregoodchoicesduringmatching.Iffewerjointsareusedformatchinginthiscase,someoftheresult-ingpathsmaystillbegoodresults,butothersmayappearsomewhatuncoordinatedwiththefullbody.

Infact,thegoalofthisprojectwasnottocreateacompletelyautomaticmethod,buttogivetheanimatoranothertoolforincor-poratingtheinformationinmotioncapturedataintohisorhercre-ations.Differentchoicesofthematchinganglescanyielddifferentresultsandprovidetheanimatorwithdifferentpossibilitiestouseinthefinalanimation.Anothersourceofdifferentmotionscomesfromexaminingdifferentpathsthroughthebestmatches.Theani-matorhastheoptionoflookingatseveralpossibilitiesandmakinganartisticdecisionwhichisbest.Ultimatelywehopethatmethodssuchasthisonewillfurtherallowanimatorstotakeadvantageofthebenefitsofmotioncapturedatawithoutsacrificingthecontroltheyareusedtohavingwhenkeyframing.

BREGLER,C.1997.Learningandrecognizinghumandynamicsinvideosequences.Proc.CVPR,569–574.BRUDERLIN,A.,ANDWILLIAMS,L.1995.Motionsignalpro-cessing.proc.SIGGRAPH1995,97–104.CHENNEY,S.,ANDFORSYTH,D.A.2000.Samplingplausiblesolutionstomulti-bodyconstraintproblems.proc.SIGGRAPH2000,219–228.CHI,D.,COSTA,M.,ZHAO,L.,ANDBADLER,N.2000.Theemotemodelforeffortandshape.proc.SIGGRAPH2000,173–182.GLEICHER,M.1997.motioneditingwithspacetimeconstraints.1997SymposiumonInteractive3Dgraphics,139–148.GLEICHER,M.1998.Retargetingmotiontonewcharacters.proc.SIGGRAPH1998,33–42.HEEGER,D.J.,ANDBERGEN,J.R.1995.Pyramid-basedtextureanalysis/sysnthesis.proc.SIGGRAPH1995,229–238.HODGINS,J.,WOOTEN,W.L.,BROGAN,D.C.,ANDO’BRIEN,J.F.1995.Animatinghumanathletics.proc.SIGGRAPH1995,229–238.KOVAR,L.,GLEICHER,M.,ANDPIGHIN,F.2002.Motiongraphs.Proc.SIGGRAPH2002.LEE,J.,ANDSHIN,S.Y.1999.Ahierarchicalapproachtointer-activemotioneditingforhuman-likefigures.proc.SIGGRAPH1999,39–48.LEE,J.,ANDSHIN,S.Y.2001.Acoordinate-invariantapproachtomultiresolutionmotionanalysis.GraphicalModels63,2,87–105.LEE,J.,CHAI,J.,REITSMA,P.S.A.,HODGINS,J.K.,ANDPOLLARD,N.S.2002.Interactivecontrolofavatarsanimatedwithhumanmotiondata.Proc.SIGGRAPH2002.LI,Y.,WANG,T.,ANDSHUM,H.2002.Motiontexture:Atwo-levelstatisticalmodelforcharactermotionsynthesis.Proc.SIGGRAPH2002.PERLIN,K.,ANDGOLDBERG,A.1996.Improv:asystemforscriptinginteractiveactorsinvirtualrealityworlds.proc.SIG-GRAPH1996,205–216.PERLIN,K.1985.Animagesynthesizer.ComputerGraphics19,3,287–296.POPOVIC,Z.,ANDWITKIN,A.1999.Physicallybasedmotiontransformation.proc.SIGGRAPH1999,159–168.PULLEN,K.,ANDBREGLER,C.2000.Animatingbymulti-levelsampling.IEEEComputerAnimationConference,36–42.SCHODL,A.,SZELISKI,R.,SALESIN,D.H.,ANDESSA,I.2000.Videotextures.Proc.SIGGRAPH00,489–498.UNUMA,M.,ANJYO,K.,ANDTEKEUCHI,R.1995.Fourierprinciplesforemotion-basedhumanfigureanimation.proc.SIG-GRAPH1995,91–96.WITKIN,A.,ANDKASS,M.1988.Spacetimeconstraints.Com-puterGraphics22,159–168.WITKIN,A.,ANDPOPOVIC,Z.1995.Motionwarping.proc.SIGGRAPH1995,105–108.

6Acknowledgments

SpecialthankstoReardenSteelstudiosforprovidingthemotioncapturedata,andtoElectronicArtsforprovidingtheottermodel.

References

ARIKAN,O.,ANDFORSYTH,D.A.2002.Interactivemotiongenerationfromexamples.Proc.SIGGRAPH2002.BODENHEIMER,B.,SHLEYFMAN,A.,ANDHODGINS,J.1999.Theeffectsofnoiseontheperceptionofanimatedhumanrun-ning.ComputerAnimationandSimulation’99,EurographicsAnimationWorkshop(September),53–63.BONET,J.S.D.1997.Multiresolutionsamplingprocedureforanalysisandsynthesisoftextureimages.proc.SIGGRAPH1997,361–368.BRAND,M.,ANDHERTZMANN,A.2000.Stylemachines.proc.SIGGRAPH2000,183–192.BRAND,M.1999.Voicepuppetry.Proc.SIGGRAPH1999,21–28.

508