On Single-Sequence and Multi-Sequence Factorizations
L. Zelnik-Manor and M. Irani

Subspace based factorization methods are commonly used for a variety of applications, such as 3D reconstruction, segmentation and optical flow estimation. These are usually applied to a single video sequence. In this paper we present an analysis of the multi-sequence case and place it under a single framework with the single sequence case. In particular, we start by analyzing the characteristics of subspace based spatial and temporal segmentation. We show that in many cases objects moving with different 3D motions will be captured as a single object using multi-body (spatial) factorization approaches. Similarly, frames viewing different shapes might be grouped as displaying the same shape in the temporal factorization framework (Temporal factorization provides temporal grouping of frames by employing a subspace based approach to capture non-rigid shape changes). We analyze what causes these degeneracies and show that in the case of multiple sequences these can be made useful and provide information for both temporal synchronization of sequences and spatial matching of points across sequences.

CVPR'03 paper in pdf

IJCV (to appear) paper in pdf

Video sequences used for some of the results in the paper:

Multi body dependence video
Sequence showing two objects moving with the same rotation but different translation
(0.2M avi)
Smiley sequence
Sequence showing a smiley face in two different expressions, smiling and sad, while moving with random rigid motions.
(0.01M avi)
Step camera 1
Sequence showing a person steping forward viewed by first camera.
(2M avi)
step camera 2
Sequence showing a person steping forward viewed by second camera.
(2M avi)
Step synchorinization result
Synchronization result of the two sequences above.
(2.7M avi)
Dance input video
A tiling of two video sequence each viewing a different person "dancing".
(3.4M avi)
Dance synchronization result
Synchronization result of the dance sequences above. Now the people are dancing together.
(2.2M avi)