Perceiving meaningful activities in a long video sequence is a challengingproblem due to ambiguous definition of 'meaningfulness' as well as clutters inthe scene. We approach this problem by learning a generative model for regularmotion patterns, termed as regularity, using multiple sources with very limitedsupervision. Specifically, we propose two methods that are built upon theautoencoders for their ability to work with little to no supervision. We firstleverage the conventional handcrafted spatio-temporal local features and learna fully connected autoencoder on them. Second, we build a fully convolutionalfeed-forward autoencoder to learn both the local features and the classifiersas an end-to-end learning framework. Our model can capture the regularitiesfrom multiple datasets. We evaluate our methods in both qualitative andquantitative ways - showing the learned regularity of videos in various aspectsand demonstrating competitive performance on anomaly detection datasets as anapplication.