In modern smart cities, there is a quest for the highest level of integration and automation service. Inthe surveillance sector, one of the main challenges is to automate the analysis of videos in real-time to identifycritical situations. This paper presents intelligent models based on Convolutional Neural Networks (in which theMobileNet, InceptionV3 and VGG16 networks had used), LSTM networks and feedforward networks for the taskof classifying videos under the classes "Violence" and "Non-Violence", using for this the RLVS database. Differentdata representations held used according to the Temporal Fusion techniques. The best outcome achieved was 0.91and 0.90 of Accuracy and F1-Score, respectively, a higher result compared to those found in similar researchesfor works conducted on the same database.