Skip to search boxSkip to navigationSkip to main content

Deep Learning-Based Context-Aware Video Content Analysis on IoT Devices

  • Gad Gad(corresponding author)
    ,
  • Eyad Gad
    ,
  • ,
  • Zubair Fadlullah
    ,
  • Bassem Mokhtar
*Corresponding author for this work
  • Lakehead University
    ,
  • Nile University
    ,
  • Trakya University
    ,
  • University of Fujairah
    ,
  • Thunder Bay Regional Health Research Institute
    ,
  • Alexandria University
Research Output:
Contribution to journal
Article
Research
Peer-review

Open access

Abstract

Integrating machine learning with the Internet of Things (IoT) enables many useful applica-tions. For IoT applications that incorporate video content analysis (VCA), deep learning models are usually used due to their capacity to encode the high-dimensional spatial and temporal representations of videos. However, limited energy and computation resources present a major challenge. Video captioning is one type of VCA that describes a video with a sentence or a set of sentences. This work proposes an IoT-based deep learning-based framework for video captioning that can (1) Mine large open-domain video-to-text datasets to extract video-caption pairs that belong to a particular domain. (2) Preprocess the selected video-caption pairs including reducing the complexity of the captions’ language model to improve performance. (3) Propose two deep learning models: A transformer-based model and an LSTM-based model. Hyperparameter tuning is performed to select the best hyperparameters. Models are evaluated in terms of accuracy and inference time on different platforms. The presented framework generates captions in standard sentence templates to facilitate extracting information in later stages of the analysis. The two developed deep learning models offer a trade-off between accuracy and speed. While the transformer-based model yields a high accuracy of 97%, the LSTM-based model achieves near real-time inference.

Research Output Information

Output type

Research Output:
Contribution to journal
Article
Research
Peer-review

Original language

English

Article number

1785

Journal (Volume, Issue Number)

Electronics (Switzerland) (Volume 11, Issue 11)

Publication milestones

  • Published - 01/06/2022

Publication status

Published - 01/06/2022

Publication IDs

  • Scopus: 85131188064

Publication metrics

Metrics

Scopus
citations
SciVal
FWCI
0.39

PlumX, opens in new tab

Captures
15
Citation count
8