Abstract
Since I was at the University of Illinois at Urbana-Champaign in the 1990s, I have been conducting multimedia research and witnessing the development of this field for nearly 30 years. The advancement of deep learning, which gave rise to a multimodality-based algorithm framework, has pushed forward and will continue to push forward this field. In my talk at MIPR'21, I will look into the past and present of multimedia research, and share my personal reflections on its future.

Abstract
With the rapid growth of multimedia applications and services, semantic information, such as objects' motion, action, & property, is of increasing importance in many emerging multimedia applications whose data has become extremely "big". This imposes a huge demand for the efficient extraction and compression of semantic information. In this talk, I will introduce our works on multi-modality multimedia information analysis and compression. Firstly, I will introduce our work on object activity and interaction recognition. We re-model the existing action detection architectures, and develop a long-term parsing & short-term sampling structure. Secondly, we will introduce our work on multi-modality multimedia analysis, which aims to accurately localize and analyze objects based on the joint analysis between audio and video streams. Thirdly I will also present our new work on semantic information compression. We construct a new model to describe the spatial-temporal redundancies in semantic data, and design a new architecture that can compress more than 70% of the semantic data. Finally, I will give some industry application examples of our work.

Abstract
A long-term goal of AI research is to build intelligent agents to see and understand the complex visual environment in the physical world and then communicate their understanding to other agents or humans with natural language. This goal brings new problems and research challenges. In this talk, I will introduce the motivations and advantages of integrating knowledge and natural language processing techniques for improving visual understanding. Specifically, I will present our recent discoveries at the intersection of vision and language, including image scene graph generation, visual dialogue, visual captioning, VQA and GAN-related image generation with natural language description or prior knowledge.

Abstract
The fact that there exists a gap between pixels and semantics of images, called the semantic gap, is known for decades. Its resolution is a long-standing problem. Judged by today’s norm, the sizes of labeled datasets were quite small before 2010. The situation began to change with the introduction of the ImageNet, which was viewed as the engine to drive deep learning in the last decade. The chase of more and more annotated data is a clear evidence of supervision’s role. Supervision manifests itself through two aspects: 1) large-scale, high quality annotated data, and 2) well-designed optimization objectives. The two aspects come into play synergistically. For example, design of optimization objectives highly depends on annotations. The optimization procedure often entails a minimum amount of labeled data, and it is expected to scale well with more data. To illustrate various forms of supervision, experiences are drawn from two application domains in this talk: object detection and metric learning for content-based image retrieval (CBIR).

Abstract
Vision and Language are two fundamental capabilities of human intelligence. Humans routinely perform tasks through the interactions between vision and language, supporting the uniquely human capacity to talk about what they see. That motivates us researchers to expand the horizons of such cross-modal analysis. In particular, vision to language is probably one of the hottest topics in the past five years, with a significant growth in both volume of publications and extensive applications. In this talk, we look into the problem of vision to language, from three different perspectives: 1) Independency – aim for a thorough image/video understanding for language generation; 2) Interaction – explore the (1st, 2nd, …) interaction across vision and language inputs; 3) Symbiosis – learn a universal encoder-decoder structure for vision-language tasks. Moreover, we will also discuss the real-world deployments or services of vision to language.

Abstract
From its earliest days, multimedia community developed fundamental principles in a context of a challenge the society was facing. A well-defined challenging application focuses research and development while offering concrete problems that help in evaluating technology. The last few years have seen more focus on machine learning related tools as applied to enhancing old applications practical and more efficient. The last two years, however, have shown that the most important challenge for human society is HEALTH, which is not the focus of multimedia community. Fortunately, sensing, knowledge representation, machine learning, data management as well as biological and health sciences have now created a nexus that makes the next challenge clear as well as practical. I believe that multimedia, and its natural new reincarnation the multimodal, technology is the most relevant technology to make the world a healthy and happy place. Healthy society depends on aggregation of its individual’s health as well as on planetary health. Human food and lifestyle are clearly the source of human health and happiness, but also the source of planetary health. In this presentation we will discuss how to create personal health models and build and use these in recommending lifestyle and food that is both enjoyable, healthy, and sensitive to planetary health. It is now within the realm of sciences and technology to help redefine food ecosystem that will help in making individuals as well as our society and the planet healthy. I believe that the multimedia technology is ready to address this important challenge. Let’s seize this opportunity to make multimodal computing the most impactful area of computing.