VIS(US) Members at ECCV 2022: One Oral Presentation and Two Awards

The European Conference on Computer Vision (ECCV) 2022 was held in Tel Aviv from October 23-27, 2022. Alongside CVPR and ICCV, the biennial conference is considered one of the top conferences in computer vision. In the 2022 edition of the conference, VIS(US) members achieved several successes.

At the main conference, doctoral student Jenny Schmalfuss presented the paper “A Perturbation-Constrained Adversarial Attack for Evaluating the Robustness of Optical Flow” (authors: Jenny Schmalfuss, Philipp Scholze, Andrés Bruhn) in the session „Motion and Tracking“. In their paper, they found methods that allow to estimate motion very well to be very sensible to imperceptible changes in the input images, which then cause the wrong motion to be detected. With an acceptance rate of only 2.7% for oral presentations, this was a great opportunity to present findings about adversarial robustness of methods for motion estimation to the scientific community.

Abstract

Recent optical flow methods are almost exclusively judged in terms of accuracy, while their robustness is often neglected. Although adversarial attacks offer a useful tool to perform such an analysis, current attacks on optical flow methods focus on real-world attacking scenarios rather than a worst case robustness assessment. Hence, in this work, we propose a novel adversarial attack - the Perturbation-Constrained Flow Attack (PCFA) - that emphasizes destructivity over applicability as a real-world attack. PCFA is a global attack that optimizes adversarial perturbations to shift the predicted flow towards a specified target flow, while keeping the L2 norm of the perturbation below a chosen bound. Our experiments demonstrate PCFA's applicability in white- and black-box settings, and show it finds stronger adversarial samples than previous attacks. Based on these strong samples, we provide the first joint ranking of optical flow methods considering both prediction quality and adversarial robustness, which reveals state-of-the-art methods to be particularly vulnerable. Code is available at https://github.com/cv-stuttgart/PCFA.

Azin Jahedi, Maximilian Luz, Lukas Mehl, Marc Rivinius, and Andrés Bruhn won the ECCV 2022 Robust Vision Challenge (RVC 2022) in the category “Optical Flow.” Of all the submissions in the flow category, their submitted method “MS_RAFT+”, an extension of the “MS-RAFT” approach presented early this year at the IEEE International Conference on Image Processing (ICIP),  performed best across the different datasets considered for the challenge (KITTI, Middlebury, MPI Sintel, and VIPER).

The Robust Vision Challenge 2022 was a virtual-only full day event held on October 23rd 2022 in conjunction with ECCV 2022. Rewards for the winners of the challenge included a presentation at the ECCV 2022 RVC Workshop as well as an invitation to publish in the IJCV Special Issue.

At the ECCV 2022 Workshop on Adversarial Robustness In the Real World (AROW) held as virtual-only full day event also on October 23rd 2022, Jenny Schmalfuss, Lukas Mehl, and Andrés Bruhn received a Best Paper Award for their publication "Attacking Motion Estimation with Adversarial Snow". Their work found that adding “unfortunately placed” snowflakes to video sequences can cause neural networks for motion estimation to detect no motion where motion is actually taking place.

Abstract

Current adversarial attacks for motion estimation (optical flow) optimize small per-pixel perturbations, which are unlikely to appear in the real world. In contrast, we exploit a real-world weather phenomenon for a novel attack with adversarially optimized snow. At the core of our attack is a differentiable renderer that consistently integrates photorealistic snowflakes with realistic motion into the 3D scene. Through optimization we obtain adversarial snow that significantly impacts the optical flow while being indistinguishable from ordinary snow. Surprisingly, the impact of our novel attack is largest on methods that previously showed a high robustness to small L_p perturbations.