Weijian Deng @ANU

Weijian Deng

I am a Research Fellow at the Australian National University, collaborating with Prof. Stephen Gould. My research primarily focuses on 3D Content Modeling and Machine Learning Safety. I am also fortunate to work alongside Dr. Dylan Campbell.

Previously, I pursued my PhD at the same institution, where I explored model generalization prediction. My work involved close collaboration with Prof. Stephen Gould, Dr. Yumin Suh, and Dr. Liang Zheng.

Robust Foundational Vision-Language Models: My research centers on improving the robustness and reliability of AI systems, with a particular emphasis on vision-language models. I am currently exploring model generalization prediction, developing methods to estimate model accuracy on unlabeled test sets. Additionally, I investigate model robustness, focusing on the geometric understanding within CLIP models and assessing their sensitivity to variations in visual factors.

Developing Advanced Simulation and 3D Modeling Techniques: I am also interested in creating advanced simulation frameworks using neural rendering and large language models (LLMs). These simulations are designed to enhance AI adaptability and reliability, especially in autonomous systems, by accurately reflecting complex, real-world scenarios and incorpo- rating physics-based principles into 3D modeling.

Email / CV / Google Scholar / Github / Linkedin / PhD Thesis

News

[Nov 2023] Honoured to be recognized by NeurIPS 2024 as a top reviewer

[Oct 2023] Honoured to be recognized by ACM MM 2024 as outstanding Area Chair. [My talk] on my PhD research

[Sep 2024] One paper on model generalization prediction is accepted to NeurIPS 2024 [paper]

[May 2024] One paper on unsupervised model ranking is accepted to TMLR 2024 [paper]

[May 2024] One paper on calibration analysis of VLM is accepted to ICML 2024 [paper]

[Mar 2024] I'm honored to serve as an Action Editor for [Transactions on Machine Learning Research], a new open-review journal in machine learning. I warmly invite you to join us in this innovative publication journey.

[Feb 2024] One paper on 3D reconstruction is accepted to CVPR 2024

[Jan 2024] Excited for my Journey with HEX International Singapore's Youth Entrepreneurship Programs [My Reflection]

[Dec 2023] Enthusiastically teaching Introduction to Computer Science at Beloved SDUW

[Nov 2023] My PhD thesis is available on the [ANU Open Research]

[Nov 2023] Honoured to be recognized by NeurIPS 2023 as a top reviewer

[Oct 2023] One paper on novel view synthesis of refrative objects is accepted to WACV 2024 [paper]

[Sep 2023] One paper on robustness of visual foundation model is accepted to NeurIPS 2023 [Paper]

[Jul 2023] One paper on (out-of-distribution) predictive calibration is accepted to ICCV 2023 [Paper]

[Apr 2023] One paper on model (out-of-distribution) generalization prediction is accepted to ICML 2023 [Paper]

[Feb 2023] One paper on dataset-level analysis is accepted to CVPR 2023 [Paper]

--- click to read more ---

[Jan 2023] Started the Research Fellow position

[Jan 2023] Submitted PhD thesis

[Dec 2022] Completed PhD oral presentation

[Oct 2022] One paper on multi-task learning is accepted to WACV 2023 [Paper]

[Oct 2022] Honoured to receive NeurIPS 2022 Scholar Award (Travel Award)

[Sep 2022] One paper on model invariance and generalization is accepted to NeurIPS 2022 [Paper]

[Jul 2022] Honoured to be recognized by ICML 2022 as a top 10% reviewer

[Jun 2022] Organized the CVPR 2022 Tutorial on ["Evaluating Models Beyond the Textbook: Out-of-distribution and Without Labels"]

[May 2022] One paper on fine-grained clasification is accepted to IEEE TIP [Paper]

[Dec 2021] One paper on model decision understanding is accepted to IEEE TPAMI [Project]

[May 2021] One paper on generalization prediction is accepted to ICML 2021 [Paper, Project]

[Mar 2021] One paper on generalization prediction is accepted to CVPR 2021 [Paper, Project]

[Jul 2020 - Sep 2020] Summer intern in NEC Labs America. Wonderful experience with my talented mentors Yumin Suh, Xiang Yu, Masoud Faraki, and Manmohan Chandraker.

[Aug 2020] VisDA-2020 challenge successfully ends. Congratulations to the [final teams]!

Big thanks to committee members Kate Saenko, Liang Zheng, and Xingchao Peng!

[Jun 2020] Honoured to be recognized by ECCV2020 as a top reviewer

[Jan 2020] One paper is accepted to IEEE TCSVT [Paper]

[Jul 2019] Study PhD program at The Australian National University. My reseach is supported by Australian Governmen Scholarship [AGRTP].

[Jun 2019] Receive M.Eng. from the University of the Chinese Academy of Sciences

[Jun 2019] Win 3rd place out of 84 participants in vehicle re-identification in CVPR 2019 AI-City Challenge

[Paper, Code]

[Jul 2018 - Nov 2018] Reserch assistant in Singapore University of Technology and Design (SUTD)

[Mar 2018] One paper is accepted to CVPR 2018 [Paper, Code]

[Jul 2017] One paper is accepted to ICCV 2017 [Paper, Code]

Research

My research works focus on Predicting Model Generalization, Monitoring Model Reliability, Enhancing Visual Recognition, and 3D Modeling & Generation.

I. Predicting Model Generalization

PhD Research Topic. At the core of the research lies the quest to unravel how deep neural networks interact with data. This exploration has led to innovative criteria designed to predict model resilience without human annotations. It provides insights to identify and address failure cases, guiding enhancements in future model training. Here are [some slides] to introduce my research.

	MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts Renchunzi Xie, Ambroise Odonnat, Vasilii Feofanov, Weijian Deng, Jianfeng Zhang, Bo An NeurIPS 2024 [Paper, BibTex]
	What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions? Weijie Tu, Weijian Deng, Liang Zheng, Tom Gedeon TMLR 2024 [Paper, BibTex]
	A Bag-of-Prototypes Representation for Dataset-Level Applications Weijie Tu, Weijian Deng, Tom Gedeon, Liang Zheng CVPR 2023 [Paper, Code, BibTex, Poster]
	Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation Weijian Deng, Yumin Suh, Stephen Gould, Liang Zheng ICML 2023 [Paper, Slides, Poster, BibTex, Codes ]
	AutoEval: Are Labels Always Necessary for Classifier Accuracy Evaluation? Weijian Deng, Liang Zheng IEEE TPAMI 2022 [Project, BibTex, Paper]
	What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments? Weijian Deng, Stephen Gould, Liang Zheng ICML 2021 (Spotlight) [Paper, BibTex, Project, Slides, Poster]
	Are Labels Always Necessary for Classifier Accuracy Evaluation? Weijian Deng, Liang Zheng CVPR 2021 [Paper, Project, BibTex, Poster, Slides]

II. Monitoring Model Reliability

	An Empirical Study into What Matters for Calibrating Vision-Language Models Weijie Tu, Weijian Deng, Dylan Campbell, Stephen Gould, Tom Gedeon ICML 2024 [Paper, BibTex, ]
	A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP) Weijie Tu, Weijian Deng, Tom Gedeon NeurIPS 2023 [Paper, BibTex, OpenReview ]
	Adaptive Calibrator Ensemble: Navigating Test Set Difficulty in Out-of-Distribution Scenarios Yuli Zou, Weijian Deng (equal contribution), Liang Zheng ICCV 2023 [Paper, Poster, Code, BibTex ]
	On the Strong Correlation Between Model Invariance and Generalization Weijian Deng, Stephen Gould, Liang Zheng NeurIPS 2022 (Spotlight) [Paper, OpenReview, Slides, Poster, BibTex]
	Ranking Models in Unlabeled New Environments Xiaoxiao Sun, Yunzhong Hou, Weijian Deng, Hongdong Li, Liang Zheng ICCV 2021 [Paper, Code, BibTex ]

III. 3D Modeling & Generation

Ray Deformation Networks for Novel View Synthesis of Refractive Objects

Weijian Deng, Dylan Campbell, Chunyi Sun, Shubham Kanitkar, Matthew Shaffer, Stephen Gould.

WACV 2024 [ Project, Paper, Poster, Slides, BibTex ]

Differentiable Neural Surface Refinement for Transparent Objects

Weijian Deng, Dylan Campbell, Chunyi Sun, Shubham Kanitkar, Matthew Shaffer, Stephen Gould.

CVPR 2024 [ Project, Poster, Slides, BibTex ]

IV. Enhancing Visual Recognition

	Image-Image Domain Adaptation with Preserved Self-Similarity and Domain-Dissimilarity for Person Re-identification Weijian Deng, Liang Zheng, Qixiang Ye, Guoliang Kang, Yi Yang, Jianbin Jiao CVPR 2018 [Paper, Code, BibTex, Poster]
	Similarity-preserving Image-image Domain Adaptation for Person Re-identification Weijian Deng, Liang Zheng, Qixiang Ye, Yi Yang, Jianbin Jiao Arxiv 2019 [Paper]
	Domain Alignment with Triplets Weijian Deng, Liang Zheng, Jianbin Jiao Arxiv 2019 [Paper]
	Rethinking Triplet Loss for Domain Adaptation Weijian Deng, Liang Zheng, Yifan Sun, Jianbin Jiao IEEE TCSVT 2020 [Paper, BibTex] (Journal version of "Domain alignment with triplets")
	Fine-grained Classification via Categorical Memory Networks Weijian Deng, Joshua Marsh, Stephen Gould, Liang Zheng IEEE TIP 2022 [ BibTex, Paper ]
	Split to Learn: Gradient Split for Multi-Task Human Image Analysis Weijian Deng, Yumin Suh, Xiang Yu, Masoud Faraki, Liang Zheng, Manmohan Chandraker WACV 2023 [ Paper, US Patent, Poster, Slides, BibTex ]
	SVDNet for Pedestrian Retrieval Yifan Sun, Liang Zheng, Weijian Deng, Shengjin Wang ICCV 2017 (Spotlight) [Paper, Code, BibTex, Poster]

Academic Activity

Action Editor, Transactions on Machine Learning Research

Lecturer, "Introduction to Computer Science", SDUW (Joint ANU-SDUW Program, Winter Semester 2023)

ACM Multimedia 2024 Area Chair

Reviewer: NeurIPS 2022-2023; ICML 2022-2024; ICLR 2022-2023; ICCV 2021, 2023; CVPR 2021-2024; ECCV 2020; ACM MM 2020-2023; IEEE-TPAMI; IEEE-TIP; IJCV

Co-organizer: ECCV 2020 Workshop on "Visual Domain Adaptation Challenge"

Co-organizer: CVPR 2022 Tutorial on "Evaluating Models Beyond the Textbook: Out-of-distribution and Without Labels"

Guest speaker: SUTD 2018/12 (image-image translation); ANU 2019/09 (SVDNet)

Awards & Honors

NeurIPS 2024 Top Reviewer, 2024

ACM MM 2024 Outstanding Area Chair, 2024

NeurIPS 2023 Top Reviewer, 2023

ICML 2022 Top 10% Reviewer, 2022

ECCV 2020 Outstanding Reviewer, 2022

Australian Government Research Training Program (AGRTP) Scholarship, 2019-2023

The Third Place in Vehicle Re-identification track of CVPR 2019 AI-City Challenge, 2019

China National Scholarship (Master), 2018

China National Scholarship (Bachelor), 2014, 2015

Grants & Funds

Academic research grant from the Google PaliGemma Academic Program, 2024 - 2025

Academic research grant from the Google Cloud Research Credits Program, 2024 - 2025

ICML Early-Career Travel Fund, 2024

ANU Early-Career Travel Fund, 2024

ANU-SDUW Teaching Fellowship, 2023

NeurIPS 2022 Scholar Award, 2022

Website design from Jon Barron.

--- click to read more ---

I. Predicting Model Generalization

MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts

What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions?

A Bag-of-Prototypes Representation for Dataset-Level Applications

Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation

AutoEval: Are Labels Always Necessary for Classifier Accuracy Evaluation?

What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?

Are Labels Always Necessary for Classifier Accuracy Evaluation?

II. Monitoring Model Reliability

An Empirical Study into What Matters for Calibrating Vision-Language Models

A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)

Adaptive Calibrator Ensemble: Navigating Test Set Difficulty in Out-of-Distribution Scenarios

On the Strong Correlation Between Model Invariance and Generalization

Ranking Models in Unlabeled New Environments

III. 3D Modeling & Generation

Ray Deformation Networks for Novel View Synthesis of Refractive Objects

Differentiable Neural Surface Refinement for Transparent Objects

IV. Enhancing Visual Recognition

Image-Image Domain Adaptation with Preserved Self-Similarity and Domain-Dissimilarity for Person Re-identification

Similarity-preserving Image-image Domain Adaptation for Person Re-identification

Domain Alignment with Triplets

Rethinking Triplet Loss for Domain Adaptation

Fine-grained Classification via Categorical Memory Networks

Split to Learn: Gradient Split for Multi-Task Human Image Analysis

SVDNet for Pedestrian Retrieval