Hierarchy Flow For High-Fidelity Image-to-Image Translation

Weichen Fan^*,1  Jinghuan Chen^*,1  Ziwei Liu^✉,1

¹S-Lab, Nanyang Technological University 
^* Equal Contribution  ^✉ Corresponding Author

---

This repository contains the official implementation of the following paper:

Hierarchy Flow For High-Fidelity Image-to-Image Translation
https://arxiv.org/abs/2308.06909

Abstract: Image-to-image (I2I) translation comprises a wide spectrum of tasks. Here we divide this problem into three levels: strong-fidelity translation, normal-fidelity translation, and weak-fidelity translation, indicating the extent to which the content of the original image is preserved. Although existing methods achieve good performance in weak-fidelity translation, they fail to fully preserve the content in both strong- and normal-fidelity tasks, e.g. sim2real, style transfer and low-level vision. In this work, we propose Hierarchy Flow, a novel flow-based model to achieve better content preservation during translation. Specifically, 1) we first unveil the drawbacks of standard flow-based models when applied to I2I translation. 2) Next, we propose a new design, namely hierarchical coupling for reversible feature transformation and multi-scale modeling, to constitute Hierarchy Flow. 3) Finally, we present a dedicated aligned-style loss for a better trade-off between content preservation and stylization during translation. Extensive experiments on a wide range of I2I translation benchmarks demonstrate that our approach achieves state-of-the-art performance, with convincing advantages in both strong- and normal-fidelity tasks.

Install

conda create -n lcl python=3.10
conda activate lcl
conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=10.2 -c pytorch
pip install -r requirements.txt

Experiments

1. GTA2Cityscapes

Please download the GTA5 dataset and Cityscapes dataset (leftImg8bit) first.

Modify the configs/GTA2Cityscapes/config.yaml

#change the source_root and target_root for train and test respectively
  train:
    source_list: 'datasets/GTA/train.txt'
    target_list: 'datasets/Cityscapes/train.txt'
    source_root: '{YOUR PATH TO GTA}'
    target_root: '{YOUR PATH TO CITYSCAPES}'

  test:
    source_list: 'datasets/GTA/test.txt'
    target_list: 'datasets/Cityscapes/test.txt'
    source_root: '{YOUR PATH TO GTA}'
    target_root: '{YOUR PATH TO CITYSCAPES}'

Training

#FOR SLURM
bash scripts/GTA2CITY/train.sh partition GPU_NUM

#NO SLURM
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 main.py --config configs/config.yaml

Test

#FOR SLURM
bash scripts/GTA2CITY/eval.sh partition GPU_NUM {ckpt_path}

#NO SLURM
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 main.py --config configs/config.yaml --load_path {ckpt_path}

Todo

1. Release the Code.
2. Add GTA2Cityscapes exp.
3. Add COCO2Wikiart exp.
4. Add more exp.

Cite

@article{fan2023hierarchy,
  title={Hierarchy Flow For High-Fidelity Image-to-Image Translation},
  author={Fan, Weichen and Chen, Jinghuan and Liu, Ziwei},
  journal={arXiv preprint arXiv:2308.06909},
  year={2023}
}

Under construction...