Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataset loader issue #20

Open
Techzist16 opened this issue Jun 10, 2022 · 8 comments
Open

Dataset loader issue #20

Techzist16 opened this issue Jun 10, 2022 · 8 comments

Comments

@Techzist16
Copy link

Techzist16 commented Jun 10, 2022

I have used only DIV2K dataset which is stored in the folder named "datasets". The config.py file is changed accordingly.
`from easydict import EasyDict as edict

class Config:
# dataset
DATASET = edict()
DATASET.TYPE = 'MixDataset'
DATASET.DATASETS = ['DIV2K']
DATASET.SPLITS = ['TRAIN']
DATASET.PHASE = 'train'
DATASET.INPUT_HEIGHT = 48
DATASET.INPUT_WIDTH = 48
DATASET.SCALE = 4
DATASET.REPEAT = 1
DATASET.VALUE_RANGE = 255.0
DATASET.SEED = 100`
The init.py is also changed.

 `from easydict import EasyDict as edict
 import importlib
 import os

from utils.common import scandir


dataset_root = os.path.dirname(os.path.abspath(__file__))
print(dataset_root)
dataset_filenames = [
os.path.splitext(os.path.basename(v))[0] for v in scandir(dataset_root)
    if v.endswith('_dataset.py')
 ]
dataset_modules = [
       importlib.import_module(f'datasets.{file_name}')
       for file_name in dataset_filenames
]


class DATASET:
     LEGAL = ['DIV2K', 'Flickr2K', 'Set5', 'Set14', 'BSDS100', 'Urban100', 'Manga109']

# training dataset
DIV2K = edict()
DIV2K.TRAIN = edict()
DIV2K.TRAIN.HRx2 = '/home/BebyGAN/Simple-SR-master/datasets/DIV2K/DIV2K_train_HR_sub'
DIV2K.TRAIN.HRx3 = '/home/BebyGAN/Simple-SR-master/datasets/DIV2K/DIV2K_train_HR_sub'
DIV2K.TRAIN.HRx4 = '/home/BebyGAN/Simple-SR-master/datasets/DIV2K/DIV2K_train_HR_sub'

DIV2K.TRAIN.LRx2 = '/home/BebyGAN/Simple-SR-master/datasets/DIV2K/DIV2K_train_LR_bicubic_sub/X2'
DIV2K.TRAIN.LRx3 = '/home/BebyGAN/Simple-SR-master/datasets/DIV2K/DIV2K_train_LR_bicubic_sub/X3'
DIV2K.TRAIN.LRx4 = '/home/BebyGAN/Simple-SR-master/datasets/DIV2K/DIV2K_train_LR_bicubic_sub/X4'

# testing dataset
Set5 = edict()
Set5.VAL = edict()
Set5.VAL.HRx2 = None
Set5.VAL.HRx3 = None
Set5.VAL.HRx4 = None
Set5.VAL.LRx2 = None
Set5.VAL.LRx3 = None
Set5.VAL.LRx4 = None

Set14 = edict()
Set14.VAL = edict()
Set14.VAL.HRx2 = None
Set14.VAL.HRx3 = None
Set14.VAL.HRx4 = None
Set14.VAL.LRx2 = None
Set14.VAL.LRx3 = None
Set14.VAL.LRx4 = None

BSDS100 = edict()
BSDS100.VAL = edict()
BSDS100.VAL.HRx2 = '/data/liwenbo/datasets/benchmark_SR/BSDS100/HR/modX2'
BSDS100.VAL.HRx3 = '/data/liwenbo/datasets/benchmark_SR/BSDS100/HR/modX3'
BSDS100.VAL.HRx4 = '/data/liwenbo/datasets/benchmark_SR/BSDS100/HR/modX4'
BSDS100.VAL.LRx2 = '/data/liwenbo/datasets/benchmark_SR/BSDS100/LR_bicubic/X2'
BSDS100.VAL.LRx3 = '/data/liwenbo/datasets/benchmark_SR/BSDS100/LR_bicubic/X3'
BSDS100.VAL.LRx4 = '/data/liwenbo/datasets/benchmark_SR/BSDS100/LR_bicubic/X4'

Urban100 = edict()
Urban100.VAL = edict()
Urban100.VAL.HRx2 = None
Urban100.VAL.HRx3 = None
Urban100.VAL.HRx4 = None
Urban100.VAL.LRx2 = None
Urban100.VAL.LRx3 = None
Urban100.VAL.LRx4 = None

Manga109 = edict()
Manga109.VAL = dict()
Manga109.VAL.HRx2 = None
Manga109.VAL.HRx3 = None
Manga109.VAL.HRx4 = None
Manga109.VAL.LRx2 = None
Manga109.VAL.LRx3 = None
Manga109.VAL.LRx4 = None


 def get_dataset(config):
     dataset_type = config.TYPE
     dataset_cls = None
      for module in _dataset_modules:
         dataset_cls = getattr(module, dataset_type, None)
         if dataset_cls is not None:
            break
         if dataset_cls is None:
            raise ValueError(f'Dataset {dataset_type} is not found.')

  hr_paths = []
  lr_paths = []
D = DATASET()

for dataset, split in zip(config.DATASETS, config.SPLITS):
    if dataset not in D.LEGAL or split not in eval('D.%s' % dataset):
        raise ValueError('Illegal dataset.')
    hr_paths.append(eval('D.%s.%s.HRx%d' % (dataset, split, config.SCALE)))
    lr_paths.append(eval('D.%s.%s.LRx%d' % (dataset, split, config.SCALE)))

return dataset_cls(hr_paths, lr_paths, config)`

Traceback (most recent call last):
File "train.py", line 17, in
from dataset import get_dataset
File "/home/BebyGAN/Simple-SR-master/exps/BebyGAN/../../dataset/init.py", line 15
importlib.import_module(f'datasets.{file_name}')
^
SyntaxError: invalid syntax

Please help me on this regards @fenglinglwb

@fenglinglwb
Copy link
Collaborator

Are you using a version of python lower than 3.6?

@Techzist16
Copy link
Author

Are you using a version of python lower than 3.6?

Yes. I'm using python version 3.5.6

@fenglinglwb
Copy link
Collaborator

That's it. Python 3.5.6 doesn't support f-string.

@Techzist16
Copy link
Author

That's it. Python 3.5.6 doesn't support f-string.

Which version should I use?

@fenglinglwb
Copy link
Collaborator

Python>=3.6

@Techzist16
Copy link
Author

Techzist16 commented Jun 11, 2022

I have installed python 3.6.10 version and pytorch version 1.4.0. But I'm getting the following error now:

Traceback (most recent call last):
File "train.py", line 17, in
from dataset import get_dataset
File "/home/BebyGAN/Simple-SR-master/exps/BebyGAN/../../dataset/init.py", line 16, in
for file_name in dataset_filenames
File "/home/BebyGAN/Simple-SR-master/exps/BebyGAN/../../dataset/init.py", line 16, in
for file_name in dataset_filenames
File "/home/anaconda3/envs/Bebygan/lib/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'datasets.mix_dataset'

@fenglinglwb
Copy link
Collaborator

Hi, deeply thanks for your attention. But all these errors could be solved by yourself. Please try to get familiar with the code. It will save you a lot of communication time.

@Techzist16
Copy link
Author

Hi, deeply thanks for your attention. But all these errors could be solved by yourself. Please try to get familiar with the code. It will save you a lot of communication time.

Thank you very much for your time and consideration. Yes, I'm trying to comprehend the code. I just asked since the error appears to be caused by a version issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants