Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

terminate called after throwing an instance of 'c10::Error' #3

Open
geekzyn opened this issue Dec 22, 2020 · 8 comments
Open

terminate called after throwing an instance of 'c10::Error' #3

geekzyn opened this issue Dec 22, 2020 · 8 comments

Comments

@geekzyn
Copy link

geekzyn commented Dec 22, 2020

Hey there, I can successfully run the project but the training doesn't start, when's sampling is done the second Coppeliasim thread closes, what makes me assume its a parallelism issue. Great work btw!
Environment: Azure VM, Ubuntu 18.04.5 LTS (no GPU)

source catalystenv/bin/activate
CUDA_VISIBLE_DEVICES=0 catalyst-rl run-trainer --config ./configs/config.yml
root@Linux:~/Downloads/catalyst-rl-tutorial# source catalystenv/bin/activate
(catalystenv) root@Linux:~/Downloads/catalyst-rl-tutorial# CUDA_VISIBLE_DEVICES=0 catalyst-rl run-trainer --config ./configs/config.yml
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
-----------Actor------------
UR5Actor(
  (state_net): StateNet(
    (main_net): Sequential(
      (0): Linear(in_features=1120, out_features=256, bias=False)
      (1): ReLU(inplace=True)
      (2): Linear(in_features=256, out_features=256, bias=False)
      (3): ReLU(inplace=True)
    )
    (image_net): Sequential(
      (0): Conv2d(1, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (1): InstanceNorm2d(16, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (2): ReLU(inplace=True)
      (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (4): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (5): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (6): ReLU(inplace=True)
      (7): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (8): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (9): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (10): ReLU(inplace=True)
      (11): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (12): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (13): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (14): ReLU(inplace=True)
      (15): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (16): Conv2d(32, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (17): InstanceNorm2d(16, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (18): ReLU(inplace=True)
      (19): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    )
  )
  (head_net): PolicyHead(
    (head_net): SequentialNet(
      (net): Sequential(
        (block_0): Sequential(
          (layer): Linear(in_features=256, out_features=3, bias=True)
          (act): Tanh()
        )
      )
    )
  )
)
---------Critic--------
UR5StateActionCritic(
  (state_action_net): StateActionNet(
    (main_net): Sequential(
      (0): Linear(in_features=1184, out_features=256, bias=False)
      (1): ReLU(inplace=True)
      (2): Linear(in_features=256, out_features=256, bias=False)
      (3): ReLU(inplace=True)
    )
    (image_net): Sequential(
      (0): Conv2d(1, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (1): InstanceNorm2d(16, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (2): ReLU(inplace=True)
      (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (4): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (5): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (6): ReLU(inplace=True)
      (7): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (8): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (9): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (10): ReLU(inplace=True)
      (11): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (12): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (13): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (14): ReLU(inplace=True)
      (15): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (16): Conv2d(32, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (17): InstanceNorm2d(16, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (18): ReLU(inplace=True)
      (19): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    )
    (action_net): Sequential(
      (0): Linear(in_features=3, out_features=64, bias=False)
      (1): ReLU(inplace=True)
      (2): Linear(in_features=64, out_features=64, bias=False)
      (3): ReLU(inplace=True)
    )
  )
  (head_net): ValueHead(
    (value_heads): ModuleList(
      (0): Linear(in_features=256, out_features=1, bias=True)
    )
  )
)
---------Critic--------
UR5StateActionCritic(
  (state_action_net): StateActionNet(
    (main_net): Sequential(
      (0): Linear(in_features=1184, out_features=256, bias=False)
      (1): ReLU(inplace=True)
      (2): Linear(in_features=256, out_features=256, bias=False)
      (3): ReLU(inplace=True)
    )
    (image_net): Sequential(
      (0): Conv2d(1, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (1): InstanceNorm2d(16, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (2): ReLU(inplace=True)
      (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (4): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (5): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (6): ReLU(inplace=True)
      (7): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (8): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (9): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (10): ReLU(inplace=True)
      (11): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (12): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (13): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (14): ReLU(inplace=True)
      (15): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (16): Conv2d(32, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (17): InstanceNorm2d(16, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False)
      (18): ReLU(inplace=True)
      (19): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    )
    (action_net): Sequential(
      (0): Linear(in_features=3, out_features=64, bias=False)
      (1): ReLU(inplace=True)
      (2): Linear(in_features=64, out_features=64, bias=False)
      (3): ReLU(inplace=True)
    )
  )
  (head_net): ValueHead(
    (value_heads): ModuleList(
      (0): Linear(in_features=256, out_features=1, bias=True)
    )
  )
)
================================================================================
Something go wrong with trajectory:
'NoneType' object has no attribute 'get'
None
================================================================================
================================================================================
Something go wrong with trajectory:
'NoneType' object has no attribute 'get'
None
================================================================================
================================================================================
Something go wrong with trajectory:
'NoneType' object has no attribute 'get'
None
================================================================================
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000000 | transitions: 000000000 | buffer size: 000000000/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000001 | transitions: 000000101 | buffer size: 000000101/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000001 | transitions: 000000101 | buffer size: 000000101/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000001 | transitions: 000000101 | buffer size: 000000101/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000001 | transitions: 000000101 | buffer size: 000000101/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000001 | transitions: 000000101 | buffer size: 000000101/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000001 | transitions: 000000101 | buffer size: 000000101/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000001 | transitions: 000000101 | buffer size: 000000101/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000001 | transitions: 000000101 | buffer size: 000000101/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000001 | transitions: 000000101 | buffer size: 000000101/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000001 | transitions: 000000101 | buffer size: 000000101/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000001 | transitions: 000000101 | buffer size: 000000101/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000002 | transitions: 000000202 | buffer size: 000000202/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000002 | transitions: 000000202 | buffer size: 000000202/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000002 | transitions: 000000202 | buffer size: 000000202/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000002 | transitions: 000000202 | buffer size: 000000202/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000002 | transitions: 000000202 | buffer size: 000000202/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000002 | transitions: 000000202 | buffer size: 000000202/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000002 | transitions: 000000202 | buffer size: 000000202/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000002 | transitions: 000000202 | buffer size: 000000202/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000002 | transitions: 000000202 | buffer size: 000000202/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000002 | transitions: 000000202 | buffer size: 000000202/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000003 | transitions: 000000303 | buffer size: 000000303/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000003 | transitions: 000000303 | buffer size: 000000303/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000003 | transitions: 000000303 | buffer size: 000000303/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000003 | transitions: 000000303 | buffer size: 000000303/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000003 | transitions: 000000303 | buffer size: 000000303/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000003 | transitions: 000000303 | buffer size: 000000303/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000003 | transitions: 000000303 | buffer size: 000000303/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000003 | transitions: 000000303 | buffer size: 000000303/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000003 | transitions: 000000303 | buffer size: 000000303/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000003 | transitions: 000000303 | buffer size: 000000303/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000003 | transitions: 000000303 | buffer size: 000000303/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000004 | transitions: 000000404 | buffer size: 000000404/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000004 | transitions: 000000404 | buffer size: 000000404/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000004 | transitions: 000000404 | buffer size: 000000404/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000004 | transitions: 000000404 | buffer size: 000000404/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000004 | transitions: 000000404 | buffer size: 000000404/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000004 | transitions: 000000404 | buffer size: 000000404/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000004 | transitions: 000000404 | buffer size: 000000404/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000004 | transitions: 000000404 | buffer size: 000000404/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000004 | transitions: 000000404 | buffer size: 000000404/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000004 | transitions: 000000404 | buffer size: 000000404/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000004 | transitions: 000000404 | buffer size: 000000404/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000005 | transitions: 000000505 | buffer size: 000000505/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000005 | transitions: 000000505 | buffer size: 000000505/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000005 | transitions: 000000505 | buffer size: 000000505/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000005 | transitions: 000000505 | buffer size: 000000505/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000005 | transitions: 000000505 | buffer size: 000000505/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000005 | transitions: 000000505 | buffer size: 000000505/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000005 | transitions: 000000505 | buffer size: 000000505/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000005 | transitions: 000000505 | buffer size: 000000505/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000005 | transitions: 000000505 | buffer size: 000000505/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000005 | transitions: 000000505 | buffer size: 000000505/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000006 | transitions: 000000606 | buffer size: 000000606/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000006 | transitions: 000000606 | buffer size: 000000606/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000006 | transitions: 000000606 | buffer size: 000000606/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000006 | transitions: 000000606 | buffer size: 000000606/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000006 | transitions: 000000606 | buffer size: 000000606/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000006 | transitions: 000000606 | buffer size: 000000606/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000006 | transitions: 000000606 | buffer size: 000000606/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000006 | transitions: 000000606 | buffer size: 000000606/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000006 | transitions: 000000606 | buffer size: 000000606/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000006 | transitions: 000000606 | buffer size: 000000606/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000006 | transitions: 000000606 | buffer size: 000000606/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000007 | transitions: 000000707 | buffer size: 000000707/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000007 | transitions: 000000707 | buffer size: 000000707/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000007 | transitions: 000000707 | buffer size: 000000707/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000007 | transitions: 000000707 | buffer size: 000000707/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000007 | transitions: 000000707 | buffer size: 000000707/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000007 | transitions: 000000707 | buffer size: 000000707/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000007 | transitions: 000000707 | buffer size: 000000707/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000007 | transitions: 000000707 | buffer size: 000000707/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000007 | transitions: 000000707 | buffer size: 000000707/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000007 | transitions: 000000707 | buffer size: 000000707/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000008 | transitions: 000000808 | buffer size: 000000808/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000008 | transitions: 000000808 | buffer size: 000000808/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000008 | transitions: 000000808 | buffer size: 000000808/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000008 | transitions: 000000808 | buffer size: 000000808/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000008 | transitions: 000000808 | buffer size: 000000808/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000008 | transitions: 000000808 | buffer size: 000000808/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000008 | transitions: 000000808 | buffer size: 000000808/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000008 | transitions: 000000808 | buffer size: 000000808/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000008 | transitions: 000000808 | buffer size: 000000808/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000008 | transitions: 000000808 | buffer size: 000000808/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000008 | transitions: 000000808 | buffer size: 000000808/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000009 | transitions: 000000909 | buffer size: 000000909/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000009 | transitions: 000000909 | buffer size: 000000909/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000009 | transitions: 000000909 | buffer size: 000000909/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000009 | transitions: 000000909 | buffer size: 000000909/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000009 | transitions: 000000909 | buffer size: 000000909/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000009 | transitions: 000000909 | buffer size: 000000909/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000009 | transitions: 000000909 | buffer size: 000000909/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000009 | transitions: 000000909 | buffer size: 000000909/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000009 | transitions: 000000909 | buffer size: 000000909/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000009 | transitions: 000000909 | buffer size: 000000909/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000009 | transitions: 000000909 | buffer size: 000000909/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000010 | transitions: 000001010 | buffer size: 000001010/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000011 | transitions: 000001111 | buffer size: 000001111/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000011 | transitions: 000001111 | buffer size: 000001111/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000011 | transitions: 000001111 | buffer size: 000001111/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000011 | transitions: 000001111 | buffer size: 000001111/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000011 | transitions: 000001111 | buffer size: 000001111/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000011 | transitions: 000001111 | buffer size: 000001111/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000011 | transitions: 000001111 | buffer size: 000001111/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000011 | transitions: 000001111 | buffer size: 000001111/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000011 | transitions: 000001111 | buffer size: 000001111/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000011 | transitions: 000001111 | buffer size: 000001111/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000011 | transitions: 000001111 | buffer size: 000001111/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000012 | transitions: 000001212 | buffer size: 000001212/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000012 | transitions: 000001212 | buffer size: 000001212/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000012 | transitions: 000001212 | buffer size: 000001212/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000012 | transitions: 000001212 | buffer size: 000001212/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000012 | transitions: 000001212 | buffer size: 000001212/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000012 | transitions: 000001212 | buffer size: 000001212/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000012 | transitions: 000001212 | buffer size: 000001212/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000012 | transitions: 000001212 | buffer size: 000001212/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000012 | transitions: 000001212 | buffer size: 000001212/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000012 | transitions: 000001212 | buffer size: 000001212/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000013 | transitions: 000001313 | buffer size: 000001313/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000013 | transitions: 000001313 | buffer size: 000001313/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000013 | transitions: 000001313 | buffer size: 000001313/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000013 | transitions: 000001313 | buffer size: 000001313/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000013 | transitions: 000001313 | buffer size: 000001313/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000013 | transitions: 000001313 | buffer size: 000001313/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000013 | transitions: 000001313 | buffer size: 000001313/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000013 | transitions: 000001313 | buffer size: 000001313/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000013 | transitions: 000001313 | buffer size: 000001313/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000013 | transitions: 000001313 | buffer size: 000001313/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000013 | transitions: 000001313 | buffer size: 000001313/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000014 | transitions: 000001414 | buffer size: 000001414/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000014 | transitions: 000001414 | buffer size: 000001414/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000014 | transitions: 000001414 | buffer size: 000001414/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000014 | transitions: 000001414 | buffer size: 000001414/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000014 | transitions: 000001414 | buffer size: 000001414/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000014 | transitions: 000001414 | buffer size: 000001414/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000014 | transitions: 000001414 | buffer size: 000001414/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000014 | transitions: 000001414 | buffer size: 000001414/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000014 | transitions: 000001414 | buffer size: 000001414/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000014 | transitions: 000001414 | buffer size: 000001414/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000015 | transitions: 000001515 | buffer size: 000001515/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000015 | transitions: 000001515 | buffer size: 000001515/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000015 | transitions: 000001515 | buffer size: 000001515/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000015 | transitions: 000001515 | buffer size: 000001515/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000015 | transitions: 000001515 | buffer size: 000001515/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000015 | transitions: 000001515 | buffer size: 000001515/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000015 | transitions: 000001515 | buffer size: 000001515/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000015 | transitions: 000001515 | buffer size: 000001515/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000015 | transitions: 000001515 | buffer size: 000001515/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000015 | transitions: 000001515 | buffer size: 000001515/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000015 | transitions: 000001515 | buffer size: 000001515/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000016 | transitions: 000001616 | buffer size: 000001616/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000016 | transitions: 000001616 | buffer size: 000001616/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000016 | transitions: 000001616 | buffer size: 000001616/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000016 | transitions: 000001616 | buffer size: 000001616/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000016 | transitions: 000001616 | buffer size: 000001616/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000016 | transitions: 000001616 | buffer size: 000001616/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000016 | transitions: 000001616 | buffer size: 000001616/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000016 | transitions: 000001616 | buffer size: 000001616/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000016 | transitions: 000001616 | buffer size: 000001616/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000016 | transitions: 000001616 | buffer size: 000001616/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000017 | transitions: 000001717 | buffer size: 000001717/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000017 | transitions: 000001717 | buffer size: 000001717/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000017 | transitions: 000001717 | buffer size: 000001717/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000017 | transitions: 000001717 | buffer size: 000001717/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000017 | transitions: 000001717 | buffer size: 000001717/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000017 | transitions: 000001717 | buffer size: 000001717/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000017 | transitions: 000001717 | buffer size: 000001717/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000017 | transitions: 000001717 | buffer size: 000001717/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000017 | transitions: 000001717 | buffer size: 000001717/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000017 | transitions: 000001717 | buffer size: 000001717/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000017 | transitions: 000001717 | buffer size: 000001717/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000018 | transitions: 000001818 | buffer size: 000001818/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000018 | transitions: 000001818 | buffer size: 000001818/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000018 | transitions: 000001818 | buffer size: 000001818/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000018 | transitions: 000001818 | buffer size: 000001818/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000018 | transitions: 000001818 | buffer size: 000001818/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000018 | transitions: 000001818 | buffer size: 000001818/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000018 | transitions: 000001818 | buffer size: 000001818/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000018 | transitions: 000001818 | buffer size: 000001818/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000018 | transitions: 000001818 | buffer size: 000001818/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000018 | transitions: 000001818 | buffer size: 000001818/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000019 | transitions: 000001919 | buffer size: 000001919/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000019 | transitions: 000001919 | buffer size: 000001919/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000019 | transitions: 000001919 | buffer size: 000001919/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000019 | transitions: 000001919 | buffer size: 000001919/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000019 | transitions: 000001919 | buffer size: 000001919/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000019 | transitions: 000001919 | buffer size: 000001919/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000019 | transitions: 000001919 | buffer size: 000001919/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000019 | transitions: 000001919 | buffer size: 000001919/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000019 | transitions: 000001919 | buffer size: 000001919/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000019 | transitions: 000001919 | buffer size: 000001919/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000019 | transitions: 000001919 | buffer size: 000001919/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000019 | transitions: 000001919 | buffer size: 000001919/000002000
--- fps:     0.0 | updates per sample:     0.0 | trajectories: 000000020 | transitions: 000002020 | buffer size: 000002020/000002000
terminate called after throwing an instance of 'c10::Error'
  what():  HIP error: hipErrorNoDevice
Exception raised from deviceCount at /pytorch/aten/src/ATen/hip/impl/HIPGuardImplMasqueradingAsCUDA.h:98 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7ff75f036d12 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x57d4f1 (0x7ff75fa2f4f1 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/lib/python3.6/site-packages/torch/lib/libtorch_hip.so)
frame #2: torch::autograd::Engine::start_device_threads() + 0x442 (0x7ff7910c2252 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0xf907 (0x7ff7ab34d907 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: torch::autograd::Engine::initialize_device_threads_pool() + 0xd5 (0x7ff7910bf785 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #5: torch::autograd::Engine::execute_with_graph_task(std::shared_ptr<torch::autograd::GraphTask> const&, std::shared_ptr<torch::autograd::Node>) + 0x2f (0x7ff7910c7faf in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #6: torch::autograd::python::PythonEngine::execute_with_graph_task(std::shared_ptr<torch::autograd::GraphTask> const&, std::shared_ptr<torch::autograd::Node>) + 0x3c (0x7ff79fbb9d1c in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #7: torch::autograd::Engine::execute(std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&, std::vector<at::Tensor, std::allocator<at::Tensor> > const&, bool, bool, std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&) + 0xacd (0x7ff7910c746d in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #8: torch::autograd::python::PythonEngine::execute(std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&, std::vector<at::Tensor, std::allocator<at::Tensor> > const&, bool, bool, std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&) + 0x4e (0x7ff79fbb9b1e in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #9: THPEngine_run_backward(THPEngine*, _object*, _object*) + 0xe3f (0x7ff79fbbabef in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #10: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a4a5]
frame #11: _PyEval_EvalFrameDefault + 0x1226 (0x50cc96 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #12: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x507be4]
frame #13: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x509900]
frame #14: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #15: _PyEval_EvalFrameDefault + 0x444 (0x50beb4 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #16: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x507be4]
frame #17: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x509900]
frame #18: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #19: _PyEval_EvalFrameDefault + 0x444 (0x50beb4 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #20: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x5095c8]
frame #21: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #22: _PyEval_EvalFrameDefault + 0x444 (0x50beb4 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #23: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x507be4]
frame #24: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x509900]
frame #25: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #26: _PyEval_EvalFrameDefault + 0x1226 (0x50cc96 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #27: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x507be4]
frame #28: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x509900]
frame #29: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #30: _PyEval_EvalFrameDefault + 0x1226 (0x50cc96 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #31: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x5095c8]
frame #32: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #33: _PyEval_EvalFrameDefault + 0x444 (0x50beb4 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #34: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x5095c8]
frame #35: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #36: _PyEval_EvalFrameDefault + 0x444 (0x50beb4 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #37: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x5095c8]
frame #38: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #39: _PyEval_EvalFrameDefault + 0x444 (0x50beb4 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #40: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x5095c8]
frame #41: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #42: _PyEval_EvalFrameDefault + 0x444 (0x50beb4 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #43: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x5095c8]
frame #44: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #45: _PyEval_EvalFrameDefault + 0x444 (0x50beb4 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #46: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x5095c8]
frame #47: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #48: _PyEval_EvalFrameDefault + 0x444 (0x50beb4 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #49: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x5095c8]
frame #50: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #51: _PyEval_EvalFrameDefault + 0x444 (0x50beb4 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #52: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x5095c8]
frame #53: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x50a2fd]
frame #54: _PyEval_EvalFrameDefault + 0x444 (0x50beb4 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #55: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x507be4]
frame #56: PyEval_EvalCode + 0x23 (0x50ad03 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #57: /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python() [0x634e72]
frame #58: PyRun_FileExFlags + 0x97 (0x634f27 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #59: PyRun_SimpleFileExFlags + 0x17f (0x6386df in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #60: Py_Main + 0x591 (0x639281 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #61: main + 0xe0 (0x4b0dc0 in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)
frame #62: __libc_start_main + 0xe7 (0x7ff7ab57ebf7 in /lib/x86_64-linux-gnu/libc.so.6)
frame #63: _start + 0x2a (0x5b259a in /home/azurezyn/Downloads/catalyst-rl-tutorial/catalystenv/bin/python)

Aborted (core dumped)
@dtransposed
Copy link

Hello @geekzyn! You are not using GPU, but you specify CUDA_VISIBLE_DEVICES=0 catalyst-rl run-trainer --config ./configs/config.yml. This looks fishy to me and a potential source of problems.

Either way, I would recommend you to get at least a single, modest GPU on your VM and try to rerun the code.

@peter874j
Copy link

I also encountered the same problem, but my hardware has a GPU.
Is there a solution?
For example, packages version issues, etc.
Environment: Ubuntu 18.04 LTS (with GPU / Cuda 10.0)

The Trainer Main Error Message:

================================================================================
Something go wrong with trajectory:
'NoneType' object has no attribute 'get'
None

terminate called after throwing an instance of 'c10::Error'
what(): HIP error: hipErrorNoDevice

@HeeEun9
Copy link

HeeEun9 commented Feb 23, 2021

I have 2 GPU! but I have same problem

@MaDaO009
Copy link

MaDaO009 commented Apr 9, 2021

Hello @geekzyn! You are not using GPU, but you specify CUDA_VISIBLE_DEVICES=0 catalyst-rl run-trainer --config ./configs/config.yml. This looks fishy to me and a potential source of problems.

Either way, I would recommend you to get at least a single, modest GPU on your VM and try to rerun the code.

Hi @dtransposed . I followed the toturial and encountered the same problem. And I found the reason was that in 'pip install -r requirements.txt' the pytorch version for ROCm was installed. But my gpu is Intel HD630 which is not supported by pytorch and is not an AMD card.

I first tried to run with CUDA_VISIBLE_DEVICES=“” ( also tried =-1) to use CPU only. But it seemed that the trainer part cannot run with CPU.(?) So I tired to uninstall pytorch in the virtualenv and then install torch==1.3.1+cpu. I also tried to modified the requirements.txt to install torch==1.3.1+cpu directly when running run-training.sh. But they did not work either.

My question is that is there any way to run the training demo sucessfully in my device (Intel HD630)(using only CPU or some other ways).

Besides, I have another problem. The trainer thread creates a vrep window which does not display anything. I suppose it is not normal right?

Thank you for your time. And I'm looking forward to your reply!

@LiesMH
Copy link

LiesMH commented Dec 10, 2021 •
edited
Loading

We got this error as well, when was using the torch Dataloader in combination with cuda. I checked my OOM, and there was sufficient memory to perform the training. Setting num=workers to 1 did not help. Training on CPU did not help either.

Eventually, I found out that this error was caused because some of the data files I was trying to import with Dataloader were corrupted. The corrupted files caused this error, because they were incompatible with the torch.to_Tensor() function. This was not mentioned in the initial error message, I found it by setting the num_workers to 0. The error message then changed and included 'CORRUPTED file'. After removing the corrupted data files, everything ran smoothly with my initial num_workers=4.

@Noumanijaz744
Copy link

0%| | 0/15721 [00:00<?, ?it/s]
terminate called after throwing an instance of 'c10::CUDAError'
what(): CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception raised from createEvent at ../aten/src/ATen/cuda/CUDAEvent.h:166 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fc79731f612 in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: + 0xea8e4a (0x7fc7986dfe4a in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x339888 (0x7fc7e0ba7888 in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #3: c10::TensorImpl::release_resources() + 0x175 (0x7fc797304295 in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #4: + 0x214e5d (0x7fc7e0a82e5d in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x541618 (0x7fc7e0daf618 in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: THPVariable_subclass_dealloc(_object*) + 0x2b2 (0x7fc7e0daf912 in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x122adb (0x564df9970adb in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #8: + 0x1304ba (0x564df997e4ba in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #9: + 0x22082f (0x564df9a6e82f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #10: _PyEval_EvalFrameDefault + 0x586d (0x564df998851d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #11: + 0x152f2c (0x564df99a0f2c in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #12: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #13: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #14: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #15: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #16: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #17: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #18: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #19: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #20: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #21: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #22: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #23: + 0x152f2c (0x564df99a0f2c in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #24: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #25: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #26: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #27: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #28: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #29: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #30: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #31: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #32: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #33: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #34: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #35: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #36: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #37: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #38: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #39: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #40: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #41: + 0x1eac62 (0x564df9a38c62 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #42: PyEval_EvalCode + 0x87 (0x564df9a38ba7 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #43: + 0x1f234f (0x564df9a4034f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #44: + 0x145e21 (0x564df9993e21 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #45: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #46: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #47: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #48: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #49: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #50: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #51: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #52: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #53: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #54: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #55: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #56: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #57: _PyEval_EvalFrameDefault + 0x306 (0x564df9982fb6 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #58: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #59: _PyEval_EvalFrameDefault + 0x49d9 (0x564df9987689 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #60: + 0x1eac62 (0x564df9a38c62 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #61: PyEval_EvalCode + 0x87 (0x564df9a38ba7 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #62: + 0x1f234f (0x564df9a4034f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #63: + 0x145e21 (0x564df9993e21 in /data/nouman/.conda/envs/dcas22_task5/bin/python)

@Noumanijaz744
Copy link

i have gpu on my VM and done with extraction correctly but the training have this can anyone please help how to tackle this issue?

@xiaobai0906
Copy link

i have gpu on my VM and done with extraction correctly but the training have this can anyone please help how to tackle this issue?

hello,I have one gpu and i have same problem,have you solved it? thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@dtransposed @peter874j @MaDaO009 @geekzyn @LiesMH @Noumanijaz744 @HeeEun9 @xiaobai0906