Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

les_isdac_box config fails on GPU (config from open PR) #3222

Open
jty53 opened this issue Jul 31, 2024 · 1 comment
Open

les_isdac_box config fails on GPU (config from open PR) #3222

jty53 opened this issue Jul 31, 2024 · 1 comment

Comments

@jty53
Copy link
Contributor

jty53 commented Jul 31, 2024 •
edited
Loading

I am attempting to run ~/ClimaAtmos.jl/config/model_configs/les_isdac_box.yml from #3025. The job was run using Slurm on the Pronghorn HPC system using an NVidia Tesla P100-SXM2-16GB.
Note: before running this job, I reduced the time step dt in the config file from .5s to .03s to satisfy the cfl condition after previous cpu (single threaded) jobs with the higher timestep crashed.

Job output attached and provided in comment below (due to length limit).
les_isdac_box-gputest_5075521.txt
@jty53
Copy link
Contributor Author

jty53 commented Jul 31, 2024 

Changing to home directory
Setting environment variables
Running model
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 11.4.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:130
┌ Debug: Forward-compatible CUDA driver found at /data/gpfs/home/jeffreyyoder/.julia/artifacts/a86b67fd924e2a8c72d376d301a34b2364281978/lib/libcuda.so; known to be version 12.5.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:184
┌ Debug: Successfully loaded forwards-compatible CUDA driver
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:215
[ Info: Loading yaml file /data/gpfs/home/jeffreyyoder/ClimaAtmos.jl/src/solver/../../config/default_configs/default_config.yml
[ Info: Loading yaml file /data/gpfs/home/jeffreyyoder/ClimaAtmos.jl/config/model_configs/les_isdac_box.yml
┌ Warning: The configuration passed to ClimaAtmos contains unused keys: c_smag, smagorinsky_lilly, job_id
└ @ ClimaAtmos ~/ClimaAtmos.jl/src/solver/yaml_helper.jl:92
[ Info: Running on CUDADevice.
[ Info: Setting up single-process ClimaAtmos run
[ Info: Running ClimaCore in unthreaded mode.
┌ Info: Making AtmosConfig with config files: 
│    /data/gpfs/home/jeffreyyoder/ClimaAtmos.jl/src/solver/../../config/default_configs/default_config.yml
└    /data/gpfs/home/jeffreyyoder/ClimaAtmos.jl/config/model_configs/les_isdac_box.yml
┌ Info: numerics 
│            `energy_upwinding`::`Val{:none}`
│            `tracer_upwinding`::`Val{:none}`
│             `edmfx_upwinding`::`Val{:none}`
│     `edmfx_sgsflux_upwinding`::`Val{:none}`
│     `test_dycore_consistency`::`Nothing`
└                     `limiter`::`Nothing`
┌ Info: AtmosModel: 
│ 
│                         `model_config`::`ClimaAtmos.BoxModel`
│                       `moisture_model`::`ClimaAtmos.EquilMoistModel`
│                         `precip_model`::`ClimaAtmos.Microphysics1Moment`
│                          `cloud_model`::`ClimaAtmos.QuadratureCloud`
│     `call_cloud_diagnostics_per_stage`::`Nothing`
│                         `forcing_type`::`Nothing`
│                           `subsidence`::`ClimaAtmos.Subsidence{AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#137#138"{Float32}}}`
│                       `radiation_mode`::`ClimaAtmos.RadiationISDAC{Float32}`
│                               `ls_adv`::`Nothing`
│                     `external_forcing`::`ClimaAtmos.ISDACForcing`
│                        `edmf_coriolis`::`Nothing`
│                       `tendency_model`::`ClimaAtmos.UseAllTendency`
│                     `edmfx_entr_model`::`ClimaAtmos.NoEntrainment`
│                     `edmfx_detr_model`::`ClimaAtmos.NoDetrainment`
│                       `turbconv_model`::`Nothing`
│          `non_orographic_gravity_wave`::`Nothing`
│              `orographic_gravity_wave`::`Nothing`
│                            `hyperdiff`::`Nothing`
│                            `vert_diff`::`Nothing`
│                            `diff_mode`::`ClimaAtmos.Explicit`
│                         `sgs_adv_mode`::`ClimaAtmos.Explicit`
│                       `viscous_sponge`::`Nothing`
│                      `rayleigh_sponge`::`Nothing`
│                      `sfc_temperature`::`ClimaAtmos.ZonallySymmetricSST`
│                           `insolation`::`ClimaAtmos.IdealizedInsolation`
│                        `surface_model`::`ClimaAtmos.PrescribedSurfaceTemperature`
│                       `surface_albedo`::`ClimaAtmos.ConstantAlbedo{Float32}`
└                             `numerics`::`ClimaAtmos.AtmosNumerics{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Nothing, Nothing}`
┌ Info: Time info:
│   dt = "0.03secs"
│   t_end = "10mins"
└   floor_n_steps = 20000
┌ Info: Simulation info
│   job_id = "les_isdac_box"
└   output_dir = "output/les_isdac_box/output_0003"
┌ Info: Topography
└   topography = "NoWarp"
┌ Warning: There are known codegen bugs on CUDA 11.4 and earlier for older GPUs like your Tesla P100-SXM2-16GB.
│ Please use CUDA 11.5 or later, or switch to a different device.
└ @ CUDA ~/.julia/packages/CUDA/Tl08O/lib/cudadrv/state.jl:244
┌ Info: z heights
│   z_mesh.faces =
│    101-element LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}:
└     ZPoint(0.0), ZPoint(20.0), …, ZPoint(1980.0), ZPoint(2000.0)
┌ Info: Resolution stats: 
│   Nq = 4
│   h_elem = 6
│   z_elem = 100
│   ncols = 4096
└   ndofs_total = 409600
ERROR: GPU compilation of MethodInstance for ClimaCoreCUDAExt.knl_copyto!(::ClimaCore.DataLayouts.VIJFH{@NamedTuple{ρ::Float32, uₕ::ClimaCore.Geometry.Covariant12Vector{Float32}, ρe_tot::Float32, ρq_tot::Float32, ρq_rai::Float32, ρq_sno::Float32}, 100, 4, CUDA.CuDeviceArray{Float32, 5, 1}}, ::Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}, NTuple{5, Base.OneTo{Int64}}, typeof(ClimaAtmos.InitialConditions.atmos_center_variables), Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}, Nothing, ClimaAtmos.InitialConditions.var"#local_state#77"{ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, Nothing, @NamedTuple{cl::CloudMicrophysics.Parameters.CloudLiquid{Float32}, ci::CloudMicrophysics.Parameters.CloudIce{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}}, pr::CloudMicrophysics.Parameters.Rain{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ps::CloudMicrophysics.Parameters.Snow{Float32, CloudMicrophysics.Parameters.ParticlePDFSnow{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ce::CloudMicrophysics.Parameters.CollisionEff{Float32}, tv::CloudMicrophysics.Parameters.Blk1MVelType{Float32, CloudMicrophysics.Parameters.Blk1MVelTypeRain{Float32}, CloudMicrophysics.Parameters.Blk1MVelTypeSnow{Float32}}, aps::CloudMicrophysics.Parameters.AirProperties{Float32}, var::CloudMicrophysics.Parameters.VarTimescaleAcnv{Float32}, Ndp::Float32}, CloudMicrophysics.Parameters.WaterProperties{Float32}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.Parameters.TurbulenceConvectionParameters{Float32}, ClimaAtmos.Parameters.SurfaceTemperatureParameters{Float32}}, ClimaAtmos.InitialConditions.var"#θ_pert#76"{Bool}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#139#140"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#135#136"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#133#134"{Float32}}, ClimaAtmos.InitialConditions.ColumnInterpolatableField{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VF{Float32, 101, Matrix{Float32}}, ClimaCore.Spaces.FiniteDifferenceSpace{ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}}, ClimaCore.Grids.CellFace}}, Interpolations.Extrapolation{Float32, 1, Interpolations.GriddedInterpolation{Float32, 1, Vector{Float32}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}}}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Interpolations.Flat{Nothing}}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#129#130"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#131#132"{Float32}}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, Type{Float32}, Bool}, Tuple{ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}}, Tuple{ClimaAtmos.AtmosModel{ClimaAtmos.BoxModel, ClimaAtmos.EquilMoistModel, ClimaAtmos.Microphysics1Moment, ClimaAtmos.QuadratureCloud, Nothing, Nothing, ClimaAtmos.Subsidence{AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#137#138"{Float32}}}, ClimaAtmos.RadiationISDAC{Float32}, Nothing, ClimaAtmos.ISDACForcing, Nothing, Bool, ClimaAtmos.UseAllTendency, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Bool, Nothing, Nothing, Nothing, Nothing, Nothing, ClimaAtmos.Explicit, ClimaAtmos.Explicit, Nothing, Nothing, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.IdealizedInsolation, ClimaAtmos.PrescribedSurfaceTemperature, ClimaAtmos.ConstantAlbedo{Float32}, ClimaAtmos.AtmosNumerics{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Nothing, Nothing}}}}}) failed
KernelError: passing and using non-bitstype argument

Argument 3 to your kernel function is of type Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}, NTuple{5, Base.OneTo{Int64}}, typeof(ClimaAtmos.InitialConditions.atmos_center_variables), Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}, Nothing, ClimaAtmos.InitialConditions.var"#local_state#77"{ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, Nothing, @NamedTuple{cl::CloudMicrophysics.Parameters.CloudLiquid{Float32}, ci::CloudMicrophysics.Parameters.CloudIce{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}}, pr::CloudMicrophysics.Parameters.Rain{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ps::CloudMicrophysics.Parameters.Snow{Float32, CloudMicrophysics.Parameters.ParticlePDFSnow{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ce::CloudMicrophysics.Parameters.CollisionEff{Float32}, tv::CloudMicrophysics.Parameters.Blk1MVelType{Float32, CloudMicrophysics.Parameters.Blk1MVelTypeRain{Float32}, CloudMicrophysics.Parameters.Blk1MVelTypeSnow{Float32}}, aps::CloudMicrophysics.Parameters.AirProperties{Float32}, var::CloudMicrophysics.Parameters.VarTimescaleAcnv{Float32}, Ndp::Float32}, CloudMicrophysics.Parameters.WaterProperties{Float32}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.Parameters.TurbulenceConvectionParameters{Float32}, ClimaAtmos.Parameters.SurfaceTemperatureParameters{Float32}}, ClimaAtmos.InitialConditions.var"#θ_pert#76"{Bool}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#139#140"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#135#136"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#133#134"{Float32}}, ClimaAtmos.InitialConditions.ColumnInterpolatableField{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VF{Float32, 101, Matrix{Float32}}, ClimaCore.Spaces.FiniteDifferenceSpace{ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}}, ClimaCore.Grids.CellFace}}, Interpolations.Extrapolation{Float32, 1, Interpolations.GriddedInterpolation{Float32, 1, Vector{Float32}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}}}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Interpolations.Flat{Nothing}}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#129#130"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#131#132"{Float32}}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, Type{Float32}, Bool}, Tuple{ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}}, Tuple{ClimaAtmos.AtmosModel{ClimaAtmos.BoxModel, ClimaAtmos.EquilMoistModel, ClimaAtmos.Microphysics1Moment, ClimaAtmos.QuadratureCloud, Nothing, Nothing, ClimaAtmos.Subsidence{AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#137#138"{Float32}}}, ClimaAtmos.RadiationISDAC{Float32}, Nothing, ClimaAtmos.ISDACForcing, Nothing, Bool, ClimaAtmos.UseAllTendency, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Bool, Nothing, Nothing, Nothing, Nothing, Nothing, ClimaAtmos.Explicit, ClimaAtmos.Explicit, Nothing, Nothing, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.IdealizedInsolation, ClimaAtmos.PrescribedSurfaceTemperature, ClimaAtmos.ConstantAlbedo{Float32}, ClimaAtmos.AtmosNumerics{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Nothing, Nothing}}}}}, which is not isbits:
  .args is of type Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}, Nothing, ClimaAtmos.InitialConditions.var"#local_state#77"{ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, Nothing, @NamedTuple{cl::CloudMicrophysics.Parameters.CloudLiquid{Float32}, ci::CloudMicrophysics.Parameters.CloudIce{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}}, pr::CloudMicrophysics.Parameters.Rain{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ps::CloudMicrophysics.Parameters.Snow{Float32, CloudMicrophysics.Parameters.ParticlePDFSnow{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ce::CloudMicrophysics.Parameters.CollisionEff{Float32}, tv::CloudMicrophysics.Parameters.Blk1MVelType{Float32, CloudMicrophysics.Parameters.Blk1MVelTypeRain{Float32}, CloudMicrophysics.Parameters.Blk1MVelTypeSnow{Float32}}, aps::CloudMicrophysics.Parameters.AirProperties{Float32}, var::CloudMicrophysics.Parameters.VarTimescaleAcnv{Float32}, Ndp::Float32}, CloudMicrophysics.Parameters.WaterProperties{Float32}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.Parameters.TurbulenceConvectionParameters{Float32}, ClimaAtmos.Parameters.SurfaceTemperatureParameters{Float32}}, ClimaAtmos.InitialConditions.var"#θ_pert#76"{Bool}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#139#140"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#135#136"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#133#134"{Float32}}, ClimaAtmos.InitialConditions.ColumnInterpolatableField{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VF{Float32, 101, Matrix{Float32}}, ClimaCore.Spaces.FiniteDifferenceSpace{ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}}, ClimaCore.Grids.CellFace}}, Interpolations.Extrapolation{Float32, 1, Interpolations.GriddedInterpolation{Float32, 1, Vector{Float32}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}}}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Interpolations.Flat{Nothing}}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#129#130"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#131#132"{Float32}}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, Type{Float32}, Bool}, Tuple{ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}}, Tuple{ClimaAtmos.AtmosModel{ClimaAtmos.BoxModel, ClimaAtmos.EquilMoistModel, ClimaAtmos.Microphysics1Moment, ClimaAtmos.QuadratureCloud, Nothing, Nothing, ClimaAtmos.Subsidence{AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#137#138"{Float32}}}, ClimaAtmos.RadiationISDAC{Float32}, Nothing, ClimaAtmos.ISDACForcing, Nothing, Bool, ClimaAtmos.UseAllTendency, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Bool, Nothing, Nothing, Nothing, Nothing, Nothing, ClimaAtmos.Explicit, ClimaAtmos.Explicit, Nothing, Nothing, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.IdealizedInsolation, ClimaAtmos.PrescribedSurfaceTemperature, ClimaAtmos.ConstantAlbedo{Float32}, ClimaAtmos.AtmosNumerics{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Nothing, Nothing}}}} which is not isbits.
    .1 is of type Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}, Nothing, ClimaAtmos.InitialConditions.var"#local_state#77"{ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, Nothing, @NamedTuple{cl::CloudMicrophysics.Parameters.CloudLiquid{Float32}, ci::CloudMicrophysics.Parameters.CloudIce{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}}, pr::CloudMicrophysics.Parameters.Rain{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ps::CloudMicrophysics.Parameters.Snow{Float32, CloudMicrophysics.Parameters.ParticlePDFSnow{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ce::CloudMicrophysics.Parameters.CollisionEff{Float32}, tv::CloudMicrophysics.Parameters.Blk1MVelType{Float32, CloudMicrophysics.Parameters.Blk1MVelTypeRain{Float32}, CloudMicrophysics.Parameters.Blk1MVelTypeSnow{Float32}}, aps::CloudMicrophysics.Parameters.AirProperties{Float32}, var::CloudMicrophysics.Parameters.VarTimescaleAcnv{Float32}, Ndp::Float32}, CloudMicrophysics.Parameters.WaterProperties{Float32}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.Parameters.TurbulenceConvectionParameters{Float32}, ClimaAtmos.Parameters.SurfaceTemperatureParameters{Float32}}, ClimaAtmos.InitialConditions.var"#θ_pert#76"{Bool}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#139#140"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#135#136"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#133#134"{Float32}}, ClimaAtmos.InitialConditions.ColumnInterpolatableField{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VF{Float32, 101, Matrix{Float32}}, ClimaCore.Spaces.FiniteDifferenceSpace{ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}}, ClimaCore.Grids.CellFace}}, Interpolations.Extrapolation{Float32, 1, Interpolations.GriddedInterpolation{Float32, 1, Vector{Float32}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}}}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Interpolations.Flat{Nothing}}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#129#130"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#131#132"{Float32}}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, Type{Float32}, Bool}, Tuple{ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}} which is not isbits.
      .f is of type ClimaAtmos.InitialConditions.var"#local_state#77"{ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, Nothing, @NamedTuple{cl::CloudMicrophysics.Parameters.CloudLiquid{Float32}, ci::CloudMicrophysics.Parameters.CloudIce{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}}, pr::CloudMicrophysics.Parameters.Rain{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ps::CloudMicrophysics.Parameters.Snow{Float32, CloudMicrophysics.Parameters.ParticlePDFSnow{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ce::CloudMicrophysics.Parameters.CollisionEff{Float32}, tv::CloudMicrophysics.Parameters.Blk1MVelType{Float32, CloudMicrophysics.Parameters.Blk1MVelTypeRain{Float32}, CloudMicrophysics.Parameters.Blk1MVelTypeSnow{Float32}}, aps::CloudMicrophysics.Parameters.AirProperties{Float32}, var::CloudMicrophysics.Parameters.VarTimescaleAcnv{Float32}, Ndp::Float32}, CloudMicrophysics.Parameters.WaterProperties{Float32}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.Parameters.TurbulenceConvectionParameters{Float32}, ClimaAtmos.Parameters.SurfaceTemperatureParameters{Float32}}, ClimaAtmos.InitialConditions.var"#θ_pert#76"{Bool}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#139#140"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#135#136"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#133#134"{Float32}}, ClimaAtmos.InitialConditions.ColumnInterpolatableField{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VF{Float32, 101, Matrix{Float32}}, ClimaCore.Spaces.FiniteDifferenceSpace{ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}}, ClimaCore.Grids.CellFace}}, Interpolations.Extrapolation{Float32, 1, Interpolations.GriddedInterpolation{Float32, 1, Vector{Float32}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}}}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Interpolations.Flat{Nothing}}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#129#130"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#131#132"{Float32}}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, Type{Float32}, Bool} which is not isbits.
        .p is of type ClimaAtmos.InitialConditions.ColumnInterpolatableField{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VF{Float32, 101, Matrix{Float32}}, ClimaCore.Spaces.FiniteDifferenceSpace{ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}}, ClimaCore.Grids.CellFace}}, Interpolations.Extrapolation{Float32, 1, Interpolations.GriddedInterpolation{Float32, 1, Vector{Float32}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}}}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Interpolations.Flat{Nothing}}} which is not isbits.
          .f is of type ClimaCore.Fields.Field{ClimaCore.DataLayouts.VF{Float32, 101, Matrix{Float32}}, ClimaCore.Spaces.FiniteDifferenceSpace{ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}}, ClimaCore.Grids.CellFace}} which is not isbits.
            .values is of type ClimaCore.DataLayouts.VF{Float32, 101, Matrix{Float32}} which is not isbits.
              .array is of type Matrix{Float32} which is not isbits.
            .space is of type ClimaCore.Spaces.FiniteDifferenceSpace{ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}}, ClimaCore.Grids.CellFace} which is not isbits.
              .grid is of type ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}} which is not isbits.
                .topology is of type ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}} which is not isbits.
                  .mesh is of type ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}} which is not isbits.
                    .domain is of type ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}} which is not isbits.
                .center_local_geometry is of type ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}} which is not isbits.
                  .array is of type Matrix{Float32} which is not isbits.
                .face_local_geometry is of type ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}} which is not isbits.
                  .array is of type Matrix{Float32} which is not isbits.
          .data is of type Interpolations.Extrapolation{Float32, 1, Interpolations.GriddedInterpolation{Float32, 1, Vector{Float32}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}}}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Interpolations.Flat{Nothing}} which is not isbits.
            .itp is of type Interpolations.GriddedInterpolation{Float32, 1, Vector{Float32}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}}} which is not isbits.
              .knots is of type Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}} which is not isbits.
                .1 is of type Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}} which is not isbits.
                  .parent is of type SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true} which is not isbits.
                    .parent is of type Matrix{Float32} which is not isbits.
              .coefs is of type Vector{Float32} which is not isbits.
        .FT is of type Type{Float32} which is not isbits.


Stacktrace:
  [1] check_invocation(job::GPUCompiler.CompilerJob)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Y4hSX/src/validation.jl:92
  [2] macro expansion
    @ ~/.julia/packages/GPUCompiler/Y4hSX/src/driver.jl:128 [inlined]
  [3] macro expansion
    @ ~/.julia/packages/TimerOutputs/Lw5SP/src/TimerOutput.jl:253 [inlined]
  [4] codegen(output::Symbol, job::GPUCompiler.CompilerJob; libraries::Bool, toplevel::Bool, optimize::Bool, cleanup::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Y4hSX/src/driver.jl:126
  [5] codegen
    @ ~/.julia/packages/GPUCompiler/Y4hSX/src/driver.jl:115 [inlined]
  [6] compile(target::Symbol, job::GPUCompiler.CompilerJob; libraries::Bool, toplevel::Bool, optimize::Bool, cleanup::Bool, strip::Bool, validate::Bool, only_entry::Bool)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Y4hSX/src/driver.jl:111
  [7] compile
    @ ~/.julia/packages/GPUCompiler/Y4hSX/src/driver.jl:103 [inlined]
  [8] #1145
    @ ~/.julia/packages/CUDA/Tl08O/src/compiler/compilation.jl:254 [inlined]
  [9] JuliaContext(f::CUDA.var"#1145#1148"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}}; kwargs::@Kwargs{})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Y4hSX/src/driver.jl:52
 [10] JuliaContext(f::Function)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Y4hSX/src/driver.jl:42
 [11] compile(job::GPUCompiler.CompilerJob)
    @ CUDA ~/.julia/packages/CUDA/Tl08O/src/compiler/compilation.jl:253
 [12] actual_compilation(cache::Dict{Any, CUDA.CuFunction}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::typeof(CUDA.compile), linker::typeof(CUDA.link))
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Y4hSX/src/execution.jl:237
 [13] cached_compilation(cache::Dict{Any, CUDA.CuFunction}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::Function, linker::Function)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Y4hSX/src/execution.jl:151
 [14] macro expansion
    @ ~/.julia/packages/CUDA/Tl08O/src/compiler/execution.jl:369 [inlined]
 [15] macro expansion
    @ ./lock.jl:267 [inlined]
 [16] cufunction(f::typeof(ClimaCoreCUDAExt.knl_copyto!), tt::Type{Tuple{ClimaCore.DataLayouts.VIJFH{@NamedTuple{ρ::Float32, uₕ::ClimaCore.Geometry.Covariant12Vector{Float32}, ρe_tot::Float32, ρq_tot::Float32, ρq_rai::Float32, ρq_sno::Float32}, 100, 4, CUDA.CuDeviceArray{Float32, 5, 1}}, Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}, NTuple{5, Base.OneTo{Int64}}, typeof(ClimaAtmos.InitialConditions.atmos_center_variables), Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}, Nothing, ClimaAtmos.InitialConditions.var"#local_state#77"{ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, Nothing, @NamedTuple{cl::CloudMicrophysics.Parameters.CloudLiquid{Float32}, ci::CloudMicrophysics.Parameters.CloudIce{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}}, pr::CloudMicrophysics.Parameters.Rain{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ps::CloudMicrophysics.Parameters.Snow{Float32, CloudMicrophysics.Parameters.ParticlePDFSnow{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ce::CloudMicrophysics.Parameters.CollisionEff{Float32}, tv::CloudMicrophysics.Parameters.Blk1MVelType{Float32, CloudMicrophysics.Parameters.Blk1MVelTypeRain{Float32}, CloudMicrophysics.Parameters.Blk1MVelTypeSnow{Float32}}, aps::CloudMicrophysics.Parameters.AirProperties{Float32}, var::CloudMicrophysics.Parameters.VarTimescaleAcnv{Float32}, Ndp::Float32}, CloudMicrophysics.Parameters.WaterProperties{Float32}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.Parameters.TurbulenceConvectionParameters{Float32}, ClimaAtmos.Parameters.SurfaceTemperatureParameters{Float32}}, ClimaAtmos.InitialConditions.var"#θ_pert#76"{Bool}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#139#140"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#135#136"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#133#134"{Float32}}, ClimaAtmos.InitialConditions.ColumnInterpolatableField{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VF{Float32, 101, Matrix{Float32}}, ClimaCore.Spaces.FiniteDifferenceSpace{ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}}, ClimaCore.Grids.CellFace}}, Interpolations.Extrapolation{Float32, 1, Interpolations.GriddedInterpolation{Float32, 1, Vector{Float32}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}}}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Interpolations.Flat{Nothing}}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#129#130"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#131#132"{Float32}}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, Type{Float32}, Bool}, Tuple{ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}}, Tuple{ClimaAtmos.AtmosModel{ClimaAtmos.BoxModel, ClimaAtmos.EquilMoistModel, ClimaAtmos.Microphysics1Moment, ClimaAtmos.QuadratureCloud, Nothing, Nothing, ClimaAtmos.Subsidence{AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#137#138"{Float32}}}, ClimaAtmos.RadiationISDAC{Float32}, Nothing, ClimaAtmos.ISDACForcing, Nothing, Bool, ClimaAtmos.UseAllTendency, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Bool, Nothing, Nothing, Nothing, Nothing, Nothing, ClimaAtmos.Explicit, ClimaAtmos.Explicit, Nothing, Nothing, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.IdealizedInsolation, ClimaAtmos.PrescribedSurfaceTemperature, ClimaAtmos.ConstantAlbedo{Float32}, ClimaAtmos.AtmosNumerics{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Nothing, Nothing}}}}}}}; kwargs::@Kwargs{always_inline::Bool})
    @ CUDA ~/.julia/packages/CUDA/Tl08O/src/compiler/execution.jl:364
 [17] macro expansion
    @ ~/.julia/packages/CUDA/Tl08O/src/compiler/execution.jl:112 [inlined]
 [18] auto_launch!(f!::typeof(ClimaCoreCUDAExt.knl_copyto!), args::Tuple{ClimaCore.DataLayouts.VIJFH{@NamedTuple{ρ::Float32, uₕ::ClimaCore.Geometry.Covariant12Vector{Float32}, ρe_tot::Float32, ρq_tot::Float32, ρq_rai::Float32, ρq_sno::Float32}, 100, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}, Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}, NTuple{5, Base.OneTo{Int64}}, typeof(ClimaAtmos.InitialConditions.atmos_center_variables), Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}, Nothing, ClimaAtmos.InitialConditions.var"#local_state#77"{ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, Nothing, @NamedTuple{cl::CloudMicrophysics.Parameters.CloudLiquid{Float32}, ci::CloudMicrophysics.Parameters.CloudIce{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}}, pr::CloudMicrophysics.Parameters.Rain{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ps::CloudMicrophysics.Parameters.Snow{Float32, CloudMicrophysics.Parameters.ParticlePDFSnow{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ce::CloudMicrophysics.Parameters.CollisionEff{Float32}, tv::CloudMicrophysics.Parameters.Blk1MVelType{Float32, CloudMicrophysics.Parameters.Blk1MVelTypeRain{Float32}, CloudMicrophysics.Parameters.Blk1MVelTypeSnow{Float32}}, aps::CloudMicrophysics.Parameters.AirProperties{Float32}, var::CloudMicrophysics.Parameters.VarTimescaleAcnv{Float32}, Ndp::Float32}, CloudMicrophysics.Parameters.WaterProperties{Float32}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.Parameters.TurbulenceConvectionParameters{Float32}, ClimaAtmos.Parameters.SurfaceTemperatureParameters{Float32}}, ClimaAtmos.InitialConditions.var"#θ_pert#76"{Bool}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#139#140"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#135#136"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#133#134"{Float32}}, ClimaAtmos.InitialConditions.ColumnInterpolatableField{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VF{Float32, 101, Matrix{Float32}}, ClimaCore.Spaces.FiniteDifferenceSpace{ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}}, ClimaCore.Grids.CellFace}}, Interpolations.Extrapolation{Float32, 1, Interpolations.GriddedInterpolation{Float32, 1, Vector{Float32}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}}}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Interpolations.Flat{Nothing}}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#129#130"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#131#132"{Float32}}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, DataType, Bool}, Tuple{ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}}}, Tuple{ClimaAtmos.AtmosModel{ClimaAtmos.BoxModel, ClimaAtmos.EquilMoistModel, ClimaAtmos.Microphysics1Moment, ClimaAtmos.QuadratureCloud, Nothing, Nothing, ClimaAtmos.Subsidence{AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#137#138"{Float32}}}, ClimaAtmos.RadiationISDAC{Float32}, Nothing, ClimaAtmos.ISDACForcing, Nothing, Bool, ClimaAtmos.UseAllTendency, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Bool, Nothing, Nothing, Nothing, Nothing, Nothing, ClimaAtmos.Explicit, ClimaAtmos.Explicit, Nothing, Nothing, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.IdealizedInsolation, ClimaAtmos.PrescribedSurfaceTemperature, ClimaAtmos.ConstantAlbedo{Float32}, ClimaAtmos.AtmosNumerics{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Nothing, Nothing}}}}}}, data::ClimaCore.DataLayouts.VIJFH{@NamedTuple{ρ::Float32, uₕ::ClimaCore.Geometry.Covariant12Vector{Float32}, ρe_tot::Float32, ρq_tot::Float32, ρq_rai::Float32, ρq_sno::Float32}, 100, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}; auto::Bool, threads_s::Tuple{Int64, Int64, Int64}, blocks_s::Tuple{Int64, Int64}, always_inline::Bool, caller::Symbol)
    @ ClimaCoreCUDAExt ~/.julia/packages/ClimaCore/fqQdO/ext/cuda/cuda_utils.jl:57
 [19] auto_launch!
    @ ~/.julia/packages/ClimaCore/fqQdO/ext/cuda/cuda_utils.jl:37 [inlined]
 [20] copyto!
    @ ~/.julia/packages/ClimaCore/fqQdO/ext/cuda/data_layouts_copyto.jl:45 [inlined]
 [21] copyto!
    @ ~/.julia/packages/ClimaCore/fqQdO/src/DataLayouts/copyto.jl:5 [inlined]
 [22] copyto!
    @ ~/.julia/packages/ClimaCore/fqQdO/src/Fields/broadcast.jl:149 [inlined]
 [23] copy(bc::Base.Broadcast.Broadcasted{ClimaCore.Fields.FieldStyle{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}}, ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Grids.ExtrudedFiniteDifferenceGrid{ClimaCore.Grids.SpectralElementGrid2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.RectilinearMesh{ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.XPoint{Float32}, Nothing}, LinRange{ClimaCore.Geometry.XPoint{Float32}, Int64}}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.YPoint{Float32}, Nothing}, LinRange{ClimaCore.Geometry.YPoint{Float32}, Int64}}}, Vector{CartesianIndex{2}}, Matrix{Int64}, CUDA.CuArray{Tuple{Int64, Int64, Int64, Int64, Bool}, 1, CUDA.DeviceMemory}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, CUDA.CuArray{Tuple{Bool, Int64, Int64}, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, @NamedTuple{}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.DeviceMemory}}, ClimaCore.Quadratures.GLL{4}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.XYPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, CUDA.CuArray{Float32, 4, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.IJFH{Float32, 4, CUDA.CuArray{Float32, 4, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, CUDA.CuArray{Float32, 3, CUDA.DeviceMemory}}, @NamedTuple{}}, ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, CUDA.CuArray{Float32, 2, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, CUDA.CuArray{Float32, 2, CUDA.DeviceMemory}}}, ClimaCore.Grids.Flat, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 101, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}}, ClimaCore.Grids.CellCenter}, typeof(ClimaAtmos.InitialConditions.atmos_center_variables), Tuple{Base.Broadcast.Broadcasted{ClimaCore.Fields.FieldStyle{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}}, Nothing, ClimaAtmos.InitialConditions.var"#local_state#77"{ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, Nothing, @NamedTuple{cl::CloudMicrophysics.Parameters.CloudLiquid{Float32}, ci::CloudMicrophysics.Parameters.CloudIce{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}}, pr::CloudMicrophysics.Parameters.Rain{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ps::CloudMicrophysics.Parameters.Snow{Float32, CloudMicrophysics.Parameters.ParticlePDFSnow{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ce::CloudMicrophysics.Parameters.CollisionEff{Float32}, tv::CloudMicrophysics.Parameters.Blk1MVelType{Float32, CloudMicrophysics.Parameters.Blk1MVelTypeRain{Float32}, CloudMicrophysics.Parameters.Blk1MVelTypeSnow{Float32}}, aps::CloudMicrophysics.Parameters.AirProperties{Float32}, var::CloudMicrophysics.Parameters.VarTimescaleAcnv{Float32}, Ndp::Float32}, CloudMicrophysics.Parameters.WaterProperties{Float32}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.Parameters.TurbulenceConvectionParameters{Float32}, ClimaAtmos.Parameters.SurfaceTemperatureParameters{Float32}}, ClimaAtmos.InitialConditions.var"#θ_pert#76"{Bool}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#139#140"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#135#136"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#133#134"{Float32}}, ClimaAtmos.InitialConditions.ColumnInterpolatableField{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VF{Float32, 101, Matrix{Float32}}, ClimaCore.Spaces.FiniteDifferenceSpace{ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}}, ClimaCore.Grids.CellFace}}, Interpolations.Extrapolation{Float32, 1, Interpolations.GriddedInterpolation{Float32, 1, Vector{Float32}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}}}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Interpolations.Flat{Nothing}}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#129#130"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#131#132"{Float32}}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, DataType, Bool}, Tuple{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}, ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Grids.ExtrudedFiniteDifferenceGrid{ClimaCore.Grids.SpectralElementGrid2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.RectilinearMesh{ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.XPoint{Float32}, Nothing}, LinRange{ClimaCore.Geometry.XPoint{Float32}, Int64}}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.YPoint{Float32}, Nothing}, LinRange{ClimaCore.Geometry.YPoint{Float32}, Int64}}}, Vector{CartesianIndex{2}}, Matrix{Int64}, CUDA.CuArray{Tuple{Int64, Int64, Int64, Int64, Bool}, 1, CUDA.DeviceMemory}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, CUDA.CuArray{Tuple{Bool, Int64, Int64}, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, @NamedTuple{}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.DeviceMemory}}, ClimaCore.Quadratures.GLL{4}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.XYPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, CUDA.CuArray{Float32, 4, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.IJFH{Float32, 4, CUDA.CuArray{Float32, 4, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, CUDA.CuArray{Float32, 3, CUDA.DeviceMemory}}, @NamedTuple{}}, ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, CUDA.CuArray{Float32, 2, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, CUDA.CuArray{Float32, 2, CUDA.DeviceMemory}}}, ClimaCore.Grids.Flat, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 101, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}}, ClimaCore.Grids.CellCenter}}}}, Tuple{ClimaAtmos.AtmosModel{ClimaAtmos.BoxModel, ClimaAtmos.EquilMoistModel, ClimaAtmos.Microphysics1Moment, ClimaAtmos.QuadratureCloud, Nothing, Nothing, ClimaAtmos.Subsidence{AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#137#138"{Float32}}}, ClimaAtmos.RadiationISDAC{Float32}, Nothing, ClimaAtmos.ISDACForcing, Nothing, Bool, ClimaAtmos.UseAllTendency, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Bool, Nothing, Nothing, Nothing, Nothing, Nothing, ClimaAtmos.Explicit, ClimaAtmos.Explicit, Nothing, Nothing, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.IdealizedInsolation, ClimaAtmos.PrescribedSurfaceTemperature, ClimaAtmos.ConstantAlbedo{Float32}, ClimaAtmos.AtmosNumerics{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Nothing, Nothing}}}}})
    @ ClimaCore.Fields ~/.julia/packages/ClimaCore/fqQdO/src/Fields/broadcast.jl:95
 [24] materialize(bc::Base.Broadcast.Broadcasted{ClimaCore.Fields.FieldStyle{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}}, Nothing, typeof(ClimaAtmos.InitialConditions.atmos_center_variables), Tuple{Base.Broadcast.Broadcasted{ClimaCore.Fields.FieldStyle{ClimaCore.DataLayouts.VIJFHStyle{100, 4, CUDA.CuArray{Float32, N, CUDA.DeviceMemory} where N}}, Nothing, ClimaAtmos.InitialConditions.var"#local_state#77"{ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, Nothing, @NamedTuple{cl::CloudMicrophysics.Parameters.CloudLiquid{Float32}, ci::CloudMicrophysics.Parameters.CloudIce{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}}, pr::CloudMicrophysics.Parameters.Rain{Float32, CloudMicrophysics.Parameters.ParticlePDFIceRain{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ps::CloudMicrophysics.Parameters.Snow{Float32, CloudMicrophysics.Parameters.ParticlePDFSnow{Float32}, CloudMicrophysics.Parameters.ParticleMass{Float32}, CloudMicrophysics.Parameters.ParticleArea{Float32}, CloudMicrophysics.Parameters.Ventilation{Float32}, CloudMicrophysics.Parameters.Acnv1M{Float32}}, ce::CloudMicrophysics.Parameters.CollisionEff{Float32}, tv::CloudMicrophysics.Parameters.Blk1MVelType{Float32, CloudMicrophysics.Parameters.Blk1MVelTypeRain{Float32}, CloudMicrophysics.Parameters.Blk1MVelTypeSnow{Float32}}, aps::CloudMicrophysics.Parameters.AirProperties{Float32}, var::CloudMicrophysics.Parameters.VarTimescaleAcnv{Float32}, Ndp::Float32}, CloudMicrophysics.Parameters.WaterProperties{Float32}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.Parameters.TurbulenceConvectionParameters{Float32}, ClimaAtmos.Parameters.SurfaceTemperatureParameters{Float32}}, ClimaAtmos.InitialConditions.var"#θ_pert#76"{Bool}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#139#140"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#135#136"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#133#134"{Float32}}, ClimaAtmos.InitialConditions.ColumnInterpolatableField{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VF{Float32, 101, Matrix{Float32}}, ClimaCore.Spaces.FiniteDifferenceSpace{ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, Matrix{Float32}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, Matrix{Float32}}}, ClimaCore.Grids.CellFace}}, Interpolations.Extrapolation{Float32, 1, Interpolations.GriddedInterpolation{Float32, 1, Vector{Float32}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Tuple{Base.ReshapedArray{Float32, 1, SubArray{Float32, 2, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}, Tuple{}}}}, Interpolations.Gridded{Interpolations.Linear{Interpolations.Throw{Interpolations.OnGrid}}}, Interpolations.Flat{Nothing}}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#129#130"{Float32}}, AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#131#132"{Float32}}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, DataType, Bool}, Tuple{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}, ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Grids.ExtrudedFiniteDifferenceGrid{ClimaCore.Grids.SpectralElementGrid2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.RectilinearMesh{ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.XPoint{Float32}, Nothing}, LinRange{ClimaCore.Geometry.XPoint{Float32}, Int64}}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.YPoint{Float32}, Nothing}, LinRange{ClimaCore.Geometry.YPoint{Float32}, Int64}}}, Vector{CartesianIndex{2}}, Matrix{Int64}, CUDA.CuArray{Tuple{Int64, Int64, Int64, Int64, Bool}, 1, CUDA.DeviceMemory}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, CUDA.CuArray{Tuple{Bool, Int64, Int64}, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, @NamedTuple{}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.DeviceMemory}}, ClimaCore.Quadratures.GLL{4}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.XYPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, CUDA.CuArray{Float32, 4, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.IJFH{Float32, 4, CUDA.CuArray{Float32, 4, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, CUDA.CuArray{Float32, 3, CUDA.DeviceMemory}}, @NamedTuple{}}, ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, CUDA.CuArray{Float32, 2, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, CUDA.CuArray{Float32, 2, CUDA.DeviceMemory}}}, ClimaCore.Grids.Flat, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 101, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}}, ClimaCore.Grids.CellCenter}}}}, Tuple{ClimaAtmos.AtmosModel{ClimaAtmos.BoxModel, ClimaAtmos.EquilMoistModel, ClimaAtmos.Microphysics1Moment, ClimaAtmos.QuadratureCloud, Nothing, Nothing, ClimaAtmos.Subsidence{AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#137#138"{Float32}}}, ClimaAtmos.RadiationISDAC{Float32}, Nothing, ClimaAtmos.ISDACForcing, Nothing, Bool, ClimaAtmos.UseAllTendency, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Bool, Nothing, Nothing, Nothing, Nothing, Nothing, ClimaAtmos.Explicit, ClimaAtmos.Explicit, Nothing, Nothing, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.IdealizedInsolation, ClimaAtmos.PrescribedSurfaceTemperature, ClimaAtmos.ConstantAlbedo{Float32}, ClimaAtmos.AtmosNumerics{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Nothing, Nothing}}}}})
    @ Base.Broadcast ./broadcast.jl:903
 [25] atmos_state(local_state::Function, atmos_model::ClimaAtmos.AtmosModel{ClimaAtmos.BoxModel, ClimaAtmos.EquilMoistModel, ClimaAtmos.Microphysics1Moment, ClimaAtmos.QuadratureCloud, Nothing, Nothing, ClimaAtmos.Subsidence{AtmosphericProfilesLibrary.ZProfile{AtmosphericProfilesLibrary.var"#137#138"{Float32}}}, ClimaAtmos.RadiationISDAC{Float32}, Nothing, ClimaAtmos.ISDACForcing, Nothing, Bool, ClimaAtmos.UseAllTendency, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Bool, Nothing, Nothing, Nothing, Nothing, Nothing, ClimaAtmos.Explicit, ClimaAtmos.Explicit, Nothing, Nothing, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.IdealizedInsolation, ClimaAtmos.PrescribedSurfaceTemperature, ClimaAtmos.ConstantAlbedo{Float32}, ClimaAtmos.AtmosNumerics{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Nothing, Nothing}}, center_space::ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Grids.ExtrudedFiniteDifferenceGrid{ClimaCore.Grids.SpectralElementGrid2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.RectilinearMesh{ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.XPoint{Float32}, Nothing}, LinRange{ClimaCore.Geometry.XPoint{Float32}, Int64}}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.YPoint{Float32}, Nothing}, LinRange{ClimaCore.Geometry.YPoint{Float32}, Int64}}}, Vector{CartesianIndex{2}}, Matrix{Int64}, CUDA.CuArray{Tuple{Int64, Int64, Int64, Int64, Bool}, 1, CUDA.DeviceMemory}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, CUDA.CuArray{Tuple{Bool, Int64, Int64}, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, @NamedTuple{}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.DeviceMemory}}, ClimaCore.Quadratures.GLL{4}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.XYPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, CUDA.CuArray{Float32, 4, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.IJFH{Float32, 4, CUDA.CuArray{Float32, 4, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, CUDA.CuArray{Float32, 3, CUDA.DeviceMemory}}, @NamedTuple{}}, ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, CUDA.CuArray{Float32, 2, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, CUDA.CuArray{Float32, 2, CUDA.DeviceMemory}}}, ClimaCore.Grids.Flat, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 101, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}}, ClimaCore.Grids.CellCenter}, face_space::ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Grids.ExtrudedFiniteDifferenceGrid{ClimaCore.Grids.SpectralElementGrid2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.RectilinearMesh{ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.XPoint{Float32}, Nothing}, LinRange{ClimaCore.Geometry.XPoint{Float32}, Int64}}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.YPoint{Float32}, Nothing}, LinRange{ClimaCore.Geometry.YPoint{Float32}, Int64}}}, Vector{CartesianIndex{2}}, Matrix{Int64}, CUDA.CuArray{Tuple{Int64, Int64, Int64, Int64, Bool}, 1, CUDA.DeviceMemory}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, CUDA.CuArray{Tuple{Bool, Int64, Int64}, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, CUDA.CuArray{Int64, 1, CUDA.DeviceMemory}, @NamedTuple{}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.DeviceMemory}}, ClimaCore.Quadratures.GLL{4}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.XYPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, CUDA.CuArray{Float32, 4, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.IJFH{Float32, 4, CUDA.CuArray{Float32, 4, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, CUDA.CuArray{Float32, 3, CUDA.DeviceMemory}}, @NamedTuple{}}, ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, LinRange{ClimaCore.Geometry.ZPoint{Float32}, Int64}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 100, CUDA.CuArray{Float32, 2, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, 101, CUDA.CuArray{Float32, 2, CUDA.DeviceMemory}}}, ClimaCore.Grids.Flat, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 100, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.XYZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 101, 4, CUDA.CuArray{Float32, 5, CUDA.DeviceMemory}}}, ClimaCore.Grids.CellFace})
    @ ClimaAtmos.InitialConditions ~/ClimaAtmos.jl/src/initial_conditions/atmos_state.jl:12
 [26] macro expansion
    @ ~/ClimaAtmos.jl/src/solver/type_getters.jl:641 [inlined]
 [27] macro expansion
    @ ./timing.jl:503 [inlined]
 [28] macro expansion
    @ ~/ClimaAtmos.jl/src/utils/utilities.jl:331 [inlined]
 [29] get_simulation(config::ClimaAtmos.AtmosConfig{Float32, ClimaParams.ParamDict{Float32}, Dict{String, Any}, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Tuple{String, String}})
    @ ClimaAtmos ~/ClimaAtmos.jl/src/solver/type_getters.jl:640
 [30] top-level scope
    @ none:1
--- Saving profiler information
srun: error: gpu-0: task 0: Exited with exit code 1
srun: launch/slurm: _step_signal: Terminating StepId=5075521.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jty53