Group and order high zoom metatile jobs by rawr size threshold #88
Conversation
…er to order jobs by rawr size
|
|
||
| missing_low[c] = True | ||
| missing_low[this_coord] = True |
| counts_at_zoom[this_coord.zoom] += 1 | ||
| else: | ||
| # too big for this zoom, queue the children | ||
| top_left_child = this_coord.zoomBy(1) |
There was a problem hiding this comment.
I thiiink the tile object has a children() function that returns this list so you don't have to calculate it?
| coord, sizes = task.get() | ||
| all_sizes.update(sizes) | ||
|
|
||
| # now use all_sizes plus the size_threshold to find the lowest zoom we can group each coordinate into |
There was a problem hiding this comment.
Can you add another line or two of comment here? My intuition is that we're trying to group tiles by estimated memory usage so that we get to but don't go beyond a target usage. What you have here is good, but maybe needs the extra bit of info about why grouping is happening.
|
|
||
| # validate counts by zoom - expecting the equivalent of 4^10 zoom 10 jobs. | ||
| counts_at_zoom_sum = 0 | ||
| for z in counts_at_zoom.keys(): |
There was a problem hiding this comment.
This is equivalent to:
| for z in counts_at_zoom.keys(): | |
| for z in counts_at_zoom: |
On second thought, maybe what you have is easier to read and you should ignore me.
| count_at_this_zoom = counts_at_zoom[z] | ||
| zoom_10_equiv_count = count_at_this_zoom * (4 ** (10 - z)) | ||
| counts_at_zoom_sum += zoom_10_equiv_count | ||
| if counts_at_zoom_sum == 4**10: |
There was a problem hiding this comment.
Maybe extract 4**10 into a single variable so this check and the math below are easier to read.
| cfg.yml["batch"]["memory"] = int(min(cfg.yml["batch"]["memory"] * mem_multiplier, mem_max)) | ||
| # now that we know what we want, pick something AWS actually supports | ||
| viable_mem_request, required_min_cpus = viable_container_overrides(adjusted_mem) | ||
| print("REMOVEME: [%s] enqueueing %s at %s mem mb and %s cpus" % (time.ctime(), coord_line, viable_mem_request, required_min_cpus)) |
There was a problem hiding this comment.
Remove this line or remove the REMOVEME
Instead of simply splitting between grouping at size 7 and 10, group at sizes 7, 8, 9, and 10 to minimize build time. Also order by sum of constituent rawr tile sizes because they correlate strongly with build times.
This was confirmed working in last tile build.