Skip to content

Commit

Permalink
source-asana: don't cache Tasks records
Browse files Browse the repository at this point in the history
A production task that only has the `Tasks` binding enabled is OOM-ing
because every `Tasks` record is being cached. I suspect this was done
because `Tasks` is used as a parent stream for some other streams. Not
caching `Tasks` records will likely slow down those other streams, but
right now no one is using those streams & we'd have to address these
OOMs due to caching regardless.

I also added some logging to give some indication that the connector is
doing something while the connector works on the full refresh `Tasks`
stream.
  • Loading branch information
Alex-Bair committed Dec 5, 2024
1 parent 9fc0ac4 commit c399df8
Showing 1 changed file with 5 additions and 3 deletions.
8 changes: 5 additions & 3 deletions source-asana/source_asana/streams.py
Original file line number Diff line number Diff line change
Expand Up @@ -179,10 +179,14 @@ class ProjectRelatedStream(AsanaStream, ABC):
"""

def stream_slices(self, **kwargs) -> Iterable[Optional[Mapping[str, Any]]]:
yield from self.read_slices_from_records(
project_slices = self.read_slices_from_records(
stream_class=Projects, slice_field="project_gid"
)

for s in project_slices:
self.logger.info(f"Fetching {self.name} for project {s["project_gid"]}.")
yield s


class AttachmentsCompact(AsanaStream):
use_cache = True
Expand Down Expand Up @@ -480,8 +484,6 @@ def path(self, **kwargs) -> str:


class Tasks(ProjectRelatedStream):
use_cache = True

def path(self, **kwargs) -> str:
return "tasks"

Expand Down

0 comments on commit c399df8

Please sign in to comment.