Create tutorial to get ZTF light curves from HATS catalog#325
Create tutorial to get ZTF light curves from HATS catalog#325jaladh-singhal wants to merge 4 commits into
Conversation
|
|
||
| with Client(n_workers=get_nworkers(ztf_lc_cone), | ||
| threads_per_worker=1, | ||
| memory_limit=None # each partition can be several GB; avoid per-worker cap |
There was a problem hiding this comment.
It was running out of memory locally without this. Let's see how it performs on CI.
| right_on='oid', | ||
| how='inner' | ||
| ) | ||
| combined_df |
There was a problem hiding this comment.
Just FYI, I tried 3 more approaches before settling on this one (which takes takes 2m±5s both compute calls combined):
ztf_objects_cone.join(ztf_lc_cone, ...).compute()takes 2m±5sztf_objects_cone.merge(ztf_lc_cone, ...).compute()takes 2m±5sztf_objects_cone.compute(); ztf_lc_cone.id_search(values={'objectid': list(ztf_objects_cone_df.oid)}).compute()took 2m50s
I kept this approach because of time as well as to maintain the narrative of keeping objects table search optional.
|
I've pushed a commit directly that fixes the oldestdeps job failure, review is to come separately |
|
I'm not sure what goes on in circleCI, we may actually hit that memory limit even though the graph doesn;t show it (but the resolution of the graph is pretty bad, so it's still my prime suspect for the reason for the failure) -- keep an eye on the GHA buildhtml job instead for now. |
|
@bsipocz buildhtml job is getting skipped (I tried re-triggering it) |
2 participants
Fixes IRSA-7768