I'm using CloudPath.walk to search a large nested directory, but I'm only descending into a very small subset of subdirectories (by removing elements from dirnames). However, it seems the current walk implementation builds the entire file tree in advance before walking. Have you considered an iterative walk similar to os.walk to support sparse searches like this? I imagine this would require more network requests, but possibly save time and data requested if the search is sufficiently sparse. An iterative implementation might also be more consistent with users' expectation, coming from os.walk/Path.walk.
Thanks for this super useful library :)
I'm using
CloudPath.walkto search a large nested directory, but I'm only descending into a very small subset of subdirectories (by removing elements fromdirnames). However, it seems the currentwalkimplementation builds the entire file tree in advance before walking. Have you considered an iterative walk similar toos.walkto support sparse searches like this? I imagine this would require more network requests, but possibly save time and data requested if the search is sufficiently sparse. An iterative implementation might also be more consistent with users' expectation, coming fromos.walk/Path.walk.Thanks for this super useful library :)