use_paired_jpg_as_mipmap

[wpferguson]

Canon R series cameras don't include a full size embedded jpeg so generating mipmap cache for anything larger than mipmap 3 requires the raw to be loaded and processed. This script copies the paired jpeg from the RAW+JPEG pair to the full resolution mipmap. This can then be used to generate the remaining mipmap cache. The jpeg image can be discarded after being copied to the cache or retained based on a preference in Lua options.

Fixes darktable-org/darktable#19470

[jdchristensen]

It looks like "responsive_cache" needs to be updated to this script name.

[jdchristensen]

I misunderstood what the message function does, so now I see that this is interprocess communication, not a message to the user. So my previous comment should be ignored. Is the responsive_cache script available somewhere? What happens if the user doesn't have it installed?

[wpferguson]

Once upon a time long, long ago in a...

Before the crawler existed the only way to generate cache was darktable-generate-cache. The problem was it generated all the cache sizes for all of the images in the database. What I wanted was a way to generate just the cache I needed, when I needed it (right after import). So I added dt_lua_image_t:generate_cache() to the Lua API and started working on a script, gen_cache, to generate just the cache I needed (at that time HD and 2K) right after an import. The next iteration was import_cache, which was smarter with more features and options. The third iteration is responsive_cache which finally satisfies all my needs. It runs after import generating size 3 and 6 cache. When I edit I tend to do long sessions hitting the space bar to go to the next image. I hated waiting when I returned to lighttable for all the thumbnails to reload, so responsive_cache regenerates the updated thumbnail in background after I change to the next image and when I return to lighttable all the thumbnails are up to date.

I have an R7 and the largest embedded JPG is mipmap size 3. So I could generate cache for size 3 at 100+ images/sec. When I did size 6 it was a little more than 3/sec. I shoot sports so most imports are 1000+ images and generating the cache took 15 minutes or so. While responsive cache was running, no other scripts could run because Lua is single threaded. So, I added dt.util.message() so that I could have "cooperative multitasking". With the addition of a simple FIFO scheduler I could "pause" responsive_cache and let other scripts run then resume. When I added use_paired_jpg_as_mipmap, responsive_cache would try and run before use_paired_jpg_as_mipmap, so then I changed responsive_cache to not run on import and wait for a message from use_paired_jpg_as_mipmap to run. Now cache generation after import is around a minute.

The crawler works, but you have to wait for it and it generates way too much cache. I tried it as Hanno was implementing it and let it run for awhile and it ran me out of disk space. It will be even worse now since the new max mipmap size is 10, so it will generate size 8 and 9 too.

I hadn't thought to add responsive_cache to the "merge fest" but I guess I could...

[jdchristensen]

Should the commented out lines be removed?

[wpferguson]

Probably. When I write a script I use a boilerplate and a bash script that fills in some of the blanks. I tend to leave the above lines commented out because too often I found myself adding something and needing another library. But, when I'm done I'm sometimes lazy and don't clean up after myself.

[jdchristensen]

It looks like there are some other functions defined that aren't used. I read through all of them when learning how the script worked, only to find that some of them were unused, so it would be helpful to trim to just what is needed.

[jdchristensen]

Suggested change

	for smaller mipmap sizes can be satisfied by down sampling the full resolution mipmap. User's can
	for smaller mipmap sizes can be satisfied by down sampling the full resolution mipmap. Users can

[wpferguson]

I tend to use apostrophes way too often... :-(

[jdchristensen]

I just tested this on Darktable 5.4.1, importing 96 raw files with 96 paired jpg files (in the same folder, with deletion enabled). After import, I hit F for a full screen view and then zoomed to 100% and started quickly moving through photos. Darktable started fully using all 8 cpu cores (but no gpu), and the UI became frozen for a bit with a "working..." message on the screen. After about two minutes it became responsive again. ... Oh, I now see that I accidentally had OpenCL disabled. (It's glitchy for me after a suspend-resume cycle.) So it looks like DT was generating the mipmap cache from the raw files, which I didn't expect to happen given that this script had already created full size mipmaps. [Edit: Based on the tests below, I don't think it was using the raw files to generate the mipmap cache.]

Relevant settings:

• Use raw file instead of embedded jpg: never
• high quality processing from size: never
• enable disk backend for thumbnail cache: checked
• enable disk backend for full preview cache: checked
• generate thumbnails in background: WQXGA

Maybe the last one caused an issue? But I thought it would have just downscaled the jpgs already in the cache. And I thought that this only ran when darktable was idle for at least 5s.

[jdchristensen]

Let me know what log options I can use to provide more info. (And of course thanks so much for providing this!)

[jdchristensen]

I started over, removing the folder from darktable, and restoring the jpgs. I changed the start time for the background thumbnails to 25s, and made sure to not be idle that long. I also enabled OpenCL. Still, darktable used all 8 cpu cores right after import.

I started over again, this time completely disabling the background thumbnail generation ("never"). Still, after I had scrolled through a few of the images, darktable started using all 8 cpu cores and no GPU.

Another issue: for some reason, after each attempt, between 10 and 20 of the jpg files were not deleted (out of the 96 I had).

[jdchristensen]

As another experiment, I wrote a version that works when you import a folder that just contains raw files, with the corresponding jpg files available in a jpg subfolder (or really, a symlink to the right place). This avoids having to import the jpg and then remove it from darktable. Also, I hardlink the jpg to the cache dir, so no data needs to be written to disk, and therefore this is very fast. (This only works because my cache directory is on the same filesystem as my images. If you want to get rid of the jpg files, then instead a "mv" is what would accomplish this quickly.) It runs as each image is imported, rather than at the end. I verified that it correctly and quickly makes the hardlinks.

However, again I found that when I start viewing the images at 100%, darktable uses all 8 cores and the UI gets very laggy. Is it possible that darktable doesn't know about these files that got added to the cache, and we need to tell it to update its view of the cache? I also noticed that no cache files got created at different sizes, besides the ones at size 8. (I'm on 5.4.1.)

-- Rough draft.  Needs checking of return codes, etc.
dt.register_event(MODULE, "post-import-image",
  function(event, image)
    dt.print_log("Importing " .. image.filename)
    local extension = string.lower(df.get_filetype(image.filename))
    if not string.match(extension, "jpg") then
      local mipmap_dir = get_mipmap_dir() .. PS .. MAX_MIPMAP_SIZE .. PS
      local rc = df.mkdir(mipmap_dir)
      dt.print_log("rc from mkdir is " .. rc)
      local basename = df.get_basename(image.filename)
      local raw_image_id = image.id
      -- todo: check existence; handle JPG vs jpg
      local fname = image.path .. PS .. "jpg" .. PS .. basename .. ".JPG"
      local mipname = mipmap_dir .. raw_image_id .. ".jpg"
      -- todo: quote filenames:
      local command = "ln " .. fname .. " " .. mipname
      dt.print_log("running: " .. command)
      rc = dtsys.external_command(command)
      dt.print_log("rc: " .. rc)
      -- If successful, could also trigger creation of size 5 here.
      -- Or could do via responsive_cache in the background after import.
    end
  end
)

[wpferguson]

The above code will work on Linux and probably MacOS, but not on Windows.

Let me know what log options I can use

I'll add some debug logging so that we can see what is going on.

It runs as each image is imported, rather than at the end

I usually avoid that because on a slow filesystem (or if you're doing a lot) the import slows to a crawl. It's safer to buffer up the images and process them in a job after so darktable isn't "hung" waiting on the import to complete.

[jdchristensen]

The above code will work on Linux and probably MacOS, but not on Windows.

Yes, it's very specific to my situation, so I don't think it's useful for others as it is. But the idea of using "mv" rather than "copy and later delete" in the case the user wants the jpg files deleted might be a good one. On some filesystems, "mv" will be instantaneous, but it will fall back to doing the right thing. And there could be a "hardlink_or_copy" function that does a hardlink on systems that support it, and falls back to a copy when needed.

Let me know what log options I can use

I'll add some debug logging so that we can see what is going on.

Great, I'll definitely test more. Let me know if I should test with an empty config dir, and if so, if there are some settings I should change before doing the import.

The fact that some jpg files remained at the end is really suspicious, so some debugging related to that might be helpful. Maybe the issue has to do with the jpg files getting imported, and then removed one by one after the import is done? Can we instead use the post-import-image hook to prevent them from being imported in the first place (but still keeping a list of them to use in the post-import-film hook)?

It runs as each image is imported, rather than at the end

I usually avoid that because on a slow filesystem (or if you're doing a lot) the import slows to a crawl. It's safer to buffer up the images and process them in a job after so darktable isn't "hung" waiting on the import to complete.

In general, I agree, but hardlinks are instantaneous, so this is really fast in my situation, and I know when the import is done that the script has finished.