Digikam/Tutorials/Switch from Microsoft OneDrive or WLPG to digiKam

From KDE Wiki Sandbox

Introduction

This tutorial is for people who would like to use open-source digiKam in-parallel or instead-of some Microsoft software that organizes photos (especially if you use people face tags):

DigiKam is an excellent replacement for discontinued WLPG and can even be made cross-compatible to use both digiKam and Microsoft software interchangeably (see configuration tutorial) but you should proceed with care and caution when you introduce digiKam or any other software that can alter your existing metadata.

In addition to this tutorial it is highly recommended to also be familiar with related tutorials by the same author: “Setup of digiKam for Windows compatibility” and “Tagging and Face Tags in digiKam”.

As an alternative to this tutorial, there is "Migrate from Windows Photo Gallery to digiKam" of author Farblos. It covers more of the metadata managed by Windows Photo Gallery than this tutorial, but at a certain cost: The migration process described there is one-way only, where you start with exclusive Windows Photo Gallery usage and end up with exclusive digiKam usage. And it requires a lot of command line interaction, most notably with ExifTool.

Back it up! Seriously

Before you allow digiKam (or any other metadata editor) import (show) your previously tagged image files, make sure you have taken all the precautions:

  1. You have to back-up all your existing image & video files, preferably so that you can very quickly copy them back to the main location (you may need to do it multiple times).
  2. Create a test photo gallery folder where you will copy-in a few files to test in digiKam, before you give digiKam access to your real photo gallery.
  3. Make an additional Permanent backup of photo metadata – see below. This is to save you from any issues that may not even be related to migration to digiKam.
  4. Test as much as possible before you accept that migration was successful and consider overwriting your older backups with digiKam modified files:
    1. Try tagging over 100 photos will all types of tags that you use.
    2. Try searching for photos using different tags (hierarchical, star rating, etc.).
    3. Check how files from different cameras, different sources and formats have migrated.
    4. Make sure you know what will happen with video metadata, people tags on rotated pictures and other potential issues (see "Microsoft image metadata" section).
    5. If you plan to use cross-compatibility with OneDrive tagging, WLPG or other software – try to repeatedly make edits in one program then observe the changes in another so that you have files where each type of metadata was over-written in for example:
      • WLPG → digiKam → WLPG → digiKam
      • digiKam → WLPG → digiKam → WLPG
      • (there should not be any lost, duplicated or abnormal tags created from any of the edits. Exception - digiKam will likely create descriptive tags for all People Face Tags. This is not a significant issue for cross-compatibility.)
  5. Keep your Microsoft Photo Gallery installed as long as possible. This is one more advantage of cross-compatible digiKam setup – you should be able to use old Photo Gallery in parallel to digiKam in case you need to access some video metadata or any other reason.

Permanent backup of photo metadata

Here is an easy method to make a complete metadata backup from all your photos that will easily fit on one M-Disk DVD or a cheap flash drive. You never know when some virus, child or silly mistake will irreversibly wipe or corrupt your metadata. If you notice the damage too late, all your backups may by then already hold the corrupt files and only such permanent backup will be able to help. It is not only good for tags you add manually, but also to preserve more basic metadata like the date when a picture was taken.

The basic principle is that metadata, including all camera data, tags and people face tags are completely resolution-independent – you can dramatically resize and compress image data while preserving 100% of the metadata. Such conversion must be doable by many ways, including in digiKam (please edit this tutorial to add your favorite method), but my personal favorite tool for such metadata backup is:

Method A
  1. Open the free "FastStone Image Viewer" for Windows so that you see the main toolbar (not full-screen mode). This program is reliable and has minimal risk of messing up your files.
  2. Go to Tools → Batch Convert ... and "Add" all your photo gallery folders to the "Input List".
  3. Next to “Output Format” set JPEG and in the “Settings” choose quality for example 70%. (we want images to be recognizable, not beautiful).
  4. Under "Use Advanced Options" choose “Resize based on long side” to for example 1024px. (again just to keep images recognizable, you can go lower if you need to save more space).
  5. Under “Rename” it is nice to add for example 
    *_metadata_bckp
    to each file, so that you better recognize the highly compressed files if you will work with them.
  6. Choose a reasonable output folder and press Convert (folder structure will be preserved).
  7. When everything is done, I suggest you add a text file note with date when you did the metadata backup and for what reasons. Then compress everything with 7zip to reduce the size even more and have one easy to move file for permanent storage.
Note
Such batch conversion in "FastStone Image Viewer" will not back up metadata of videos and likely will skip RAW files. We assume that the key metadata is stored in normal files such as .jpg or in the file and folder names. If you have sidecar .xmp files, they would need a different backup approach, but for backing up from Windows Live Photo Gallery, this method is almost perfect.


Method B

Use command line ExifTool to create recursive metadata-only backups as described in detail here: Farblos/wpg2dk - backup. This method is more powerful than "method A", because ExifTool is also a tool that can be used to restore your damaged metadata (following the same tutorial by Farblos).

Microsoft image metadata

Before you start using digiKam to manage your WLPG tagged gallery, it is good to understand where the potential risks and issues are. Depending on how you used tags in WLPG, some of these concerns may not be applicable to you!

  1. Tag hierarchy needs care – Default Metadata settings in digiKam are prone to create many loose tags out of the nicely organized hierarchies such as for example “Location/USA/New York/Central Park”. If you set up digiKam correctly ("Setup of digiKam for Windows compatibility"), this will be 100% resolved.
  2. Face / People tags are a challenge – digiKam can fully support people tags as they were handled in WLPG (only for backwards compatibility to Microsoft software, you need correct metadata settings in digiKam). More importantly most (or all) vertical orientation images will have “Face tag rotation issue” which is covered in detail further below.
  3. No Flag – setting “flag” in WLPG is not saved to any file (only internal database), so this data is lost any time you move to another computer or another program [source].
  4. No metadata for videos. In contrast to photos, common video file formats do not support embedded metadata. As a result WLPG only stores video metadata in the internal database without any method to export the data Therefore WLPG video metadata is lost any time to you move to another computer or another program. DigiKam actually solves this issue by creating extra .xmp sidecar files next to each un-writable file, so as long as the extra files are not lost, digiKam will always preserve your video tags in the future. In fact, to be safe, it is a good idea to always keep year-month-day date and some keywords in the filename of important video files, because it is easy to separate video file from its metadata.
  5. Beware of "Read-Only" files! – It is worth double-checking that you do not have any Read-Only files and folders in your collection. We must have all metadata stored inside the files for digiKam to read it, but no software will write your metadata into files if the files are write-protected.
    • In case, you just changed some read-only files to writable, you need to open WLPG and to be safe - add some new tag to all files that were previously Read-Only. Wait for WLPG to write that tag along with all other metadata to all files (then you can easily remove that dummy tag in WLPG or later in digiKam). + you can always double-check which metadata is saved in a file by opening file properties in Windows Explorer.
  6. The rest of the metadata is straightforward. There is marginal risk that captions, star rating and individual descriptive tags from WLPG or OneDrive would get lost. Only reason for this data to get lost is if the image files were set to "Read-Only" or if WLPG did not manage to synchronize tag information to files as it is supposed to. In fact, most software can deal with these more basic tags and you may find other software, such as built-in Windows tools as good-enough alternatives to digiKam which still save metadata into files.

For more information you are very welcome to read excellent articles by José Oliver-Didier which cover a lot more about Microsoft image metadata specifics: https://jmoliver.wordpress.com/2017/02/12/accessing-windows-photo-gallery-metadata-using-exiftool/ , https://jmoliver.wordpress.com/2020/02/16/onedrive-photo-captions-and-tags/ and other related blog posts.

Tip - use digiKam to examine your metadata structure.
digiKam offers excellent built-in ExifTool integration in the right side "Metadata" panel that can be configured in the digiKam settings to show 100% of the Metadata in your images. Together with "XMP" and "IPTC" tabs in the same "Metadata" panel you have a convenient tool to see how the imported metadata looks when digiKam opens your old tagged file for the first time and then see which metadata is written by digiKam after you try to change old tags and add new ones. (ideally, we want digiKam to overwrite all existing metadata fields when you make an edit - so that file does not contain both current and left-over obsolete Tags.)


The Migration

Now when you are ready to Switch to using digiKam you need to decide on one of the 2 routes:

  1. Simple data import using Metadata written inside your image files - approach that can provide nice results for most people (this tutorial is mostly about this approach - see the limitations of this approach in the paragraphs above and below).
    • Using this approach - you only need to correctly configure digiKam and then let it scan your photo collection - digiKam will just read all the Metadata out of your image files.
    • Main Dis-advantage of the simple approach - Tags and ratings of videos will be lost along with "Flags" (if you use them) + the Face Tag rotation issue asks for some manual work (which can be done at a later point).
  2. Use the advanced "WPG-metadata-database-to-XMP" method by Farblos: https://github.com/farblos/wpg2dk . The main things consider:
    • The database migration method is applicable only once - usually only if you have not started editing your photo gallery in digiKam previously.
    • You have WLPG installed - the method reads data out of WLPG database, not image files.
    • Main advantage - This method can export metadata on videos, "Flags" and automatically fixes the Face Tag rotation issue (migration is more comprehensive than the simple approach - assuming you have a lot of tagged videos, you use "Flags" or know another reason why the simple approach does not meet your requirements).
    • Main Dis-advantage 1 - This method requires a lot of command line steps - not an easy process.
    • Main Dis-advantage 2 - Any time and convenience gained compared to the "Simple data import" will likely be offset by the increased risk of data damage and more additional testing needed to verify that everything worked well.


Microsoft Face tag rotation issue

Microsoft’s face tags (usually from WLPG) are affected by long-term-bug which results in face regions being shown rotated by 90 degrees to one side on some of the pictures.

It seems that we want a one-click fix for the whole collection, but in practice it is not very safe to run a script which will make changes to hundreds or thousands of images which are distributed all around your collection. The backups and testing to avoid potential disasters would take much more time than the “one-click” that we all would like.

Therefore I recommend a semi-automated approach which takes 2-3 button presses per image, but includes direct verification that you are making correct changes only where needed.


Root cause and potential workaround

As long as the image “Exif orientation tag” is not “Normal”, Microsoft WLPG will always treat face tags differently than all other software that recognizes “Exif orientation tag” (for example digiKam). Therefore to make a file universally interpreted by all software including Microsoft WLPG any affected file needs to be losslessly rotated so that “Exif orientation tag” is re-set to “Normal”.

The issue exists because Microsoft is the only company in the world which does not correctly support image rotation tags which are used by most digital cameras.

In practice, the problem is almost exclusively present in portrait orientation images with face tags (but only if the image contained “Exif orientation tag” which is typically written by digital cameras with an orientation sensor - again not in all cameras).

Fixing one file at a time

In case you have only a few affected files, it is possible to make corrections on each individual file – please see the other tutorial for information on how to Rotate face tags on individual files.

Semi-automated approach

WARNING - not yet available
before the “Rotate Face Tags” feature (id=447674) is implemented into digiKam, this approach is not completely feasible. But you can perform the complete procedure at a later point, after you are already using digiKam.

https://bugs.kde.org/show_bug.cgi?id=447674

This approach practically does not require any testing, because we will manually check and rotate face tags on each potentially affected image. When done correctly it will take a couple of seconds per file and most likely you have around 1000 affected images (1000 x 3sec = 50min of work). Plus this approach can be used at any time – when you migrate your collection, after you have already fixed a few images or when you add a couple or thousands more images that need to be fixed.

Important
Perform this face tag rotation fix only when you are confident that your digiKam metadata settings are good and your data is safe


  1. Open the Search panel
  2. Click Advanced Search
  3. Select images which have “People” Tags. (When digiKam reads in Microsoft face tags, it automatically creates descriptive tags with “People” parent tag.)
  4. Select “In Tree” option because we are interested in sub-tags of the “People” parent tag
  5. Select “Portrait Orientation” (because this is the orientation data which causes face tag rotation issue in Microsoft’s software).
  6. You can add additional conditions, for example limit the search to specific Album, Date or camera model. This is especially useful if you are adding new photos to a collection that was previously fixed.
  7. Press OK to find all photos that are most likely affected by the issue. (Note, you can always save the current search if you like.)
  8. Now you have a list of all photos that need to be checked by simply switching to the next photo one after another. Make sure you have “Show Face Tags” activated to directly see all face tags on every photo.
  9. The actual fix is done by using “Rotate Face Tags” command which has to be mapped to a keyboard shortcut. (Note, this feature was not yet available in v 7.4, but hopefully is there now.)
  10. The rest of the workflow is:
    • “Look if the face tags are wrong”, if Yes,
    • “Press the right keyboard shortcut to rotate face tag into correct direction”, then
    • “Press Arrow Right to see next photo”.

Fully automatic solution

Since March 2022 there actually is one fully-automatic solution for the face tag rotation issue (along with some other features) - https://github.com/farblos/wpg2dk . Note that this method is Not easy to follow and is suitable for very dedicated users only. Even if a fully-automatic solution would be easy to use (“require just one-click to run and forget"), I personally would still use the Semi-automated approach for data safety and in most cases also to actually save time (manual face tag changes take less time than all the precautions and verification necessary to run a rarely-used script that executes low-level Metadata changes on your whole photo gallery).

Notice that even if you fix all your old files at some point, you may still get more affected images in the future.


Other Metadata issues

For most users, the instructions above will give great results and you do not need to read any further. In case you encounter specific problems, here are some extra hints:

Summary

Metadata is tricky – different software looks for different data and interprets it in different ways. Nevertheless, everyone can retain all their metadata from WLPG and other Microsoft software in digiKam. But with some additional digiKam configuration you will also retain cross-compatibility to continue searching through your metadata directly in Windows, OneDrive or any other Microsoft software.

Finally, if you make some configuration mistake – it is not the end of the world. Perhaps just using your previous backup will solve your issue, but even in really bad cases there are command-line tools such as ExifTool can come to the rescue.



About this tutorial

This tutorial was originally written for digiKam 7.4 in 2021-12. You are very welcome to edit, update and Upgrade this tutorial page to make it better and keep it up to date. No attribution or copyrights on this page (all illustrations are original as well)!

Retrieved from "https://wikisandbox.kde.org/index.php?title=Digikam/Tutorials/Switch_from_Microsoft_OneDrive_or_WLPG_to_digiKam&oldid=470790"