HDDS-14721. Make OM bootstrap implementation configurable.

[sadanand48]

What changes were proposed in this pull request?

With the recent bootstrap V2 impl done as part of HDDS-12984, make the impl controllable via a config.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14721

How was this patch tested?

unit and integration tests

[adoroszlai]

Thanks @sadanand48 for the patch.

[yandrey321]

are we going to retry after the next start or files would stay there forever?

[sadanand48]

We only delete the files in case of v2 , yeah they are not cleaned up later if not for the first time. Shouldn't cause functional issues as they are not sst files but will contribute to the space usage on OM.

[yandrey321]

we need a solution for cleaning up files in case of getting exception here.

[SaketaChalamchala]

we need a solution for cleaning up files in case of getting exception here.

I agree we should cleanup stale files on OM but would recommend addressing the failed cleanup in separate fix and keep this limited to making bootstrap version configurable.

[yandrey321]

how do we handle exception when something cannot be moved? or resolved

[sadanand48]

The IOException from the failed move would propagate to eventually return null here thereby leading to a failed installSnapshotAttempt. this would be retried from Ratis again.

public synchronized TermIndex installSnapshotFromLeader(String leaderId) {
    if (omRatisSnapshotProvider == null) {
      LOG.error("OM Snapshot Provider is not configured as there are no peer " +
          "nodes.");
      return null;
    }

    DBCheckpoint omDBCheckpoint;
    try {
      omDBCheckpoint = omRatisSnapshotProvider.
          downloadDBSnapshotFromLeader(leaderId);
    } catch (IOException ex) {
      LOG.error("Failed to download snapshot from Leader {}.", leaderId,  ex);
      return null;
    }

[yandrey321]

v2 is not descriptive name, it's not clear what is the difference between v1 and v2.

[sadanand48]

v1 uses OMDbCheckpointServlet , v2 uses OMDBCheckpointServletInodeBasedXfer. I've changed it to ozone.om.db.checkpoint.use.inode.based.transfer now.

[SaketaChalamchala]

LGTM

[yandrey321]

we need a solution for cleaning up files in case of getting exception here.

[yandrey321]

lgtm

[aswinshakil]

This only checks if the directory exists, What happens if the checkpointLocation/om.db already exists from a previous failed bootstrap attempt? Then we would skip this and we would have an inconsistent and incompleteom.db

The candidate directory is only cleaned up in two cases:

• Leader changes (checkLeaderConsistency calls init())
• downloadDBSnapshotFromLeader returns successfully (the finally block in installSnapshotFromLeader calls cleanupCheckpoint())

If downloadDBSnapshotFromLeader throws, the catch block returns null with no cleanup:

DBCheckpoint omDBCheckpoint;
    try {
      omDBCheckpoint = omRatisSnapshotProvider.
          downloadDBSnapshotFromLeader(leaderId);
    } catch (IOException ex) {
      LOG.error("Failed to download snapshot from Leader {}.", leaderId,  ex); --> No cleanup,  Can have partial OM DB.
      return null;
    }

In case moveActiveDbFilesToOmDbIfNeeded fails in between in the previous checkpoint, The new checkpoint will skip this because we have checkpointLocation/om.db directory from the previous failed checkpoint.