HBASE-29958 Improve log messages by sharmaar12 · Pull Request #7857 · apache/hbase

sharmaar12 · 2026-03-05T15:27:44Z

No description provided.

anmolnar · 2026-03-05T22:35:27Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java

-            + "(read-write) mode on this storage location. Active Cluster Id: {} " + acs
-            + " This cluster Id: " + getClusterId());
+            + "(read-write) mode on this storage location. Active Cluster Id: " + acs
+            + ", This cluster Id: " + getClusterId());


getClusterId() alone is not enough. We need to log the cluster id + suffix like this:

dcf9e176-8437-49a9-9e91-b221b94e8fe3:<blank>

or

dcf9e176-8437-49a9-9e91-b221b94e8fe3:replica1

Updated the code accordingly. Sorry missed that part.

sharmaar12

Updated the code please review.

anmolnar

This is how it looks like with this patch:

2026-03-09T15:02:52,121 INFO  [main] hbase.TableName: Read Replica Cluster suffix value:
2026-03-09T15:02:53,260 INFO  [master/hbase-docker:16000:becomeActiveMaster] master.MasterFileSystem: Read Replica Cluster: This is the active cluster on this storage location with cluster id: 0115d976-1306-44c7-b8b6-f2644f8867c9:

I suggest to use "<blank>" value in the log messages if the suffix is empty, because log messages currently are very confusing to me.

Second, what does this log message mean?

2026-03-09T15:02:53,259 WARN  [master/hbase-docker:16000:becomeActiveMaster] util.FSUtils: Read Replica Cluster id file file:/data-store/hbase/active.cluster.suffix.id is empty

What's empty? Why is it empty?
I restarted the active cluster and the active cluster suffix file was present with the cluster id + blank suffix value: 0115d976-1306-44c7-b8b6-f2644f8867c9:<blank>

sharmaar12 · 2026-03-09T15:42:27Z

This is how it looks like with this patch:
2026-03-09T15:02:52,121 INFO  [main] hbase.TableName: Read Replica Cluster suffix value:
2026-03-09T15:02:53,260 INFO  [master/hbase-docker:16000:becomeActiveMaster] master.MasterFileSystem: Read Replica Cluster: This is the active cluster on this storage location with cluster id: 0115d976-1306-44c7-b8b6-f2644f8867c9:
I suggest to use "" value in the log messages if the suffix is empty, because log messages currently are very confusing to me.

Will do the changes accordingly.

Second, what does this log message mean?
2026-03-09T15:02:53,259 WARN  [master/hbase-docker:16000:becomeActiveMaster] util.FSUtils: Read Replica Cluster id file file:/data-store/hbase/active.cluster.suffix.id is empty
What's empty? Why is it empty? I restarted the active cluster and the active cluster suffix file was present with the cluster id + blank suffix value: 0115d976-1306-44c7-b8b6-f2644f8867c9:<blank>

@anmolnar I am not much familiar with the nitty-gritty of this file based implementation as some other engineer worked on this but IIRC this may means that active.cluster.id.file may have been created without any data in it. I am not sure in which scenario this will happen.

anmolnar · 2026-03-09T16:05:11Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java

+          new String(getSuffixFileDataToWrite(), StandardCharsets.UTF_8));
      } catch (FileNotFoundException fnfe) {
        // this is the active cluster, create active cluster suffix file if it does not exist
        FSUtils.setActiveClusterSuffix(fs, rootdir, getSuffixFileDataToWrite(), wait);


Logging of this branch is also inaccurate. This is the point where we first create the suffix file when file system is initialized in the active cluster and no logs are emitted. The FSUtil method setActiveClusterSuffix will log, but only at debug level. Please add some meaningful log message here too.

anmolnar · 2026-03-09T16:06:33Z

@anmolnar I am not much familiar with the nitty-gritty of this file based implementation as some other engineer worked on this but IIRC this may means that active.cluster.id.file may have been created without any data in it. I am not sure in which scenario this will happen.

I mentioned already:

I restarted the active cluster when the active cluster suffix file was present already with the cluster id + blank suffix value: 0115d976-1306-44c7-b8b6-f2644f8867c9:<blank>

sharmaar12 · 2026-03-09T16:23:37Z

@anmolnar Let me change accordingly.

Kota-SH · 2026-03-09T16:02:39Z

hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java

    String suffix_val = conf.get(HConstants.HBASE_META_TABLE_SUFFIX,
      HConstants.HBASE_META_TABLE_SUFFIX_DEFAULT_VALUE);
-    LOG.info("Meta table suffix value: {}", suffix_val);
+    LOG.info("Read Replica Cluster suffix value: {}", suffix_val);


This method is called by every cluster, active/replica. Labeling it "Read Replica Cluster" in the log will cause confusion for standard (non-replica) deployments.

I'd like to connect the message to the feature somehow. What do you recommend?
We're not implementing a big feature flag for the feature, so once this is merged every HBase cluster will be either an active or a read-only cluster.

Perhaps we could move this log into the else block (line 98)? That way it only fires when the suffix exists. If not, adding a prefix like "[Read-replica feature]" to the log line would also work well.

Since we’re initializing the meta table name here, I think having "meta table name suffix" in the log would be a bit more informative, wdyt?

We might want to remove this log line completely. Meta table name is already logged as:

2026-03-09T16:12:48,322 INFO [main] hbase.TableName: Meta table name: hbase:meta

Which should be enough.

...or move it to debug level with the "[Read-replica feature]" prefix that you suggested.

Yeah, it might be redundant at info level, we can move it to debug.

Added this to debug log.

Kota-SH · 2026-03-09T16:38:58Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java

Detail about which file and which config in this log line would be more helpful while debugging.

@Kota-SH Which config you are referring here?

In line#402, File Suffix {} : Configured suffix {}, adding some details about which file and which configured suffix would be helpful

Kota-SH · 2026-03-09T16:40:09Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java

        } else {
          // throw error
-          LOG.info("rootdir {} : Active Cluster File Suffix {} ", rootdir, acs);
+          LOG.info("rootdir {} : Read replica active cluster file suffix {} ", rootdir, acs);


This is a mismatch scenario, logging something like "Active cluster suffix mismatch -- rootdir: {} but the file contains: {}, another cluster is running in active" would be more helpful imo.

Change this to:
LOG.info("[Read-replica feature] Another cluster is running in active (read-write) mode on this storage location. Active cluster ID: {}, This cluster ID {}. Rootdir location {} ", acs, getSuffixFromConfig(), rootdir);
Hope this helps.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java

hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java

…s.java Co-authored-by: Kota-SH <shanmukhaharipriya@gmail.com>

…erFileSystem.java Co-authored-by: Kota-SH <shanmukhaharipriya@gmail.com>

sharmaar12

Updated few log messages.

Remaining things are appending when suffix is empty
Why log message "LOG.warn("[Read-replica feature] id file {} is empty ", idPath);" is present?

…#7854)

HBASE-29958 Improve log messages

4dfea60

sharmaar12 marked this pull request as ready for review March 5, 2026 15:27

anmolnar requested changes Mar 5, 2026

View reviewed changes

Address review comments

d347361

sharmaar12 commented Mar 7, 2026

View reviewed changes

anmolnar requested changes Mar 9, 2026

View reviewed changes

anmolnar reviewed Mar 9, 2026

View reviewed changes

Kota-SH reviewed Mar 9, 2026

View reviewed changes

sharmaar12 and others added 3 commits March 9, 2026 23:57

Update hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtil…

7647599

…s.java Co-authored-by: Kota-SH <shanmukhaharipriya@gmail.com>

Update hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtil…

6193503

…s.java Co-authored-by: Kota-SH <shanmukhaharipriya@gmail.com>

Update hbase-server/src/main/java/org/apache/hadoop/hbase/master/Mast…

f928a95

…erFileSystem.java Co-authored-by: Kota-SH <shanmukhaharipriya@gmail.com>

sharmaar12 commented Mar 10, 2026

View reviewed changes

sharmaar12 added 3 commits March 10, 2026 23:52

HBASE-29961 Secondary cluster is unable to replayWAL for meta (apache…

7af14f2

…#7854)

Add <blank> when no suffix provided

4e9e33d

Address few review comments

7a1ea18

Conversation

sharmaar12 commented Mar 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sharmaar12 left a comment

Choose a reason for hiding this comment

Uh oh!

anmolnar left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sharmaar12 commented Mar 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anmolnar commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sharmaar12 commented Mar 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anmolnar Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anmolnar Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sharmaar12 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

anmolnar left a comment •

edited

Loading

anmolnar commented Mar 9, 2026 •

edited

Loading

anmolnar Mar 9, 2026 •

edited

Loading

anmolnar Mar 9, 2026 •

edited

Loading