DAOS-15204 cart: add crt_perf utils by soumagne · Pull Request #13825 · daos-stack/daos

soumagne · 2024-02-16T22:59:33Z

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2024-02-16T22:59:50Z

Bug-tracker data:
Ticket title is 'Add crt_perf utilities'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-15204

daosbuild1 · 2024-02-16T23:03:22Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/1/execution/node/376/log

daosbuild1 · 2024-02-16T23:03:42Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/1/execution/node/383/log

daosbuild1 · 2024-02-16T23:06:30Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/1/execution/node/305/log

daosbuild1 · 2024-02-16T23:06:42Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/1/execution/node/361/log

daosbuild1 · 2024-02-16T23:07:59Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/1/execution/node/349/log

daosbuild1 · 2024-02-16T23:11:32Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/1/execution/node/327/log

github-actions · 2024-02-16T23:37:16Z

Bug-tracker data:
Ticket title is 'Add crt_perf utilities'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-15204

daosbuild1 · 2024-02-16T23:40:18Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/2/execution/node/295/log

daosbuild1 · 2024-02-16T23:41:15Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/2/execution/node/380/log

daosbuild1 · 2024-02-16T23:43:07Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/2/execution/node/278/log

daosbuild1 · 2024-02-16T23:46:21Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/2/execution/node/370/log

daosbuild1 · 2024-02-16T23:47:01Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/2/execution/node/383/log

github-actions · 2024-02-16T23:50:43Z

Bug-tracker data:
Ticket title is 'Add crt_perf utilities'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-15204

daosbuild1 · 2024-02-16T23:50:55Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/2/execution/node/324/log

daosbuild1 · 2024-02-16T23:58:04Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/3/execution/node/287/log

daosbuild1 · 2024-02-16T23:59:25Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/3/execution/node/310/log

daosbuild1 · 2024-02-17T00:01:20Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/3/execution/node/315/log

daosbuild1 · 2024-02-17T00:06:11Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/3/execution/node/344/log

daosbuild1 · 2024-02-17T00:26:05Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/4/execution/node/383/log

daosbuild1 · 2024-02-17T00:26:18Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/4/execution/node/378/log

daosbuild1 · 2024-02-17T00:26:20Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/4/execution/node/322/log

github-actions · 2024-02-17T00:26:59Z

Bug-tracker data:
Ticket title is 'Add crt_perf utilities'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-15204

daosbuild1 · 2024-02-17T00:30:04Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/4/execution/node/377/log

github-actions · 2024-02-20T20:26:18Z

Bug-tracker data:
Ticket title is 'Add crt_perf utilities'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-15204

daosbuild1 · 2024-02-20T20:35:33Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/5/execution/node/363/log

daosbuild1 · 2024-02-20T20:35:40Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/5/execution/node/278/log

github-actions · 2024-02-20T21:00:40Z

Bug-tracker data:
Ticket title is 'Add crt_perf utilities'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-15204

frostedcmos · 2024-03-20T17:34:04Z

src/utils/crt_perf/crt_bw_read.c

+			CRT_PERF_CHECK_D_ERROR(error, rc, "could not create request");
+
+			in_struct = crt_req_get(request->rpc);
+			in_struct->handle_id =


a comment here would help to explain what you are doing:)

frostedcmos · 2024-03-20T17:40:22Z

src/utils/crt_perf/crt_bw_write.c

+/*******************/
+
+static int
+crt_perf_run(const struct crt_perf_info *perf_info, struct crt_perf_context_info *info,


it looks like crt_perf_run() implementations in most tests is very similar with the main difference being how input args are populated. any way to move this into crt_perf.c perhaps and pass it a function/cb to populate args instead?

frostedcmos · 2024-03-20T17:42:18Z

src/utils/crt_perf/crt_perf.h

+#define CRT_PERF_LAT_SKIP_SMALL 100
+#define CRT_PERF_LAT_SKIP_LARGE 10
+#define CRT_PERF_LARGE_SIZE     8192


would help to explain those, i dont quite understand what they mean by just reading them

src/utils/crt_perf/crt_perf_server.c

frostedcmos · 2024-03-20T17:53:28Z

src/utils/crt_perf/crt_perf.c

+				     sizeof(uri_name) * info->mpi_info.size);
+
+		rc = crt_perf_mpi_allgather(&info->mpi_info, uri_name, sizeof(uri_name), uri_list,
+					    sizeof(uri_name));


hmm.. shoudnt the last arg here be sizeof(uri_name) * info->mpi_info.size?

frostedcmos · 2024-03-20T17:55:51Z

src/utils/crt_perf/crt_perf.c

+			char *rank_uri = uri_list + rank * sizeof(uri_name);
+
+			if (rank == info->mpi_info.rank)
+				continue; /* our rank is already added */
+
+			rc = crt_group_primary_rank_add(info->context_info[0].context, NULL, rank,
+							rank_uri);
+			CRT_PERF_CHECK_D_ERROR(error, rc, "could not add rank %d %s", rank,


you can also use crt_group_primary_modify() and add all ranks and uris in a bulk if you want to avoid the loop

src/utils/crt_perf/crt_perf.c

frostedcmos · 2024-03-20T18:04:10Z

src/utils/crt_perf/crt_perf.c

+	rc = crt_init_opt(listen ? CRT_PERF_GROUP_ID : NULL, crt_init_flags, &crt_init_options);
+	CRT_PERF_CHECK_D_ERROR(error, rc, "could not init CART");
+
+	if (attach_info_path) {


from past experience might be helpful to debug/info print the attach info path

frostedcmos · 2024-03-20T18:05:06Z

src/utils/crt_perf/crt_perf.c

+	CRT_PERF_CHECK_ERROR(cb_info->cci_rc != 0, out, info->rc, cb_info->cci_rc,
+			     "callback failed");
+
+	if (info->cb) {


surprised checkpatch doesnt complain about single statement being wrapped in { } 's

github-actions · 2024-09-24T17:44:08Z

Ticket title is 'Add crt_perf utilities'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-15204

daosbuild1 · 2024-09-24T18:21:02Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13825/7/testReport/

daosbuild1 · 2024-09-24T18:55:27Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13825/7/testReport/

daosbuild1 · 2024-10-01T22:19:57Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/11/execution/node/816/log

frostedcmos · 2024-09-24T18:11:27Z

src/cart/SConscript

+    # compiler = env.get('COMPILER').lower()
+    # if compiler != 'covc':
+    #     pp_env = denv.Clone()
+    #     pp_files = []
+    #     for src in SRC:
+    #         # Some day, the preprocess builder should be fixed so it can do multiple commands
+    #         # in parallel but until then, just submit them one at a time
+    #         pp_files.extend(pp_env.Preprocess(src))
+    #     parsed_files = parse_pp(pp_env, pp_files)
+    #     header = consolidate_pp(pp_env, parsed_files)
+
+    #     Depends(cart_targets, header)


while its annoying, some people use post-processed headers in their IDE. i think ideally this should be done via a flag/env/scons option

frostedcmos · 2024-10-17T16:49:40Z

src/cart/crt_internal_types.h

 	unsigned int             cg_provider_is_primary : 1;
 	/** use legacy progress method */
 	bool                     cg_progress_legacy;
+	/** use single thread to access context */


might want to add new field to be dumped in crt_gdata_dump(void). even though we dump envs automatically, someone might set new field via init_opt, and would be good to see that in logs.

daosbuild1 · 2025-02-20T23:46:51Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13825/12/execution/node/804/log

add crt_hg_event_progress() and query wait fd from context add D_PROGRESS_LEGACY to revert to old progress relax progress thread safety using thread mode introduce spin flag to busy spin until deadline Signed-off-by: Jerome Soumagne <jerome.soumagne@hpe.com>

Avoid reacquiring time and use coarse clock Signed-off-by: Jerome Soumagne <jerome.soumagne@hpe.com>

Required-githooks: true Signed-off-by: Jerome Soumagne <jerome.soumagne@intel.com>

Add crt_rate / crt_perf_server executables. CRT perf utils can be run with and without MPI support. MPI support is automatically detected. Add crt_bw_read/write benchmarks. Required-githooks: true Signed-off-by: Jerome Soumagne <jerome.soumagne@intel.com>

Required-githooks: true Signed-off-by: Jerome Soumagne <jerome.soumagne@intel.com>

daosbuild3 · 2026-03-06T18:57:48Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-13825/4/testReport/

soumagne force-pushed the soumagne/crt_perf branch from e5ef127 to d8d6aaa Compare February 16, 2024 23:36

soumagne force-pushed the soumagne/crt_perf branch from 18cc1c5 to 7d2ffe3 Compare February 20, 2024 21:00

frostedcmos reviewed Mar 20, 2024

View reviewed changes

soumagne force-pushed the soumagne/crt_perf branch from 7d2ffe3 to cc432f1 Compare September 24, 2024 17:43

soumagne force-pushed the soumagne/crt_perf branch 3 times, most recently from 434c1ca to 9bbc837 Compare September 24, 2024 21:35

soumagne force-pushed the soumagne/crt_perf branch from 9bbc837 to c6a58cd Compare October 1, 2024 21:49

frostedcmos reviewed Oct 17, 2024

View reviewed changes

soumagne force-pushed the soumagne/crt_perf branch from c6a58cd to 10fbe20 Compare February 20, 2025 23:11

soumagne force-pushed the soumagne/crt_perf branch from 10fbe20 to 5aa501c Compare April 23, 2025 22:47

soumagne and others added 6 commits March 4, 2026 18:30

DAOS-15204 cart: switch RPC timeout to use timespec deadlines

2099fc9

Avoid reacquiring time and use coarse clock Signed-off-by: Jerome Soumagne <jerome.soumagne@hpe.com>

DAOS-15204 cart: fix thread-safety in threaded test

5fffa37

Required-githooks: true Signed-off-by: Jerome Soumagne <jerome.soumagne@intel.com>

DAOS-15204 cart: disable intermediate preprocessor build

945bc0c

Required-githooks: true Signed-off-by: Jerome Soumagne <jerome.soumagne@intel.com>

DAOS-15204 build: add crt_perf to packaging

aefa04b

Required-githooks: true Signed-off-by: Jerome Soumagne <jerome.soumagne@intel.com>

soumagne force-pushed the soumagne/crt_perf branch from 5aa501c to aefa04b Compare March 6, 2026 18:16

Conversation

soumagne commented Feb 16, 2024

Before requesting gatekeeper:

Gatekeeper:

Uh oh!

github-actions bot commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

github-actions bot commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

github-actions bot commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 16, 2024

Uh oh!

daosbuild1 commented Feb 17, 2024

Uh oh!

daosbuild1 commented Feb 17, 2024

Uh oh!

daosbuild1 commented Feb 17, 2024

Uh oh!

daosbuild1 commented Feb 17, 2024

Uh oh!

daosbuild1 commented Feb 17, 2024

Uh oh!

github-actions bot commented Feb 17, 2024

Uh oh!

daosbuild1 commented Feb 17, 2024

Uh oh!

github-actions bot commented Feb 20, 2024

Uh oh!

daosbuild1 commented Feb 20, 2024

Uh oh!

daosbuild1 commented Feb 20, 2024

Uh oh!

github-actions bot commented Feb 20, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 24, 2024

Uh oh!

daosbuild1 commented Sep 24, 2024