Skip to content

fix: For the openedx-events broker, use kafka _everywhere_ instead of redis #142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pwnage101 merged 1 commit into from
May 15, 2025
Merged

fix: For the openedx-events broker, use kafka _everywhere_ instead of redis #142

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion docker-compose-host.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ services:
- ${DEVSTACK_WORKSPACE}/src:/edx/src
- ${PWD}/py_configuration_files/cms.py:/edx/app/edxapp/edx-platform/cms/envs/devstack.py
- ${PWD}/py_configuration_files/lms.py:/edx/app/edxapp/edx-platform/lms/envs/devstack.py
- ${PWD}/py_private_requirements/lms.txt:/edx/private_requirements.txt
lms-worker:
volumes:
- ${DEVSTACK_WORKSPACE}/edx-platform:/edx/app/edxapp/edx-platform
Expand Down Expand Up @@ -71,6 +72,7 @@ services:
- ${DEVSTACK_WORKSPACE}/src:/edx/src
- ${PWD}/py_configuration_files/cms.py:/edx/app/edxapp/edx-platform/cms/envs/devstack.py
- ${PWD}/py_configuration_files/lms.py:/edx/app/edxapp/edx-platform/lms/envs/devstack.py
- ${PWD}/py_private_requirements/cms.txt:/edx/private_requirements.txt
Copy link
Contributor

@adamstankiewicz adamstankiewicz May 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[curious] Are there no private_requirements.txt for the LMS?

For example, is there an equivalent like the following needed:

${PWD}/py_private_requirements/lms.txt:/edx/private_requirements.txt

Copy link
Member Author

@pwnage101 pwnage101 May 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly, I don't know why settings are installed for both cms and lms for both images. But, in this case for requirements it would be redundant because they go to the same destination path inside the container.

cms-worker:
volumes:
- ${DEVSTACK_WORKSPACE}/edx-platform:/edx/app/edxapp/edx-platform
Expand All @@ -95,7 +97,11 @@ services:
- ${DEVSTACK_WORKSPACE}/enterprise-subsidy:/edx/app/enterprise-subsidy
- ${DEVSTACK_WORKSPACE}/src:/edx/src
- ${PWD}/py_configuration_files/enterprise_subsidy.py:/edx/app/enterprise-subsidy/enterprise_subsidy/settings/devstack.py

enterprise-subsidy-consume_learner_credit_course_enrollment_lifecycle:
volumes:
- ${DEVSTACK_WORKSPACE}/enterprise-subsidy:/edx/app/enterprise-subsidy
- ${DEVSTACK_WORKSPACE}/src:/edx/src
- ${PWD}/py_configuration_files/enterprise_subsidy.py:/edx/app/enterprise-subsidy/enterprise_subsidy/settings/devstack.py
enterprise-catalog:
volumes:
- ${DEVSTACK_WORKSPACE}/enterprise-catalog:/edx/app/enterprise_catalog/enterprise_catalog
Expand Down
20 changes: 18 additions & 2 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -449,7 +449,7 @@ services:

lms:
# Switch to `--settings devstack_with_worker` if you want to use lms-worker
command: bash -c 'source /edx/app/edxapp/edxapp_env && while true; do python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack; sleep 2; done'
command: bash -c 'source /edx/app/edxapp/edxapp_env && (pip install -r /edx/private_requirements.txt; while true; do python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack; sleep 2; done)'
Copy link
Contributor

@adamstankiewicz adamstankiewicz May 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Semi-related to my comment/question around LMS private_requirements.txt above; just trying to understand why these commands need to include pip install -r /edx/private_requirements.txt now when they didn't before.

Copy link
Member Author

@pwnage101 pwnage101 May 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are several loosely related facts which are required to understand the situation:

  • openedx-events is architected to support either Redis or Kafka as the message broker, and the docs recommend to use the same in both your prod and dev environments.
  • Redis python libraries are already installed in many/most of our django apps, including edx-platform.
  • Kafka python libraries are NOT installed in openedx/edx-platform, and it seems like Axim wants to keep it that way.
  • For the 2U edx.org prod environment, Kafka was chosen as the platform-wide message broker. As such, we have custom python requirements [1, 2, 3] installed during our edxapp image build process to supplement the default requirements from production.txt.
  • Devstack also used to be an openedx repo, and could not contain references to Kafka, therefore by default used the Redis broker. This is no longer a constraint because we forked Devstack.
  • enterprise-subsidy in devstack was configured to use Kafka, and never successfully consumed events produced by edxapp (until this PR).

So, basically I needed a way to install extra Kafka python requirements into the edxapp container without modifying the edxapp image. The images Devstack pulls are all the standard openedx images published to docker hub, and there's no pre-existing pipeline to build 2u-devstack-specific images, so adding a docker-compose step to install additional requirements seemed appropriate.

container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.lms"
hostname: lms.devstack.edx
depends_on:
Expand Down Expand Up @@ -638,7 +638,7 @@ services:

cms:
# Switch to `--settings devstack_with_worker` if you want to use cms-worker
command: bash -c 'source /edx/app/edxapp/edxapp_env && while true; do python /edx/app/edxapp/edx-platform/manage.py cms runserver 0.0.0.0:18010 --settings devstack; sleep 2; done'
command: bash -c 'source /edx/app/edxapp/edxapp_env && (pip install -r /edx/private_requirements.txt; while true; do python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack; sleep 2; done)'
container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.cms"
hostname: cms.devstack.edx
depends_on:
Expand Down Expand Up @@ -905,6 +905,22 @@ services:
volumes:
- /edx/var/enterprise-subsidy

enterprise-subsidy-consume_learner_credit_course_enrollment_lifecycle:
image: edxops/enterprise-subsidy-dev
container_name: edx.devstack.enterprise-subsidy-consume_learner_credit_course_enrollment_lifecycle
depends_on:
- mysql80
- memcached
command: >-
bash -c 'while true; do python /edx/app/enterprise-subsidy/manage.py consume_events -t learner-credit-course-enrollment-lifecycle -g enterprise_subsidy_dev; sleep 2; done'
tty: true
environment:
DJANGO_SETTINGS_MODULE: enterprise_subsidy.settings.devstack
working_dir: /edx/app/enterprise-subsidy
ports:
- "18281:18281"
stdin_open: true

# ==========================================================================
# edX Microfrontends
#
Expand Down
11 changes: 6 additions & 5 deletions py_configuration_files/cms.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@

################################ DEBUG TOOLBAR ################################

INSTALLED_APPS += ['debug_toolbar']
INSTALLED_APPS += ('debug_toolbar',)

MIDDLEWARE.append('debug_toolbar.middleware.DebugToolbarMiddleware')
INTERNAL_IPS = ('127.0.0.1',)
Expand Down Expand Up @@ -296,11 +296,12 @@ def should_show_debug_toolbar(request): # lint-amnesty, pylint: disable=missing
]

#################### Event bus backend ########################

EVENT_BUS_PRODUCER = 'edx_event_bus_redis.create_producer'
EVENT_BUS_REDIS_CONNECTION_URL = 'redis://:[email protected]:6379/'
INSTALLED_APPS += ('edx_event_bus_kafka',)
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL = 'http://edx.devstack.schema-registry:8081'
EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS = 'edx.devstack.kafka:29092'
EVENT_BUS_PRODUCER = 'edx_event_bus_kafka.create_producer'
EVENT_BUS_CONSUMER = 'edx_event_bus_kafka.KafkaEventConsumer'
EVENT_BUS_TOPIC_PREFIX = 'dev'
EVENT_BUS_CONSUMER = 'edx_event_bus_redis.RedisEventConsumer'

course_catalog_event_setting = EVENT_BUS_PRODUCER_CONFIG['org.openedx.content_authoring.course.catalog_info.changed.v1']
course_catalog_event_setting['course-catalog-info-changed']['enabled'] = True
Expand Down
9 changes: 6 additions & 3 deletions py_configuration_files/course_discovery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,12 @@

CELERY_TASK_ALWAYS_EAGER = False

EVENT_BUS_CONSUMER = 'edx_event_bus_redis.RedisEventConsumer'
EVENT_BUS_PRODUCER = 'edx_event_bus_redis.create_producer'
EVENT_BUS_REDIS_CONNECTION_URL = 'redis://:[email protected]:6379/'
#################### Event bus backend ########################
INSTALLED_APPS += ('edx_event_bus_kafka',)
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL = 'http://edx.devstack.schema-registry:8081'
EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS = 'edx.devstack.kafka:29092'
EVENT_BUS_PRODUCER = 'edx_event_bus_kafka.create_producer'
EVENT_BUS_CONSUMER = 'edx_event_bus_kafka.KafkaEventConsumer'
EVENT_BUS_TOPIC_PREFIX = 'dev'

#####################################################################
Expand Down
6 changes: 2 additions & 4 deletions py_configuration_files/enterprise_access.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,10 +61,7 @@

# Install django-extensions for improved dev experiences
# https://github.com/django-extensions/django-extensions#using-it
INSTALLED_APPS += (
'django_extensions',
'edx_event_bus_kafka',
)
INSTALLED_APPS += ('django_extensions',)

# BEGIN CELERY
CELERY_WORKER_HIJACK_ROOT_LOGGER = True
Expand Down Expand Up @@ -114,6 +111,7 @@
################### Kafka Related Settings ##############################

# "Standard" Kafka settings as defined in https://github.com/openedx/event-bus-kafka/tree/main
INSTALLED_APPS += ('edx_event_bus_kafka',)
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL = 'http://edx.devstack.schema-registry:8081'
EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS = 'edx.devstack.kafka:29092'
EVENT_BUS_PRODUCER = 'edx_event_bus_kafka.create_producer'
Expand Down
5 changes: 1 addition & 4 deletions py_configuration_files/enterprise_subsidy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,6 @@

}

INSTALLED_APPS += (
'edx_event_bus_kafka',
)

# Generic OAuth2 variables irrespective of SSO/backend service key types.
OAUTH2_PROVIDER_URL = 'http://edx.devstack.lms:18000/oauth2'

Expand Down Expand Up @@ -83,6 +79,7 @@

# Kafka Settings
# "Standard" Kafka settings as defined in https://github.com/openedx/event-bus-kafka/tree/main
INSTALLED_APPS += ('edx_event_bus_kafka',)
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL = 'http://edx.devstack.schema-registry:8081'
EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS = 'edx.devstack.kafka:29092'
EVENT_BUS_PRODUCER = 'edx_event_bus_kafka.create_producer'
Expand Down
10 changes: 6 additions & 4 deletions py_configuration_files/lms.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@

################################ DEBUG TOOLBAR ################################

INSTALLED_APPS += ['debug_toolbar']
INSTALLED_APPS += ('debug_toolbar',)
MIDDLEWARE += [
'lms.djangoapps.discussion.django_comment_client.utils.QueryCountDebugMiddleware',
'debug_toolbar.middleware.DebugToolbarMiddleware',
Expand Down Expand Up @@ -504,10 +504,12 @@ def should_show_debug_toolbar(request): # lint-amnesty, pylint: disable=missing
CLOSEST_CLIENT_IP_FROM_HEADERS = []

#################### Event bus backend ########################
EVENT_BUS_PRODUCER = 'edx_event_bus_redis.create_producer'
EVENT_BUS_REDIS_CONNECTION_URL = 'redis://:[email protected]:6379/'
INSTALLED_APPS += ('edx_event_bus_kafka',)
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL = 'http://edx.devstack.schema-registry:8081'
EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS = 'edx.devstack.kafka:29092'
EVENT_BUS_PRODUCER = 'edx_event_bus_kafka.create_producer'
EVENT_BUS_CONSUMER = 'edx_event_bus_kafka.KafkaEventConsumer'
Comment on lines +508 to +511
Copy link

@brobro10000 brobro10000 May 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥳

EVENT_BUS_TOPIC_PREFIX = 'dev'
EVENT_BUS_CONSUMER = 'edx_event_bus_redis.RedisEventConsumer'

certificate_revoked_event_config = EVENT_BUS_PRODUCER_CONFIG['org.openedx.learning.certificate.revoked.v1']
certificate_revoked_event_config['learning-certificate-lifecycle']['enabled'] = True
Expand Down
1 change: 1 addition & 0 deletions py_private_requirements/cms.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
confluent-kafka[avro,schema-registry]==2.10.0
1 change: 1 addition & 0 deletions py_private_requirements/lms.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
confluent-kafka[avro,schema-registry]==2.10.0
Loading